Scaling Laws and Pretraining

The forecast behind a training run

A frontier pretraining run ties up tens of thousands of GPUs for months and can cost tens of millions of dollars in compute alone. You get one shot. Scaling laws are the reason anyone dares to press the button: they let a lab predict how good the model will be before spending the money.

What is it?

A scaling law is an empirical formula that predicts a model's loss from three quantities: how many parameters it has, how many tokens of text it trains on, and how much compute the training uses. Across the range where they have been measured, these relationships are astonishingly regular. Kaplan and colleagues at OpenAI (2020) found smooth power-law trends spanning more than seven orders of magnitude of scale.

Pretraining is the stage those laws describe: a transformer learns to predict the next token over trillions of tokens of text and code. It consumes most of a model's training compute and supplies most of its knowledge. Everything that comes after, from instruction tuning to reinforcement learning, starts from what pretraining built.

How does it work?

Labs train a ladder of small models, each with the same recipe but a different size or budget, plot their losses, fit a curve and extrapolate to the big run. OpenAI's GPT-4 technical report describes exactly this: GPT-4's final loss was predicted in advance from models trained with the same method but at most 10,000 times less compute.

Why does it matter?

Scaling laws explain most of the big strategic swings in AI over the past six years:

• Why models ballooned from 1.5 billion to 175 billion parameters between 2019 and 2020, and then to 280 and 530 billion, while training data stayed around 300 billion tokens.
• Why, after 2022, parameter counts stopped exploding and data exploded instead: Llama 3 405B trained on 15.6 trillion tokens, fifty times GPT-3's diet.
• Why small open models you can run on a laptop are now far stronger than their size suggests: they are deliberately trained far past the textbook optimum.
• Why, since 2024, the frontier talks about test-time compute and reasoning, and worries about running out of human-written text.

This lesson works through each of those in turn, with the published equations running live in your browser so you can make the same trade-offs the labs make.

Key takeaways

• Scaling laws predict a model’s loss from its parameters, training tokens and compute, and have held across many orders of magnitude.
• Labs use them to forecast an expensive run from a ladder of cheap ones; GPT-4’s loss was predicted from runs using up to 10,000× less compute.
• The laws explain the shift from ever-bigger models to ever-more data, the strength of small open models, and the new focus on test-time compute.

What pretraining optimises

Every scaling law is a law about one number: the loss. Before we can read the curves we need to know exactly what that number measures, and what it means for it to fall from 2.0 to 1.9.

What is the pretraining objective?

Take a long stream of text, cut it into tokens, and at every position ask the model for a probability distribution over the next token. The loss at that position is the negative log of the probability the model gave to the token that actually came next. Average it over billions of positions and you have the cross-entropy loss, measured in nats when the log is natural.

That is the whole objective. There are no labels, no human judgements and no task definitions: the text supervises itself. Yet predicting the next token well requires grammar, facts, arithmetic, the plot of a story and the intent of the person writing, so all of those get learned on the way.

What does a loss of 2 nats mean?

Three ways to read the same number:

• Probability. If the model gives the right token probability 0.25, the loss at that position is −ln 0.25 = 1.39 nats. A loss of 2 nats is the geometric-mean equivalent of giving the right token probability e⁻² ≈ 0.135.
• Perplexity. e^loss is the number of equally likely options the model is effectively choosing between. Loss 2 means a perplexity of 7.4: on average the model is as unsure as if it were picking among seven or eight tokens, out of a vocabulary of 100,000 or more.
• Compression. A model with loss L can compress text to L / ln 2 bits per token using arithmetic coding. Lower loss literally means a better compressor, which is why some researchers describe pretraining as learning to compress the internet.

Loss values are only comparable between models that share a tokenizer and an evaluation set. A token is roughly four characters of English for typical tokenizers, so a loss per token is not a loss per character; the demo below works in characters.

The pretraining loss over a corpus of T tokens. Every scaling law in this lesson predicts this quantity on held-out text.

Scaling in miniature

We can run a real, if tiny, version of the whole story. A character n-gram model predicts the next character by counting what followed the previous few characters in its training text. Its “size” is its context length: a longer context can capture more structure, but it has many more counts to fill in. Below, 56 models (7 context lengths × 8 data sizes) are trained on Alice's Adventures in Wonderland and scored on the book's last 15,000 characters, which none of them see.

Three things in that chart reappear at the scale of real LLMs. First, more data always helps a fixed model, with diminishing returns. Second, curves cross: with little data the small models win; with more data the bigger ones overtake them. Third, the grey envelope (the best model at each data size) keeps falling, but only if you grow the model as the data grows. With 1,000 characters the best context is one character; with the whole book it is three. That is the Chinchilla result in miniature, which we will meet properly two sections from now.

Where the analogy breaks: n-gram models memorise counts and cannot generalise across similar contexts, so they need exponentially more data as context grows. Neural networks share structure through their weights, which is why their curves are smooth power laws rather than this abrupt overfitting.

Key takeaways

• Pretraining minimises cross-entropy: the average negative log-probability of the true next token, in nats.
• Perplexity is e to the loss; loss divided by ln 2 is the bits per token a model needs to compress text.
• Even a tiny n-gram model shows the core pattern: bigger models need more data, and the best model size grows with the data.

Power laws

In 2020 an OpenAI team trained hundreds of transformers, from under a thousand to over a billion parameters, and found that loss falls along the simplest curve imaginable: a straight line on log-log paper.

What is a power law?

A power law says that y is proportional to x raised to some exponent: y = a · x^−α. Its signature is scale invariance. Multiplying x by ten always multiplies y by the same factor, 10^−α, no matter where you start. Take logs of both sides and you get log y = log a − α log x: a straight line with slope −α.

Kaplan et al. found three such laws for transformer language models, each holding when the other resources are not the bottleneck:

Kaplan et al. 2020, fitted on WebText2. N counts non-embedding parameters, D tokens, C_min the compute of an efficiently sized run. N_c = 8.8 × 10¹³ and D_c = 5.4 × 10¹³.

How should you read the exponents?

The exponents are small. With α = 0.076, ten times more parameters multiplies the loss by 10^−0.076 ≈ 0.84, a 16% drop. Halving the loss would take roughly 10⁴ times more parameters. Progress by scaling is slow but reliable, and it compounds: each new order of magnitude buys a similar step.

Kaplan's pure power laws head towards zero loss, which cannot be right, because text has genuine randomness that no model can predict. Later work adds an irreducible loss E, the entropy of the text itself, and writes the loss as a floor plus two shrinking terms, one for limited parameters and one for limited data:

The parametric form used by Hoffmann et al. 2022. Only the part above E shrinks as a power law. Their fitted exponents are about 0.34 for parameters and 0.28 for data.

Why does it matter?

A straight line is something you can extrapolate with a ruler. That changed how AI research is funded: instead of hoping a larger model would be better, labs could state how much better, and what it would cost. Kaplan et al. also found that within a wide range, architectural details such as depth versus width barely moved the curves, while larger models were more sample-efficient, reaching a given loss with fewer tokens.

From these fits they concluded that as compute grows, most of it should go into model size (N growing roughly as C^0.73) and relatively little into data. That advice shaped the GPT-3 generation: 175 billion parameters trained on only 300 billion tokens, under two tokens per parameter. As we will see, it turned out to be wrong.

Key takeaways

• A power law is a straight line on log-log axes: each tenfold increase in scale multiplies the loss by the same factor.
• Kaplan’s exponents are small (0.076 for parameters), so gains are slow but predictable; modern fits add an irreducible floor E.
• Kaplan’s fits advised spending extra compute mainly on parameters, which is why GPT-3 saw under two tokens per parameter.

Counting compute

You can estimate the cost of training any dense transformer on the back of an envelope, and the answer usually lands within a few per cent of what the lab reports.

What is the rule?

Training compute is approximately C ≈ 6 N D floating-point operations, for N parameters and D training tokens. Serving the finished model costs about 2 N FLOPs per token.

Where does the 6 come from?

Almost all of a transformer's work is matrix multiplication. Each parameter takes part in one multiply and one add per token in the forward pass: 2 FLOPs. The backward pass must compute gradients with respect to both the activations and the weights, which costs about twice the forward pass: 4 more FLOPs. Two plus four is six, per parameter, per token.

The rule ignores the attention scores themselves (which grow with context length and are a small share at typical lengths) and any recomputation used to save memory. It also assumes a dense model. In a mixture of experts, each token only passes through a few experts, so N should be the active parameters per token.

How do FLOPs become days and dollars?

Divide by what the hardware actually delivers. An H100 peaks at 989 trillion dense BF16 FLOPs per second, but real training spends time on communication between GPUs, memory traffic and stalls. The fraction of peak that becomes useful model arithmetic is the model FLOPs utilisation (MFU). Meta reports 38% to 43% for Llama 3 405B, typical for very large runs.

Then divide by the number of GPUs for wall-clock time, and multiply GPU-hours by a price for the bill. DeepSeek's headline figure of $5.576 million for V3 was exactly this calculation, at an assumed $2 per GPU-hour, and the report is explicit that it excludes prior research and ablation experiments.

Try the Llama 3 405B preset on 16,384 H100s at 40% utilisation: the calculator gives about 27 million GPU-hours and roughly 68 days. Then try Llama 3 8B. It has 50 times fewer parameters but saw almost as many tokens, so it cost about 1/50th as much to train, and every token it generates costs 1/50th as much to serve. Keep that asymmetry in mind for the section on overtraining.

Key takeaways

• Training a dense transformer costs about 6 × parameters × tokens FLOPs; generating a token costs about 2 × parameters.
• Large runs turn only part of the hardware’s peak into useful work (Llama 3 405B: 38% to 43% MFU); that fraction turns FLOPs into GPU-hours, days and dollars.
• Headline training costs cover only the final run, not the experiments that led to it.

Compute-optimal training

In 2022 DeepMind showed that most of the largest language models in the world were the wrong shape. For the compute they had used, they were far too big and had read far too little.

What is the question?

Fix a compute budget C. Because C ≈ 6ND, every choice of model size N fixes the number of tokens D you can afford. A huge model sees few tokens; a small model sees many. Somewhere between is the size that gives the lowest loss. Plotting loss against N for a fixed budget gives a U-shaped iso-FLOP curve, and its minimum is the compute-optimal model.

How did Chinchilla answer it?

Hoffmann and colleagues trained over 400 models from 70 million to over 16 billion parameters and estimated the optimum three independent ways: by taking the lower envelope of many training curves, by fitting the minima of iso-FLOP curves directly, and by fitting the parametric law L(N, D) = E + A/N^α + B/D^β to every run. All three agreed on the headline: the optimal N and D both grow roughly as the square root of compute. Double the compute and you should make the model about 1.4 times bigger and train it on about 1.4 times more data. At the scales they studied, that works out to roughly 20 tokens per parameter.

That was a direct contradiction of Kaplan, whose fits put most extra compute into parameters. The Chinchilla authors traced much of the difference to training details: Kaplan's runs used a learning-rate schedule whose length did not match each run's number of tokens, which made shorter runs look worse than they really were, and so understated the value of data.

To prove the point they trained Chinchilla on the same budget as their own 280-billion-parameter Gopher. Four times smaller and fed four times more data, Chinchilla outperformed Gopher, GPT-3 and Megatron-Turing NLG (530B) across a large set of evaluations, reaching 67.5% on the MMLU knowledge benchmark, more than 7 points above Gopher. It was also four times cheaper to run.

The loss bars explain the U shape. A model that is too big has a tiny parameter term but a large data term: it has not seen enough text to use its capacity. A model that is too small has the opposite problem. At the optimum the two shrinking terms are in balance, and moving budget in either direction costs more than it gains.

Now switch fits. With Hoffmann's published constants, the optimum at Gopher's budget lands near 32 billion parameters and over 90 tokens per parameter, disagreeing with the paper's own other two methods. Besiroglu and colleagues (2024) reconstructed the data from the paper's figures, re-fitted the same equation and found the original fit was poorly converged. Their constants put the optimum at about 72 billion parameters and 1.3 trillion tokens, almost exactly Chinchilla itself. A fitted law is a model of models: only as reliable as the fit.

Notice too how flat the bottom of each curve is. Being off by a factor of two in model size costs only a little loss, which is why labs can depart from the optimum for other reasons without much penalty in quality. The next section is about the most important of those reasons.

Key takeaways

• For a fixed budget, loss against model size is U-shaped; the minimum is the compute-optimal size.
• Chinchilla found parameters and tokens should grow together, each roughly as the square root of compute: about 20 tokens per parameter.
• Gopher-era models were undertrained; Chinchilla, 4× smaller on 4× more data at equal compute, beat them.

Training past Chinchilla

Chinchilla tells you the cheapest way to train a model of a given quality. But a successful model is trained once and then runs trillions of tokens for millions of users. Once you count that, the answer changes completely.

What is overtraining?

Overtraining means training a model on many more tokens than the compute-optimal amount. It is not a mistake, and it does not mean overfitting: with trillions of unique tokens, held-out loss keeps falling. It means deliberately spending extra training compute to get a smaller model that is as good as a larger one.

How does it pay off?

Serving costs about 2N FLOPs per token, so a smaller model is proportionally cheaper to run, and it needs less memory, fewer GPUs and responds faster. Sardana and colleagues (2024) added this inference term to the Chinchilla analysis. For a target quality, the total lifetime cost is training plus serving:

For each model size N, D(N) is the number of tokens needed to reach the target loss, from the fitted law. T is the number of tokens the model will process in its lifetime.

Real models moved this way fast. Meta noted that the Chinchilla-optimal data for an 8-billion-parameter model is about 200 billion tokens, yet Llama 3 8B kept improving when trained on two orders of magnitude more: it saw over 15 trillion tokens, nearly 1,900 per parameter. Alibaba's Qwen3 family, which runs from 0.6 to 235 billion parameters, was pretrained on about 36 trillion tokens.

With little demand, the minimum sits on the Chinchilla size. Raise demand to 10¹³ tokens and the cheapest model shrinks by more than half and trains on hundreds of tokens per parameter. The curve on the left rises steeply, because for every target there is a smallest model that can never reach it however long it trains: its parameter term alone is too big.

That wall is the limit of overtraining. Returns diminish, and at some point doubling the data buys almost nothing. It is also why the frontier still trains very large models: to reach a quality nobody has reached, you need the capacity, and smaller descendants can then be distilled from them.

Key takeaways

• Chinchilla-optimal minimises training compute; once serving is counted, heavily used models should be smaller and trained longer.
• Llama 3 8B trained on over 15T tokens, about 75 times its Chinchilla-optimal 200B, and was still improving.
• Every target loss has a minimum model size below which no amount of data helps, which caps overtraining.

Smooth loss, sudden skills

Loss falls smoothly. But on some benchmarks, small models score zero, zero, zero, and then a larger one suddenly succeeds. Are new abilities switching on at a threshold, or is something about how we measure fooling us?

What are emergent abilities?

Wei and colleagues (2022) defined an ability as emergent if it is absent in smaller models and present in larger ones, so that it could not have been predicted by extrapolating the smaller models' performance. They catalogued dozens of examples across benchmarks such as multi-digit arithmetic, transliteration and multi-step word problems, where performance sits near random until a certain scale and then climbs steeply.

How could a smooth improvement look sudden?

Schaeffer, Miranda and Koyejo (2023) pointed at the metrics. Many of those benchmarks use exact match: an answer scores 1 only if every token is right. If a model gets each token of an L-token answer right with probability p, it gets the whole answer right with probability roughly p^L. As p creeps up smoothly, p^L stays pinned near zero and then shoots up. Scoring the same models with a continuous metric, such as per-token accuracy or edit distance, made many of the jumps vanish.

With a one-token answer the two curves coincide. At ten tokens the exact-match curve spends about three orders of magnitude of compute near the floor, then rises through the shaded band. At forty tokens it only reaches about 10% by the right-hand edge. Nothing about the model changed between those settings; only the ruler did. The linear axis adds a second illusion: most of the curve's history is crushed against the left edge, so any rise looks like a cliff.

So is emergence a mirage?

Partly. The measurement argument is solid, and it rescues predictability: if you track a continuous metric, many capabilities can be forecast from smaller models. OpenAI reported predicting GPT-4's pass rate on a subset of coding problems from models trained with at most 1,000 times less compute.

But thresholds are real for users. A proof with one wrong step, code with one bug or a plan with one impossible action is still a failure, so the exact-match view is sometimes the right one. And not every jump disappears under every metric. The practical lesson: before believing a story about sudden abilities, look at how the ability was scored and how many model sizes were tested.

Key takeaways

• Emergent abilities are ones that appear abruptly with scale on some benchmark rather than improving gradually.
• All-or-nothing metrics turn smooth per-token gains into sharp jumps: exact match on L tokens behaves like p to the power L.
• Continuous metrics make many capabilities forecastable, but some tasks genuinely only count when every step is right.

Where the curves bend

Scaling laws assume you can always buy more of each ingredient. Compute keeps getting cheaper. Human-written text does not keep getting more plentiful, and that is starting to bend the strategy.

The data wall

How much text is there?

Epoch AI researchers Villalobos and colleagues estimated the effective stock of public, human-generated text that is good enough to train on at about 300 trillion tokens, with a 90% interval from 100 to 1,000 trillion. That already allows for filtering and for repeating data a few times. Projecting how fast training sets grow, they estimated that frontier models would be trained on datasets as large as that whole stock sometime between 2026 and 2032, earlier if models are heavily overtrained.

For scale: Llama 3 used 15.6 trillion tokens in 2024 and Qwen3 about 36 trillion in 2025. A few more doublings exhaust the public web.

Can you just read the same data twice?

Yes, up to a point. Muennighoff and colleagues (2023) trained 400 models with controlled amounts of repetition. Up to about 4 epochs, repeated data was almost as good as fresh data. Beyond that each extra pass is worth less, and they fitted a law in which repeated tokens decay in value with a characteristic constant R* ≈ 15.4 repetitions.

Muennighoff et al. 2023: effective data D′ after R repetitions of U unique tokens (R = epochs − 1). The value can never exceed U(1 + R*), about 16 times the unique data.

Four epochs lose less than a tenth of their value. By 16 epochs the next pass is worth under 40% of a fresh one, and the curve is flattening towards its ceiling. Repetition stretches the data supply by a modest factor; it does not remove the wall.

What about synthetic data?

Labs increasingly generate training data with models. Qwen3's team, for example, used their earlier math and code models to synthesise extra math and code data, and used a vision-language model to extract text from PDF-like documents. Synthetic data works best where it can be checked, such as code that must pass tests or maths with verifiable answers, or where a strong model rewrites and filters real text.

The risk is closing the loop without fresh data. Shumailov and colleagues (2024) showed in Nature that models trained recursively on their predecessors' output progressively lose the tails of the original distribution, a failure they called model collapse. Mixing in real data and verifying synthetic data are the main defences.

New axes to scale

If pretraining slows, what scales next?

Pretraining is one way to turn compute into capability. Two others now have scaling curves of their own. In September 2024, OpenAI's o1 showed performance improving smoothly both with more reinforcement-learning compute during training and with more time spent thinking at test time. Snell and colleagues (2024) showed that spending test-time compute wisely, adapted to each problem's difficulty, improved efficiency by more than four times over best-of-N sampling, and that on problems where a smaller model already had some success, extra thinking could beat a model 14 times larger.

These axes interact. A reasoning model still needs a strong pretrained base, and test-time compute moves cost from training to serving, which changes the overtraining calculation you did above. Newer architectures push further: looped transformers reuse the same layers several times per token, so depth itself becomes a dial to turn at inference.

Key takeaways

• The usable stock of public human text is estimated at about 300T tokens, likely to be fully used by frontier training sometime between 2026 and 2032.
• Repeating data is nearly free up to about 4 epochs, then decays; synthetic data helps most where it can be verified, and recursive training on model output risks collapse.
• Reinforcement learning and test-time compute are new scaling axes alongside parameters and data.

Check your understanding

Seven scenarios. Each asks you to make the kind of call a training team makes: how to spend a budget, how to read a result, when to be suspicious.

A startup has a fixed training budget and two plans: a 30B model on 600B tokens, or a 7B model on 2.6T tokens. Both cost about the same compute. Using the Chinchilla rule of thumb, which should reach lower loss?

References

The papers behind every equation and number in this lesson. The fitted constants in the demos come from Hoffmann et al. (2022), Besiroglu et al. (2024), Kaplan et al. (2020) and Muennighoff et al. (2023). For a hands-on look at next-token prediction itself, try the Next-Token Sampling Lab.

Sources

1. [1]

   Scaling Laws for Neural Language Models(opens in a new tab)

   Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D., 2020

   The OpenAI study showing that language-model loss is a power law in parameters, data and compute over many orders of magnitude.

2. [2]

   GPT-4 Technical Report(opens in a new tab)

   OpenAI, 2023

   Reports predicting GPT-4’s final loss from smaller runs trained with the same method but at most 10,000× less compute.

3. [3]

   The Llama 3 Herd of Models(opens in a new tab)

   Llama Team, AI @ Meta (Grattafiori, A. et al.), 2024

   Llama 3 405B: 15.6 trillion tokens and 3.8 × 10²⁵ FLOPs of pretraining on up to 16,384 H100 GPUs.

4. [4]

   DeepSeek-V3 Technical Report(opens in a new tab)

   DeepSeek-AI, 2024

   671B-parameter mixture of experts (37B active per token) pretrained on 14.8T tokens; full training took 2.788M H800 GPU-hours.

5. [5]

   Training Compute-Optimal Large Language Models(opens in a new tab)

   Hoffmann, J., Borgeaud, S., Mensch, A., et al., 2022

   The DeepMind “Chinchilla” paper: for a fixed budget, parameters and tokens should grow in equal proportion. Chinchilla (70B) beat Gopher (280B) at the same compute.

6. [6]

   Chinchilla Scaling: A replication attempt(opens in a new tab)

   Besiroglu, T., Erdil, E., Barnett, M., You, J., 2024

   Re-fits Chinchilla’s parametric law on data reconstructed from the paper and finds constants consistent with about 20 tokens per parameter.

7. [7]

   Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws(opens in a new tab)

   Sardana, N., Portes, J., Doubov, S., Frankle, J., 2024

   Adds inference cost to the Chinchilla analysis and shows that models expected to serve heavy demand should be smaller and trained longer. ICML 2024.

8. [8]

   Introducing Meta Llama 3: The most capable openly available LLM to date(opens in a new tab)

   Meta AI, 2024

   Notes that the Chinchilla-optimal data for an 8B model is about 200B tokens, yet performance kept improving up to 15T tokens.

9. [9]

   Qwen3 Technical Report(opens in a new tab)

   Qwen Team (Yang, A. et al.), 2025

   A family of dense and mixture-of-experts models from 0.6B to 235B parameters, pretrained on about 36 trillion tokens.

10. [10]

    Emergent Abilities of Large Language Models(opens in a new tab)

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., et al., 2022

    Catalogues abilities absent in smaller models and present in larger ones, arguing they cannot be predicted by extrapolating small-model performance. TMLR 2022.

11. [11]

    Are Emergent Abilities of Large Language Models a Mirage?(opens in a new tab)

    Schaeffer, R., Miranda, B., Koyejo, S., 2023

    Shows that many apparent emergent jumps come from nonlinear or discontinuous metrics and disappear under continuous ones. NeurIPS 2023.

12. [12]

    Will we run out of data? Limits of LLM scaling based on human-generated data(opens in a new tab)

    Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., Hobbhahn, M., 2024

    Epoch AI estimate of about 300T tokens of usable public human text (90% interval 100T to 1,000T), fully used between 2026 and 2032 on current trends. ICML 2024.

13. [13]

    Scaling Data-Constrained Language Models(opens in a new tab)

    Muennighoff, N., Rush, A. M., Barak, B., Le Scao, T., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., Raffel, C., 2023

    Up to 4 epochs of repeated data is almost as good as fresh data; beyond that the value of repetition decays. NeurIPS 2023.

14. [14]

    AI models collapse when trained on recursively generated data(opens in a new tab)

    Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., Gal, Y., 2024

    Nature 631, 755–759. Training each generation on the previous generation’s output makes the tails of the distribution disappear.

15. [15]

    Learning to reason with LLMs(opens in a new tab)

    OpenAI, 2024

    The o1 announcement: performance improves smoothly with both more reinforcement-learning compute and more time spent thinking at test time.

16. [16]

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters(opens in a new tab)

    Snell, C., Lee, J., Xu, K., Kumar, A., 2024

    Allocating test-time compute per prompt beats best-of-N by more than 4× in efficiency and, on suitable problems, lets a small model beat a 14× larger one.