Spending a Compute Budget: Train Big, Stop Early (and the Chinchilla Twist)

Scaling Laws, Made Simple · Part 10 · 7 min read

Everything in this series was preparation for one question. It is the question every AI lab faces before every big run:

We have a fixed compute budget C. How should we spend it?

Remember the constraint from Blog 6: C ≈ 6 × N × D. Compute equals model size times data (times 6). A fixed budget therefore forces a trade-off: a bigger model means fewer training tokens, and more tokens mean a smaller model. We must choose.

The old intuition

Before 2020, the standard practice felt obvious: pick a modest model and train it to convergence - keep training until the loss completely stops improving. Squeeze the full value out of the model you paid to build. Stopping early felt like buying a textbook and reading half of it.

The paper tested this intuition against the data. It failed spectacularly.

The paper’s answer

Using the measured laws, the paper computed the loss-minimizing split of any budget. Here is the recipe, stated for a 10x budget increase:

When the budget grows 10x:

make the model about 5.4x bigger
show it only about 2x more data
and the number of training steps grows by only ~7% - essentially not at all

Read that again. Almost the entire extra budget goes into the model, and hardly any into more data or longer training. Definitely not the balanced split most people would guess.

And the strangest part: this recipe means the big model stops training far short of convergence - while its loss is still clearly falling (about 10% above where it would settle if trained forever). We buy an enormous brain and deliberately leave it partly educated.

Why on earth is that optimal?

The intuition unlocks with two facts we already own.

Fact 1 (Blog 9): big models learn more per token. Every token shown to a big model buys more loss reduction than the same token shown to a small one. If tokens-processed is what compute buys, spend each token on the most efficient learner available - the biggest one you can afford.

Fact 2: the last stretch of convergence is brutally expensive. Training progress is itself a lazy power law: early steps help a lot, later steps help less and less. Squeezing out the final few percent of a small model’s potential takes an enormous number of steps - all spent on an inefficient learner.

Put together: instead of paying a fortune for the last drops from a small model, spend that same compute giving a big model its early, cheap, steep improvements. A partly-trained big model beats a fully-trained small model at the same total cost. It is better to be a genius who skimmed the book than an average student who memorized it.

The paper says it in one line: “Big models may be more important than big data.”

This sentence built GPT-3

This conclusion was not academic. The same lab followed the recipe to its logical end: GPT-3 - a model of 175 billion parameters trained on just 300 billion tokens. A colossal brain, a modest reading list, exactly as the recipe ordered. Its loss landed where the scaling laws predicted, and its abilities stunned everyone. The scaling era was on.

The twist: Chinchilla corrects the recipe (2022)

Science checks itself, and here the story has a famous second chapter.

In 2022, researchers at DeepMind redid the budget analysis more carefully and found a bias in the original experiments. Roughly: the original paper trained all its models with settings tuned for long runs, which made short training look worse than it truly is - unfairly penalizing the “train longer” option and tilting the answer toward “just grow the model”.

The corrected recipe, known as Chinchilla:

Split the budget evenly: when compute grows 10x, make the model ~3.2x bigger AND the data ~3.2x bigger. A handy rule of thumb: about 20 training tokens per parameter.

By this math, GPT-3-era giants were undertrained - brains too big for their reading lists. The proof was a model: Chinchilla, 70B parameters trained on 1.4 trillion tokens, beat a 280B model trained on 300B tokens - four times smaller, same compute, better results.

Two things to hold together here:

The recipe changed: from “almost everything into the model” to “split evenly between model and data”. Data came roaring back.
The framework survived untouched: loss still follows power laws; shape still barely matters; the method “fit laws on small runs, predict the big run” is exactly how Chinchilla itself was discovered. The 2022 paper didn’t demolish the 2020 paper - it re-ran the 2020 paper’s own playbook, better.

Today’s models push even further: they train small-ish models on FAR more than 20 tokens per parameter - “over-training” past the compute-optimal point on purpose, because a smaller final model is cheaper to serve to millions of users every day. Training cost is paid once; serving cost is paid forever.

The end of the story the paper foresaw

One last gem. The 2020 paper noticed its own laws pointing at a cliff: extended far enough, its compute recipe would eventually starve models of data, and the trends must break - somewhere around models of ~10^12 parameters. The authors conjectured that near that point, models may have extracted essentially all the reliable patterns in human text, with the loss flattening at the floor we met in Blog 5: the entropy of language - their rough estimate for it was around 1.7 nats per token. Whether that guess holds, nobody knows yet. But it is a fitting close: the paper even predicted the limits of its own predictions.

Quick recap - and the whole series in five lines

Fixed budget C = 6ND forces a choice between model size and data.
The 2020 answer: put almost everything into model size; stop training early. A partly-trained big model beats a fully-trained small one. This built GPT-3.
The 2022 Chinchilla correction: split evenly - ~20 tokens per parameter; GPT-3-era models were undertrained.
The power-law framework survived the correction completely; today labs even over-train small models to save serving costs.

And the whole series:

Huge ranges are counted in 10x jumps (orders of magnitude).
Model quality follows power laws in size, data, and compute - straight lines on log-log plots, holding across seven orders of magnitude.
Loss - the model’s average surprise - is the number these laws predict, and small drops in it hide big jumps in ability.
Shape barely matters; scale is the game; data must grow sub-linearly with the model.
A fixed budget is a math problem now - and its solution, twice refined, is the reason today’s frontier models look the way they do.

That’s it. You can now read “Scaling Laws for Neural Language Models” - and its famous figures - like someone who belongs there.