The Three Scaling Laws, With Real Numbers

Scaling Laws, Made Simple · Part 7 · 6 min read

We now have every ingredient: orders of magnitude, power laws, log-log plots, loss, and the three levers N, D, C. In this blog, the actual laws - finally - with real numbers.

The setup: isolate one lever at a time

To measure the effect of one lever cleanly, the other two must not be in the way. So the paper measured three situations:

Vary model size N, with plenty of data and training time → how does loss depend on N alone?
Vary data D, with a plenty-big model → how does loss depend on D alone?
Vary compute C, spending it as well as possible → how does loss depend on C alone?

In each case, they trained many models, plotted loss against the lever on a log-log plot, and saw… straight lines. Power laws. Here they are.

The three laws

Law 1 - Model size: L(N) = (8.8 × 10^13 / N)^0.076

Law 2 - Data size: L(D) = (5.4 × 10^13 / D)^0.095

Law 3 - Compute: L(C) = (3.1 × 10^8 / C)^0.050

Each law reads the same way: divide a big constant by your lever, raise to a tiny power, and out comes the predicted loss.

Two kinds of numbers appear here, and they matter very differently:

The big constants (8.8 × 10^13 and friends) are just calibration - they depend on the tokenizer and dataset details. The paper itself says they carry no deep meaning. Ignore them.
The exponents (0.076, 0.095, 0.050) are the real discovery. They say how fast loss falls per 10x of each lever - they are the slopes of the straight lines on the log-log plots.

Translating the exponents into plain language

Using our power-law skill: a 10x jump in the lever multiplies the loss by 10^(-exponent). Let’s compute each one:

Model size: 10^(-0.076) ≈ 0.84 → 10x more parameters cuts loss by ~16%
Data: 10^(-0.095) ≈ 0.80 → 10x more tokens cuts loss by ~20%
Compute: 10^(-0.050) ≈ 0.89 → 10x more compute (well spent) cuts loss by ~11%

And for a friendlier step: doubling the model size multiplies loss by 2^(-0.076) ≈ 0.95 - a 5% cut per doubling.

Feel how lazy these exponents are. Ten TIMES the compute for an 11% improvement. This is the “cruel” side of power laws from Blog 3, now with real values: progress is guaranteed, and each equal slice of progress costs 10x more than the last.

Using a law as a calculator

Let’s actually predict something. Law 1: what loss does a 1-billion-parameter model converge to?

L = (8.8 × 10^13 / 10^9)^0.076 = (88,000)^0.076 ≈ 2.37 nats

Perplexity check from Blog 5: e^2.37 ≈ 10.7 - the model hesitates between ~11 options per token. And a 100-million-parameter model? (880,000)^0.076 ≈ 2.83 nats ≈ 17 options.

Notice what just happened: we predicted the final quality of a multi-week, many-GPU training run with one line of arithmetic. That is the entire point of scaling laws. When the paper’s authors later checked such predictions against real giant runs, the dots landed on the line.

What the laws promised (and what they didn’t)

Three promises stood out from the straight lines:

1. No wall in sight. The dots stayed on the line up to the largest affordable experiments. Scaling had not run out at any tested size - want better, pay more, get better, predictably.

2. Every lever obeys a law. All three levers show the same power-law shape. This regularity is what made people call them “laws”, like laws of physics.

3. The exponents differ - so the levers are NOT equally valuable. Look again: data’s exponent (0.095) beats model size’s (0.076), and both beat naive compute (0.050). And there is a subtlety: these single-lever laws each assume the OTHER levers are unlimited. In real life you have one budget C to split across N and D. Which split wins is a genuinely new question - the single laws cannot answer it alone.

That question - the best split of a fixed budget - has a shocking answer, and it is where we go after one quick detour: first we must see what surprisingly DOESN’T matter for the loss at all.

Quick recap

Three laws, one shape: loss = (constant / lever)^tiny-exponent. The constants are calibration; the exponents are the discovery.
Per 10x: model size buys ~16% lower loss, data ~20%, optimally-spent compute ~11%. Doubling the model buys ~5%.
The laws work as calculators: a 1B model → ~2.37 nats ≈ hesitating between ~11 words.
No bend in the lines: scaling had no visible wall.
Single-lever laws don’t answer the real question - how to split one budget across N and D.