reading › scaling › deep dive

Decoding Scaling Laws for Neural Language Models

25 min read

In this blog, we will learn about Scaling Laws for Neural Language Models - the famous OpenAI paper by Jared Kaplan, Sam McCandlish, and team (January 2020) - by decoding it piece by piece: what a power law is, how loss depends on model size, data, and compute, why model shape barely matters, how to balance model size against data, and how to spend a fixed compute budget optimally.

This is the paper that changed how the entire AI industry thinks. Before this paper, making models bigger was a gamble. After this paper, it became an engineering calculation. GPT-3 was built directly on the insights of this paper. When we hear “scaling laws”, it sounds like heavy math. But do not worry. If we break it down into its individual parts, every single piece is simple. Our goal is to decode this paper so clearly that by the end, we will be able to explain it to anyone.

We will cover the following:

Let’s get started.

The Big Picture

Before we go into the details, let’s understand the big picture.

The paper asks one simple question: What actually makes a language model better?

Is it the architecture? The depth? The number of attention heads? The width of the layers? The answer the paper found is surprising: almost none of that matters much. What matters is scale, and scale has exactly three levers:

The paper trained hundreds of Transformer language models - from tiny models with just 768 parameters all the way up to 1.5 billion parameters - and found that the test loss follows clean, predictable power laws in each of these three factors, with trends spanning more than seven orders of magnitude.

Let’s be very clear about what “seven orders of magnitude” means, because we will use this phrase a lot. One order of magnitude means one multiplication by 10. So:

So when we say the trends span seven orders of magnitude, we mean: the same simple rule held for a tiny model AND for a model ten million times bigger. To feel the size of that gap - it is the gap between having 1 rupee and having 1 crore rupees, or between 1 second and about 4 months. The same law working across that entire range is extremely rare in machine learning. No deviation. No surprises. Just smooth, straight lines on a log-log plot (we will decode what a log-log plot is in a moment).

In simple words: The paper turned “let’s try a bigger model and hope” into “we can predict the loss of a model before we train it.”

And the single most famous conclusion of the paper is this: given a fixed compute budget, we should train a very large model and stop training early, rather than training a small model to convergence. Big models are the compute-efficient choice. This one insight is what gave the industry the confidence to build GPT-3.

The Three Levers: N, D, and C

The whole paper is written in terms of three quantities, so we must understand each one precisely.

N: Model Size (Non-Embedding Parameters)

N is the number of parameters in the model, excluding the token embedding matrix and positional embeddings.

Why exclude embeddings? This is one of the small but brilliant decisions in the paper. When the authors plotted loss against total parameters (including embeddings), models with different depths landed on different curves - the trend was messy. But when they plotted loss against non-embedding parameters, all the models collapsed onto a single clean curve. The embedding matrix is basically a lookup table - it does not do the “thinking”. The real capacity of the model lives in the attention and feed-forward layers.

For a standard Transformer, the paper gives a simple formula:

N ≈ 12 × n_layer × d_model²

where n_layer is the number of layers and d_model is the width of the residual stream. For example, a model with 12 layers and d_model = 768 has N ≈ 12 × 12 × 768² ≈ 85 million non-embedding parameters. That is roughly GPT-2 small.

D: Dataset Size in Tokens

D is simply the number of training tokens. The paper trained on WebText2 - a scrape of outbound Reddit links with at least 3 karma - containing about 23 billion tokens after BPE tokenization with a vocabulary of 50,257.

C: Compute

C is the total amount of computation used during training, measured in PF-days. One PF-day means running at 1 PetaFLOP (10¹⁵ floating point operations per second) for one full day, which equals 8.64 × 10¹⁹ operations.

Here comes one of the most reused formulas in all of AI. The paper estimates the training compute as:

C ≈ 6 × N × D

where D = B × S is the total tokens processed (batch size × number of steps). Where does the 6 come from? For every parameter and every token:

So 2 + 4 = 6 FLOPs per parameter per token. If we ever see someone estimate GPT-3’s training compute as 6 × 175B × 300B ≈ 3.15 × 10²³ FLOPs, this little formula from this paper is where it comes from.

One more detail: the loss L in the paper is the cross-entropy loss measured in nats (natural log instead of log base 2), averaged over a 1024-token context. A loss of 3.4 nats means the model’s perplexity is e^3.4 ≈ 30 - as if the model were choosing uniformly among about 30 equally likely tokens at each position.

What Is a Power Law?

The entire paper rests on one mathematical shape, so let’s decode it properly.

A power law is a relationship of the form:

L(X) = (X_c / X)^α

where X is the thing we scale (N, D, or C), and X_c and α are constants fit from data. The exponent α tells us how fast the loss falls as we scale up.

The magic property of a power law is what it looks like on a log-log plot (both axes logarithmic): it becomes a perfectly straight line with slope −α. This is why the plots in the paper are so striking - hundreds of models, seven orders of magnitude of compute, and the points just line up on a ruler-straight line.

There is an even simpler way to internalize a power law: every 10x increase in X cuts the loss by the same fixed percentage. Not the same fixed amount - the same fixed percentage. That means:

This is also the sobering part: it means diminishing returns. Each constant improvement in loss costs 10x more than the last one. Scaling always works, but it never gets cheaper.

The Three Basic Scaling Laws

Now we are ready for the core results. The paper found that when performance is bottlenecked by only one factor (and the other two are large enough not to matter), the test loss follows a clean power law in that factor.

Law 1 - Model size. For models trained to convergence on enough data:

L(N) = (N_c / N)^0.076, with N_c ≈ 8.8 × 10¹³

Law 2 - Dataset size. For large models trained on limited data with early stopping:

L(D) = (D_c / D)^0.095, with D_c ≈ 5.4 × 10¹³

Law 3 - Compute. For training with a limited compute budget, using an optimally-sized model and a sufficiently small batch size:

L(C_min) = (C_c / C_min)^0.050, with C_c ≈ 3.1 × 10⁸ PF-days

Do not let the constants scare us. N_c, D_c, and C_c are just scale factors - the paper explicitly says their exact values depend on the tokenizer and vocabulary, so they have no fundamental meaning. The exponents are the real content of the paper.

Let’s make these exponents concrete with real numbers:

We can even use Law 1 as a calculator. What loss will a 1B-parameter model converge to? L = (8.8 × 10¹³ / 10⁹)^0.076 = (88,000)^0.076 ≈ 2.37 nats. And a 100M-parameter model? ≈ 2.83 nats. The paper let researchers make exactly these kinds of predictions - and the predictions held.

These improvements look small per step, but they compound relentlessly across orders of magnitude, and here is the remarkable part - the trends showed no sign of bending across the entire range the paper could measure. The loss must flatten out eventually (language has non-zero entropy - there is irreducible uncertainty in what the next token is), but within the measured range, scaling just kept working.

Why Model Shape Barely Matters

This is the finding that surprised people the most in 2020.

The paper took models with the same total non-embedding parameter count N and varied their shape - deep and narrow, shallow and wide, more attention heads, fewer attention heads, bigger feed-forward dimension, smaller feed-forward dimension. The result:

The loss varied by only a few percent across a huge range of shapes.

The most dramatic example: the aspect ratio (d_model / n_layer) could vary by a factor of 40 with barely any impact. A shallow-wide model with 6 layers and d_model = 4288 reached a loss within 3% of the 48-layer, d_model = 1600 shape used by GPT-2.

Put differently: if a suboptimal shape costs us 1% in loss, we can compensate with just 22% more compute. Shape is a rounding error; scale is the whole game.

Why would depth barely matter? The paper points to an interesting hypothesis: deep networks may effectively behave like ensembles of shallower networks, so once there is “enough” depth, adding more just redistributes the same capacity.

The practical consequence is huge: stop obsessing over architecture search, and start obsessing over scale. Within reason, a Transformer is a Transformer. What separates a weak model from a strong one is N, D, and C - not clever tweaks to the head count.

The paper also compared Transformers to LSTMs at equal parameter counts. LSTMs kept up for tokens early in the context, but plateaued after fewer than 100 tokens, while Transformers kept improving through the entire context. This is the quantitative version of why the Transformer won: it actually uses long context.

The Overfitting Law: Balancing N and D

So far each law had one bottleneck. But the real question in practice is: if I make my model bigger, how much more data do I need?

The paper answers this with a single beautiful equation that combines model size and data size:

L(N, D) = [ (N_c/N)^(α_N/α_D) + D_c/D ]^α_D

Let’s decode why this form makes sense. The authors chose it using three principles:

The equation is the simplest form satisfying all three - and it fits the data almost perfectly.

Now the punchline. From this equation, the degree of overfitting depends on just one combination of N and D: the ratio N^0.74 / D. Keep that ratio constant, and overfitting stays constant. This gives us the paper’s famous data-scaling rule:

Every time we increase the model size by 8x, we only need about 5x more data to avoid a penalty.

Let’s verify: 8^0.74 ≈ 4.7 ≈ 5. So data can grow sub-linearly in model size. Bigger models do not just memorize more - they generalize better per token of data.

The paper even gives a concrete recipe. To train to within the run-to-run noise level (about 0.02 nats) of the converged loss without overfitting, we need roughly:

D ≳ 5,000 × N^0.74 tokens

Quick sanity check with the paper’s own setup: for a 1B-parameter model, D ≳ 5,000 × (10⁹)^0.74 ≈ 22 billion tokens - which is almost exactly the size of WebText2. This is why the paper says models up to about 1B parameters could be trained on WebText2 with minimal overfitting, but their largest models were starting to touch the limit.

One more subtle result from this section: generalization is not a separate skill. The paper evaluated WebText2-trained models on completely different distributions - Wikipedia, Books, Common Crawl - and found that the loss on those datasets improves in lockstep with the training-distribution loss, offset by a roughly constant penalty. Transfer depends almost only on how good the model is in-distribution, not on architecture depth or training duration. Make the model better on its training distribution, and it gets better everywhere.

Training Curves: The Law of Training Time

The next question: how does loss fall during training? The paper found that after an initial transient period, training curves also follow a predictable law:

L(N, S) = (N_c/N)^0.076 + (S_c / S_min)^0.76

Let’s decode the two terms:

S_min here is the number of steps adjusted as if training at an ideal (small) batch size - we will decode that adjustment in the next section.

The remarkable finding is universality: the shape of the training curve (S_c ≈ 2,100 and α_S ≈ 0.76) is roughly independent of model size. Every model, from thousands to billions of parameters, descends along the same-shaped curve toward its own floor.

This has a very practical superpower: we can extrapolate. Train a model for a short while, fit the early part of its curve, and predict the loss it would reach if we trained much longer. Loss becomes forecastable - and modern LLM labs run exactly this playbook: fit scaling laws on many small runs, then predict the big run before committing millions of dollars of compute to it.

Critical Batch Size

Before we can talk about spending compute optimally, we need one more concept: the critical batch size.

When we train with gradient descent, each batch gives us a noisy estimate of the true gradient. Batch size controls a trade-off:

The crossover point is the critical batch size B_crit. Below it, doubling the batch size roughly halves the number of steps needed - a free trade of time for parallelism. Above it, doubling the batch size barely saves any steps - pure waste. Training exactly at B_crit is the balanced point: it takes about 2× the minimum possible steps and processes about 2× the minimum possible data. A reasonable compromise on both fronts.

The paper’s finding about B_crit is elegant: it does not depend on the model size at all - only on the current loss:

B_crit(L) ≈ 2 × 10⁸ / L^4.8 tokens

As the loss decreases, the critical batch size grows - roughly doubling for every 13% decrease in loss - reaching about 1-2 million tokens at the loss levels of the paper’s largest models. Intuitively: early in training the gradient signal is strong and obvious, so small batches are enough. Late in training, the remaining improvements are subtle, gradients become dominated by noise, and we need bigger batches to see the signal through it.

This is also great news for training big models: as models get better, more and more of the work can be done in parallel (bigger batches across more GPUs) rather than serially (more steps one after another).

The Most Important Result: How to Spend a Fixed Compute Budget

Now everything comes together. This is the section that changed the industry.

Suppose we are given a fixed compute budget C - say, 1,000 GPUs for one month. We must choose:

Since C ≈ 6NBS is fixed, these choices trade off against each other. A bigger model means fewer steps within the same budget. So what is optimal?

The old intuition said: pick a modest model and train it all the way to convergence - surely we should “finish” training. The paper proved this intuition wrong. Using L(N, S_min), the authors solved for the loss-minimizing allocation mathematically (take the derivative with respect to N at fixed C, set it to zero) and confirmed it empirically. The answer:

N ∝ C^0.73, B ∝ C^0.24, S ∝ C^0.03

Let’s translate these exponents into plain words. If our compute budget grows 10x:

Almost the entire budget increase goes into model size. The paper illustrates this with a billion-fold (10⁹x) increase in compute: model size grows more than 1,000,000x, batch size grows about 100x, and serial steps grow less than 10x.

And here is the most counterintuitive part. The math in Appendix B shows that compute-optimal training should stop when the loss is still about α_N/α_S ≈ 10% above what the model would reach at convergence. In simple words:

Compute-efficient training means training a very large model and stopping significantly short of convergence.

Why does this work? Think of it this way. The last stretch of convergence is brutally expensive - the training-time term (S_c/S)^0.76 has diminishing returns like everything else, so squeezing out the final 10% of a small model’s potential costs an enormous number of steps. That same compute, given to a much larger model, buys a lower floor - and even a partially-trained big model with its lower floor beats a fully-converged small model. Convergence is inefficient. It is better to be a big model at 90% of its potential than a small model at 100% of its potential.

Note the difference between two questions that sound similar but have different answers:

Compute-optimal training is deliberately data-light and model-heavy. That was the paper’s boldest claim - the paper literally says “Big models may be more important than big data.” GPT-3 (175B parameters, only 300B training tokens) was this philosophy executed at full scale, and it worked spectacularly.

Sample Efficiency: Big Models Learn Faster

A beautiful corollary falls out of the compute-optimal result: larger models are more sample-efficient. They reach any given loss with fewer optimization steps and fewer data points than small models.

The paper shows this vividly: to reach the same test loss, a 1-billion-parameter model needs far fewer tokens than a 1-million-parameter model. The bigger model extracts more learning from every single token it sees.

This flips a common mental model. We tend to think of big models as data-hungry beasts. In terms of total data at convergence, sure. But in terms of data needed to reach a given quality level, big models are the frugal ones. Each gradient update in a big model updates a richer internal representation, so each token teaches it more.

This is also why the compute-optimal recipe works at all: it leans on exactly this property - use the biggest model we can afford, precisely because it wrings the most out of every token and every step.

The Contradiction at the End of the Paper

The paper ends with something rare and wonderful: the authors point out that their own laws predict their own breakdown.

Here is the tension. Follow compute-optimal training and two facts collide:

Data requirements for avoiding overfitting grow at C^0.54, but compute-optimal training only feeds the model data at C^0.26-0.27. At some scale, compute-optimal training must start overfitting - even if it never reuses a single token! Setting the two loss trajectories equal, the paper computes the intersection point:

C ≈ 10⁴ PF-days, N ≈ 10¹² parameters, D ≈ 10¹² tokens, L ≈ 1.7 nats/token

(with an honest caveat that these values are uncertain by an order of magnitude in either direction). At or before this point - around a trillion parameters and a trillion tokens - the scaling laws must break down.

The authors then offer a fascinating conjecture: maybe this intersection point is not just where the math breaks, but where Transformers reach maximal performance on language - the point where the model has extracted all the reliable information available in natural language data. In that reading, L* ≈ 1.7 nats/token would be a rough estimate of the entropy of natural language itself - the irreducible uncertainty that no model, however large, can remove.

Whether or not the conjecture holds, notice what the paper is doing here - deriving falsifiable predictions about trillion-parameter models from experiments that never exceeded 1.5 billion parameters. That is what the paper means when it compares its laws to the ideal gas law: macroscopic regularities (loss vs. scale) that hold regardless of microscopic details (architecture specifics), still waiting for a “statistical mechanics” - a deeper theory - to explain them.

What Came After: GPT-3 and the Chinchilla Correction

To be a true ninja of this paper, we must also know its sequel - because one of its conclusions was later corrected.

GPT-3 (2020): The direct child of this paper. OpenAI scaled to 175B parameters trained on 300B tokens - a huge model on relatively modest data, exactly the Kaplan recipe. The loss landed on the predicted trend line, and in-context learning emerged as the surprise bonus. This validated the core promise: scaling is predictable, and smooth loss improvements can hide qualitative jumps in capability - the paper itself anticipated this with the phrase “more is different.”

Chinchilla (DeepMind, 2022): Researchers redid the compute-optimal analysis more carefully and found a different answer: for a fixed compute budget, N and D should scale equally - N ∝ C^0.5 and D ∝ C^0.5 - not N ∝ C^0.73. The practical rule of thumb became roughly 20 training tokens per parameter. By this math, GPT-3-era models were significantly undertrained: DeepMind’s 70B-parameter Chinchilla, trained on 1.4 trillion tokens, outperformed the 280B-parameter Gopher trained on 300B tokens - a 4x smaller model winning on the same compute budget.

Why did the original paper get a different exponent? Two main subtleties:

Here is the important part: Chinchilla corrected the exponents, not the framework. Loss still follows power laws in N, D, and C. Shape still barely matters. L(N, D) still takes the same two-term form. Training curves still extrapolate. The entire methodology of “fit scaling laws on small runs, predict the big run” - which every serious LLM lab uses today - comes straight from this paper. Modern models (LLaMA-style training on trillions of tokens, often far beyond even the Chinchilla-optimal point, to make inference cheap) are the third iteration of a conversation that this paper started.

Why This Paper Matters

Let’s step back and appreciate what this paper actually gave us:

Predictability: Loss became a quantity we can forecast before spending millions on a training run. Every frontier lab today fits scaling laws on small models before committing to a big one.

The end of architecture obsession: Within the Transformer family, shape is a few-percent effect while scale is an orders-of-magnitude effect. This refocused the entire field’s energy onto scale, data, and systems engineering.

The compute-optimal mindset: The question “how should I spend a fixed budget across model size, batch size, and steps?” did not really exist as a quantitative discipline before this paper. Now it is the first question every training team asks.

The 6ND formula: C ≈ 6ND is used daily across the industry for back-of-the-envelope compute estimates.

The scaling hypothesis: The paper’s boldest implication - that predictable loss improvements translate into growing capabilities - is the bet behind GPT-3, GPT-4, and essentially every frontier model since. The entire modern LLM era is downstream of taking these straight lines seriously.

I personally believe this is one of the most consequential AI papers of the decade - not because it introduced a new architecture (it introduced none), but because it replaced intuition with measurement. It took the same models everyone already had and revealed the simple laws governing them. It makes our life easier.

Quick Summary

Let’s recap what we have decoded:

This is the paper that turned “bigger is better” from a hunch into a law - and quietly set the direction of the entire modern AI era.

That’s it for now.