The Three Levers: Model Size, Data, and Compute

Scaling Laws, Made Simple · Part 6 · 6 min read

The scaling laws paper says the loss of a language model depends on three quantities, and it names them with three letters: N, D, and C. Every equation in the paper is built from these three. So let’s understand each one precisely.

N: the number of parameters (the model’s size)

A neural network is, at its heart, a giant collection of adjustable numbers. Each of these numbers is called a parameter.

Think of a huge mixing desk in a music studio, covered with millions of little knobs. Training a model means turning all those knobs, a tiny bit at a time, until the desk transforms input text into good predictions. N is simply the count of knobs.

GPT-2 small: N ≈ 100 million knobs
GPT-3: N ≈ 175 billion knobs

One technical detail the paper is careful about: it counts only the thinking parameters and excludes the embedding parameters. Embeddings are the model’s lookup table that converts each token into numbers before the real processing starts. The paper found that when you exclude this lookup table from the count, models of all shapes fall onto one clean trend line; include it, and the picture gets messy. So in every scaling law, N means non-embedding parameters.

D: the amount of training data (in tokens)

D is the number of tokens the model sees during training.

We met tokens in the previous blog: small chunks of text. A handy rule of thumb: 1 token ≈ 3/4 of a word, so a 400-word blog post is about 500 tokens.

Scale that up: the paper’s training set was a big scrape of good web pages containing about 23 billion tokens - roughly a library of 50,000 thick books. Modern models train on trillions of tokens - millions of books.

The rule of intuition: N is the size of the brain, D is the amount of study material.

C: the compute (the total amount of arithmetic)

Training a model means doing arithmetic - additions and multiplications - by the trillion. C is the total count of arithmetic operations done during training.

One addition or one multiplication is called a FLOP (floating point operation). C is measured in FLOPs.

Because training runs use astronomically many FLOPs, the paper uses a bigger unit: the PF-day (petaflop-day). One PF-day = doing 10^15 operations per second, non-stop, for a whole day ≈ 8.6 × 10^19 FLOPs. Think of it as “one day of work by a very powerful machine”. Roughly speaking, C is what you pay for: it is GPU-hours, electricity, money.

The famous formula: C = 6 × N × D

Here is the most reused formula from the paper. The three levers are not independent - compute is determined by the other two:

C ≈ 6 × N × D

In words: training compute ≈ 6 FLOPs for every parameter, for every token.

Why 6? Follow one token through one knob of the model:

Forward pass (making the prediction): the token’s value gets multiplied by the knob and added into a running total. That is 2 operations (one multiply, one add) per parameter.
Backward pass (learning from the mistake): after scoring the prediction, the model works backward to figure out how to adjust every knob. This backward work costs about twice the forward work: 4 operations per parameter.

Total: 2 + 4 = 6 operations per parameter per token. Multiply by all N parameters and all D tokens: C = 6ND.

Let’s use it: the GPT-3 back-of-envelope

This formula lets anyone estimate a giant training run on a napkin. GPT-3: N = 175 billion parameters, D = 300 billion tokens.

C = 6 × (1.75 × 10^11) × (3 × 10^11) ≈ 3.15 × 10^23 FLOPs

Divide by 8.6 × 10^19 FLOPs per PF-day: ≈ 3,600 PF-days. About ten years of work for a one-petaflop machine - which is why it ran on thousands of GPUs in parallel for weeks. Engineers do this exact napkin math before any big training run, and it comes straight from this paper.

The real question the paper asks

Now we can state the paper’s central question sharply.

C = 6ND means compute is a budget that gets split between N and D. If we have a fixed budget C, we face a real trade-off:

Big brain, little study? (large N, small D)
Small brain, lots of study? (small N, large D)
Something balanced in between?

Both extremes feel wrong, so something in between must be best - but where exactly? Before this paper, nobody knew. People chose model sizes by folklore. The scaling laws turn this trade-off into a solvable math problem - and the answer (coming in the last two blogs) shocked people.

Quick recap

N = number of adjustable knobs (parameters), excluding the embedding lookup table. The brain size.
D = number of training tokens (1 token ≈ 3/4 of a word). The amount of study material.
C = total arithmetic operations (FLOPs) in training; measured in PF-days. This is what costs money.
C ≈ 6 × N × D: 2 FLOPs forward + 4 FLOPs backward, per parameter, per token.
Fixed compute = a forced trade-off between brain size and study amount. The paper’s core question: what split is best?

Next: the three scaling laws themselves - with real numbers.