Scaling Laws, Made Simple — a 10-part mini-course
A ground-up explanation of the “Scaling Laws for Neural Language Models” paper (Kaplan, McCandlish, et al., 2020), written for students. Nothing is assumed: powers of ten, power laws, log-log plots, and loss are all built from zero before the paper’s actual results appear. Each part is a ~5 minute read.
Read in order — every part uses only what came before it.
| # | Blog | You learn |
|---|---|---|
| 1 | Big Numbers and Powers of Ten | What 10^7 means, what an “order of magnitude” is, why 2 orders = 100x not 2x |
| 2 | What Does “A Trend Across Seven Orders of Magnitude” Mean? | What a trend is, why rules normally break with scale, why the paper’s rules not breaking was historic |
| 3 | What Is a Power Law? | y = x^a, the “every 10x buys the same percentage” signature, tiny and negative exponents |
| 4 | What Is a Log-Log Plot? | What a logarithm is in plain words, why power laws become straight lines, how to read the slope |
| 5 | What Is “Loss” for a Language Model? | Next-token guessing, loss as a surprise penalty, nats, perplexity (“loss 3.4 ≈ choosing among 30 words”) |
| 6 | The Three Levers: Model Size, Data, and Compute | N, D, C precisely defined, and the famous C = 6 × N × D formula decoded |
| 7 | The Three Scaling Laws, With Real Numbers | The actual laws; 10x model → −16% loss, 10x data → −20%, 10x compute → −11%; using a law as a calculator |
| 8 | The Shape of the Model Barely Matters | Depth vs width is a few-percent effect; scale is the whole game; Transformer vs LSTM is the real divide |
| 9 | Balancing Model and Data: The Overfitting Law | Overfitting via the photographic-memory student; 8x model needs only ~5x data (N^0.74) |
| 10 | Spending a Compute Budget: Train Big, Stop Early | The compute-optimal recipe, why it built GPT-3, and the 2022 Chinchilla correction |
After finishing, the full deep-dive on the paper is in posts/scaling-laws.md.