Balancing Model and Data: The Overfitting Law

We know loss follows a law in model size N and a law in data D. But in real life we change both together, and one danger appears the moment N and D go out of balance: overfitting. This blog decodes it, and the paper’s precise rule for staying safe.

What is overfitting?

Imagine a student preparing for an exam using a book of 100 practice questions.

If the student is reasonable, they solve the questions, learn the underlying methods, and can then solve new, unseen questions. They learned the pattern.

But imagine a student with a photographic memory and only those same 100 questions. The laziest path to a perfect practice score is to memorize all 100 answers - without understanding anything. Practice score: perfect. Real exam with fresh questions: disaster.

That is overfitting: when the learner has too much memory for too little material, it memorizes instead of understanding. Memorization looks great on the training material and fails on anything new.

Language models face exactly this. A model with billions of knobs (huge memory) trained on too few tokens will start storing the training text itself instead of learning the patterns of language. Its loss on training text keeps falling - lying to us - while its loss on fresh text stalls or rises. And fresh text is all we care about.

The cure is balance: a bigger brain needs more study material. But how much more, exactly? Twice the brain, twice the data? Nobody knew - until the paper measured it.

The combined law

The paper trained models across a whole grid: every combination of several model sizes and several dataset sizes. Then it found one formula that fits the entire grid - the loss for ANY pair (N, D):

L(N, D) = (size term) + (data term)

Conceptually, the loss has two ingredients that add up:

• a size term that shrinks as N grows - “how much the brain limits you”
• a data term that shrinks as D grows - “how much the study material limits you”

Each term is a little power law of its own. And the formula behaves exactly as common sense demands: with infinite data, only the size term is left (the pure N law from before); with an infinite model, only the data term is left (the pure D law). Whichever ingredient is larger is your bottleneck - the thing worth paying to fix. Pour money into a bigger model while the data term dominates, and the loss barely moves; you are upgrading the wrong lever. Balanced growth means growing N and D so neither term dominates.

The punchline: 8x the model, only 5x the data

From this formula the paper derives the safe-balance rule, and it is delightfully specific. Overfitting stays constant as long as one quantity stays constant:

N^0.74 / D

In words: data does NOT need to keep up with the model one-for-one. It only needs to grow like the model size raised to the power 0.74 - noticeably slower. Two ways to feel it:

• Make the model 8x bigger → you need only about 5x more data (because 8^0.74 ≈ 4.7).
• Make the model 1,000x bigger → about 165x more data. Not 1,000x.

There is even a concrete recipe with all constants filled in: D ≈ 5,000 × N^0.74 tokens keeps overfitting negligible. Sanity check: for a 1-billion-parameter model this gives about 22 billion tokens - and indeed the paper trained models up to about that size on its 23-billion-token dataset without trouble. The formula matches practice.

Why this rule is good news

The sub-linear power (0.74 instead of 1) says something profound: bigger models are better learners, not just bigger memorizers.

If big models just memorized proportionally more, data would need to keep up one-for-one. Instead, each 10x of model size needs only ~5.5x of data - the big model extracts more understanding per token than the small one. Bigger brains study more efficiently.

It is also strategically good news. In 2020, tokens were plentiful (the internet) but compute was the constraint. “Data can grow slower than the model” meant the road to giant models was open without needing impossible amounts of text - a green light for GPT-3.

One caution before the finale

Careful: this rule answers “given this model size, how much data avoids overfitting?” That is NOT the same question as “given a fixed compute budget, what is the best model size and data size?” - the question we set up in Blog 6 with C = 6ND.

The final blog answers that budget question, and its answer - train a shockingly large model and stop early - plus the 2022 correction to it, is the perfect end to the story.

Quick recap

• Overfitting = too much memory for too little material → memorizing, not learning. Looks great on training text, fails on fresh text.
• The combined law: loss = size term + data term; the bigger term is your bottleneck, and that is the lever worth paying for.
• Overfitting is controlled by N^0.74/D: an 8x model needs only ~5x data; the safe recipe is D ≈ 5,000 × N^0.74 tokens.
• The 0.74 means big models learn more per token - they are better learners, not just bigger memorizers.
• Balancing against overfitting ≠ optimally spending a budget. That final question is next.