What Is “Loss” for a Language Model?

Scaling Laws, Made Simple · Part 5 · 6 min read

Every scaling law in the paper predicts one single number, called the loss. If we do not understand loss, the laws are just symbols. So in this blog we make loss completely concrete.

What a language model actually does

A language model plays one game, billions of times: guess the next token.

A token is a small chunk of text - usually a word or a piece of a word. The sentence “The cat sat on the mat” might be split into tokens like The, cat, sat, on, the, mat.

The game: show the model the beginning of a text, hide the next token, and ask it to guess. But here is the key detail - the model does not answer with a single word. It answers with probabilities for every possible token.

Show it “The cat sat on the ___” and it replies something like:

” mat” → 40%
” floor” → 20%
” sofa” → 10%
” roof” → 5%
…tiny probabilities for thousands of other tokens

Scoring a guess

Now we reveal the answer. Say the true next token was ” mat”. How good was the model’s guess?

Simple idea: look at how much probability the model gave to the correct answer. It gave ” mat” 40%, i.e. 0.4. Is 0.4 good? Better than a wild guess, worse than certainty. We need to turn this probability into a score. The score used everywhere is:

loss = −ln(p)

where p is the probability the model gave to the correct token, and ln is the natural logarithm. Don’t worry about why this exact formula for now - just look at how it behaves:

Model was certain and right (p = 1): loss = −ln(1) = 0. Perfect score.
Model gave the right answer 40% (p = 0.4): loss ≈ 0.92.
Model gave it only 5% (p = 0.05): loss ≈ 3.0.
Model gave it 0.1% (p = 0.001): loss ≈ 6.9. Terrible.

So the loss is a penalty for being surprised by the correct answer. Confident and right → tiny loss. Right answer buried deep in the probabilities → big loss. Lower is always better.

The model’s overall loss is just this penalty averaged over huge amounts of text, token after token. When the paper says “a model with loss 3.0”, it means: on average, across the test text, the model’s surprise per token was 3.0.

What is a “nat”?

The paper measures loss in nats. This is nothing deep: a nat is just the unit you get when the formula uses the natural logarithm (ln). If we had used log base 2 instead, the unit would be called a bit. Same idea, different ruler - like measuring in kilometers vs miles. The paper uses nats, so we will too.

The most useful trick: turning loss into “number of options”

Raw loss values feel abstract. Is 3.4 good? Here is a beautiful trick that makes any loss value feel real:

Compute e^loss. That is roughly the number of equally likely options the model is “choosing between” at each step.

This number is called the perplexity - literally, how perplexed the model is:

loss = 3.4 → e^3.4 ≈ 30. The model is as uncertain as someone picking from 30 equally likely words.
loss = 2.3 → e^2.3 ≈ 10. Now it is down to 10 options. Much sharper.
loss = 0.7 → e^0.7 ≈ 2. Basically a coin flip between two candidates.

So when the scaling laws say a bigger model took the loss from 3.4 to 2.3, it means the model went from hesitating between ~30 words to hesitating between ~10. You can feel that difference in the quality of generated text.

Why loss can never reach zero

Could a perfect model reach loss = 0? No - and this matters for the paper’s ending.

Loss 0 would mean predicting every next token with 100% certainty. But language is genuinely uncertain. After “My favorite color is ___”, even a perfect model cannot know if I will say “blue” or “green”. The information simply is not in the text so far.

So there is a floor - an amount of loss that no model, however huge, can ever remove. This floor is called the entropy of language: the irreducible uncertainty of text. Nobody knows its exact value. The scaling laws paper makes a fascinating guess about it, which we will meet in the last blog.

For now, the picture to hold: every model’s loss lives somewhere between the entropy floor (unreachable perfection) and total confusion. Scaling pushes it down toward the floor, never through it.

Why we obsess over small differences

A warning before we move on: in the coming blogs, we will care intensely about differences like “loss dropped by 5%”. This can seem ridiculous. It is not, for two reasons.

First, because of the perplexity trick: small nat differences are real jumps in how sharp the model’s guesses are. Second - and this was one of the great surprises of the GPT era - models with slightly lower loss suddenly do things the others cannot: write working code, translate, answer questions. Smooth tiny gains in loss hide jumps in ability. The paper’s phrase for this is “more is different”.

Quick recap

A language model predicts a probability for every possible next token.
Loss = −ln(probability given to the correct token): a penalty for being surprised. Lower is better.
A nat is just the unit of this penalty when using the natural log.
e^loss ≈ how many equally likely options the model is choosing between (perplexity). Loss 3.4 ≈ 30 options; loss 2.3 ≈ 10.
Loss can never reach 0 - language has irreducible uncertainty (entropy).
Small loss differences hide big ability differences.

Next: the three levers we can pull to push the loss down - model size, data, and compute - and how each is measured.