What Does “A Trend Across Seven Orders of Magnitude” Mean?

The scaling laws paper makes one claim again and again: its trends hold across seven orders of magnitude. In this blog, we will unpack that sentence word by word, until it is completely clear.

From the previous blog, we already know that seven orders of magnitude means a range where the biggest thing is ten million times the smallest thing. Now let’s understand the other word: trend.

What is a trend?

A trend is a simple rule that connects two quantities, and keeps holding as they change.

Here is an everyday example. Suppose a taxi charges 20 rupees per kilometer.

• 1 km costs 20 rupees
• 2 km costs 40 rupees
• 5 km costs 100 rupees

The rule “cost = 20 × distance” is a trend. Given any distance, we can predict the cost without asking anyone. That is the power of a trend: prediction.

The important question: over what range does the rule hold?

Every rule works over some range. The real question is how far we can stretch it before it breaks.

The taxi rule works from 1 km to maybe 100 km. But does it work for 10,000,000 km? Of course not - no taxi drives to the Moon. The rule breaks when we stretch it too far.

This is normal. Almost every rule in life breaks when the scale changes a lot:

• A recipe for 4 people works for 8 people. Multiply it by 10,000 and nothing works - you cannot stir a swimming pool of dough with a spoon.
• An ant’s thin legs work perfectly at ant size. Scale an ant up to elephant size and its legs would snap - which is why elephants have thick legs. The “rule” for ant legs breaks at large scale.
• A medicine dose that works for a 5 kg baby cannot just be multiplied for a 500 kg patient.

So our default expectation should be: rules hold for a while, then break.

Now, the paper’s claim

The scaling laws paper trained language models of many different sizes and measured how good each one was (we will define “how good” precisely in a later blog - for now, just think “error score, lower is better”).

The models ranged from roughly a thousand parameters to over a billion parameters. The training compute ranged over an even wider span. Between the smallest and the largest experiments there were seven or more 10x jumps.

And here is the claim: one single, simple rule predicted the error score across that entire range.

The same little formula that predicted the tiny model’s score also predicted the score of a model millions of times bigger. No adjustment. No special cases. The rule just kept working, 10x jump after 10x jump, seven times in a row.

Why this is a big deal

Let’s connect it back to our examples. This is like discovering that:

• the taxi price formula works from 1 km all the way to the Moon, or
• the 4-person recipe scales perfectly to a wedding of 40 million people.

That never happens in everyday life. When it does happen in science, it usually means we have found something deep - a real law, not a coincidence. Gravity is like this: the same formula works for an apple and for the Moon.

The paper’s authors are saying: language model performance behaves like that. It follows a law.

What this bought us: prediction

Remember, a trend means prediction. If the rule held for every size we tested, from tiny to huge, we start to trust it for sizes we have NOT tested yet.

That is exactly what happened in history. This paper came out in January 2020. A few months later, its authors’ lab built GPT-3, a model about 100 times bigger than anything in the paper - and its error score landed almost exactly where the trend line predicted. They spent millions of dollars on a training run because a straight line told them it was worth it.

That is what “trends spanning seven orders of magnitude” means: a rule so stable across scale that you can bet millions on where it points next.

Quick recap

• A trend is a rule connecting two quantities, and its value is prediction.
• Normal rules break when scale changes a lot (recipes, ant legs, taxi fares).
• The paper found rules that did NOT break across seven 10x jumps - a range of ten million times.
• Rules that survive scale changes like that are called laws, and we can extrapolate them - which is exactly how GPT-3 was justified.

Next, we will meet the specific kind of rule these laws follow: the power law.