What Is a Power Law?

The scaling laws in the paper are all one kind of rule: the power law. In this blog, we will understand exactly what that is, using only small numbers.

Rules connect x and y

A rule takes an input x and produces an output y. Different rules have different shapes:

• y = x + 5 (add something)
• y = 3 × x (multiply by a constant)
• y = x² (multiply x by itself)

That last one is our first power law.

The definition

A power law is a rule where y equals x raised to some fixed power:

y = x^a

The fixed number a is called the exponent, and the exponent is the personality of the rule. Some examples:

• a = 2: y = x². If x = 3, y = 9. If x = 10, y = 100.
• a = 3: y = x³. If x = 10, y = 1,000.
• a = 0.5: y = √x (the square root - yes, powers can be fractions). If x = 100, y = 10.

The signature property: every 10x in x changes y by the same factor

Here is the one property that makes power laws special. Let’s discover it with y = x².

• x = 1 → y = 1
• x = 10 → y = 100
• x = 100 → y = 10,000
• x = 1,000 → y = 1,000,000

Look at what y does each time x jumps 10x: y goes 1 → 100 → 10,000 → 1,000,000. Every single 10x jump in x multiplies y by exactly 100. Not “roughly”. Exactly. And it does not matter where we start - from 1 or from a million, a 10x jump in x always multiplies y by 100.

This is the signature of every power law:

Multiply x by 10, and y gets multiplied by the same fixed factor - every time, at every scale.

That fixed factor is 10^a. For y = x², the factor is 10² = 100.

This is exactly why power laws can survive seven orders of magnitude. The rule treats every 10x jump identically. It has no built-in “natural size” where it starts or stops working. Compare that with rules that DO have a natural scale - a recipe has a natural size (a family), ant legs have a natural size (an ant). Power laws have none. They are scale-free.

Fractional and small exponents: slow-motion power laws

Exponents do not have to be big numbers like 2 or 3. The exponents in the scaling laws paper are tiny fractions like 0.076. What does a tiny exponent do?

A tiny exponent means y responds very sluggishly to x. Take y = x^0.076:

• Multiply x by 10 → y is multiplied by 10^0.076 ≈ 1.19 (just 19% more)
• Multiply x by 100 → y is multiplied by about 1.42
• Multiply x by 10,000,000 → y is multiplied by only about 3.4

Ten million times more input, and the output didn’t even grow 4x. Tiny exponents make very lazy curves. Remember this laziness - it is why making language models better is so expensive.

Negative exponents: things that go DOWN

One more twist, and it is the most important one for us. The exponent can be negative:

y = x^(-a) which means y = 1 / x^a

A negative exponent means y goes down as x goes up. The signature property still holds, just downward: every 10x in x divides y by the same fixed factor.

Take y = x^(-0.076), which is the actual shape of the paper’s model-size law:

• Multiply x by 10 → y is multiplied by 10^(-0.076) ≈ 0.84. So y drops by 16%.
• Multiply x by 10 again → y drops by 16% again.
• And again → another 16%.

In the paper, x is the model size and y is the error score. So the law says: every time we make the model 10x bigger, the error drops by the same 16%. From a thousand parameters to a billion, each 10x jump buys the same percentage discount on error.

The same percentage - which is both good and cruel

Notice what “the same percentage each 10x” implies.

The good part: improvement never stops. There is no wall. Want a better model? Scale up. It always works.

The cruel part: each 10x costs 10 times more than the previous 10x. Going from 1M to 10M parameters is cheap. Going from 100M to 1B is a hundred times more expensive - and buys the same 16%. Every equal slice of improvement costs ten times more than the last slice.

This one sentence explains the last five years of AI: scaling always works, and scaling never gets cheaper.

Quick recap

• A power law is y = x^a. The exponent a is fixed and defines the rule.
• Signature property: every 10x in x multiplies y by the same fixed factor (10^a), at every scale. That is why power laws are scale-free and can span seven orders of magnitude.
• Tiny exponents (like 0.076) mean very slow change: 10x in x → only ~19% change in y.
• Negative exponents mean y falls as x grows: for x^(-0.076), every 10x in x cuts y by ~16%.
• Same percentage per 10x = improvement never stops, but each step costs 10x more.

Next: how do we see a power law in experimental data? For that we need one beautiful tool - the log-log plot.