Deriving neural scaling laws from the statistics of natural language
Francesco Cagnetta, Allan Raventós, Surya Ganguli, Matthieu Wyart · 2026
Read Deriving neural scaling laws from the statistics of natural language by Francesco Cagnetta, Allan Raventós, Surya Ganguli, Matthieu Wyart (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-07488.txt.
Description
Neural scaling laws guide how much data and compute to throw at models, but no theory predicted their exponents for real language data. Francesco Cagnetta and three co-authors supply the first such theory for data-limited scaling, built from two statistical properties of natural language alone.
The decay of token correlations with separation, plus the decay of next-token entropy with context length, feed a formula with no tunable values for scaling exponents. That formula matches tests from training GPT-style models. Machine learning theorists get a first-principles account of why scaling curves bend as they do, grounded in language statistics rather than fitted curves.
Catalog record
| CREATOR | Francesco Cagnetta, Allan Raventós, Surya Ganguli, Matthieu Wyart |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-07488