Deriving neural scaling laws from the statistics of natural language

Francesco Cagnetta, Allan Raventós, Surya Ganguli, Matthieu Wyart · 2026

Read Deriving neural scaling laws from the statistics of natural language by Francesco Cagnetta, Allan Raventós, Surya Ganguli, Matthieu Wyart (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-07488.txt.

Description

Neural scaling laws guide how much data and compute to throw at models, but no theory predicted their exponents for real language data. Francesco Cagnetta and three co-authors supply the first such theory for data-limited scaling, built from two statistical properties of natural language alone.

The decay of token correlations with separation, plus the decay of next-token entropy with context length, feed a formula with no tunable values for scaling exponents. That formula matches tests from training GPT-style models. Machine learning theorists get a first-principles account of why scaling curves bend as they do, grounded in language statistics rather than fitted curves.

Catalog record

CREATORFrancesco Cagnetta, Allan Raventós, Surya Ganguli, Matthieu Wyart
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-07488

Browse the full library · About the Tower of Babel