Grokking Explained: A Statistical Phenomenon
Breno W. Carvalho, Artur S. d’Avila Garcez, Luís C. Lamb, Emílio Vital Brazil · 2025
Read Grokking Explained: A Statistical Phenomenon by Breno W. Carvalho, Artur S. d’Avila Garcez, Luís C. Lamb, Emílio Vital Brazil (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01774.txt.
Description
Breno Carvalho and co-authors argue that grokking, a strange delay in machine learning, is a statistical effect rather than a mystery of training. In grokking, a model memorizes its training data and seems stuck, then suddenly, much later, starts answering new cases correctly. Earlier explanations blamed heavy regularization and sparse data.
The authors built two synthetic datasets to isolate the cause. By skewing how often subgroups appear in training versus testing, they created a distribution shift and reproduced grokking on demand. Sparse sampling turned out to be merely a handy way to create that shift, not the cause itself. When classes relate by symmetry, the model's skill on well-sampled groups can transfer to starved ones, explaining the sudden jump. Grokking also appeared with dense data and little tuning.
Catalog record
| CREATOR | Breno W. Carvalho, Artur S. d’Avila Garcez, Luís C. Lamb, Emílio Vital Brazil |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-01774