Late-Stage Generalization Collapse in Grokking: Detecting anti-grokking with WeightWatcher
Hari K. Prakash, Charles H. Martin · 2026
Read Late-Stage Generalization Collapse in Grokking: Detecting anti-grokking with WeightWatcher by Hari K. Prakash, Charles H. Martin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02859.txt.
Description
What happens if training runs far longer than usual? Hari K Prakash and 1 co-author explore that question in a 2026 paper on grokking. Grokking is when a network memorizes first and only later learns to generalize. The authors report a third phase they call anti-grokking.
In two classic setups, test accuracy first rose, then collapsed back to chance. Training accuracy stayed perfect throughout. A tool called WeightWatcher flagged the collapse from weight patterns alone, without seeing any data. Other common measures missed it. The authors link the failure to unusual eigenvalue patterns they call correlation traps. Similar patterns appeared in some large language models.
Catalog record
| CREATOR | Hari K. Prakash, Charles H. Martin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02859