Late-Stage Generalization Collapse in Grokking: Detecting anti-grokking with WeightWatcher

Hari K. Prakash, Charles H. Martin · 2026

Read Late-Stage Generalization Collapse in Grokking: Detecting anti-grokking with WeightWatcher by Hari K. Prakash, Charles H. Martin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02859.txt.

Description

What happens if training runs far longer than usual? Hari K Prakash and 1 co-author explore that question in a 2026 paper on grokking. Grokking is when a network memorizes first and only later learns to generalize. The authors report a third phase they call anti-grokking.

In two classic setups, test accuracy first rose, then collapsed back to chance. Training accuracy stayed perfect throughout. A tool called WeightWatcher flagged the collapse from weight patterns alone, without seeing any data. Other common measures missed it. The authors link the failure to unusual eigenvalue patterns they call correlation traps. Similar patterns appeared in some large language models.

Catalog record

CREATORHari K. Prakash, Charles H. Martin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02859

Browse the full library · About the Tower of Babel