Grokking and Generalization Collapse: Insights from HTSR theory

Hari K. Prakash, Charles H. Martin · 2025

Read Grokking and Generalization Collapse: Insights from HTSR theory by Hari K. Prakash, Charles H. Martin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04434.txt.

Description

Hari Prakash and Charles Martin study a strange pattern in neural network training in a 2025 paper. Networks can suddenly jump from memorizing to grasping a task long after training scores look perfect. The authors trained a small network on digit images and watched what happened when training ran far longer than usual.

They found a third, late phase in which test scores crash while training scores stay perfect. A weight based measure from their theory flagged all three phases, while rival measures missed the late crash. The late drop came with outlier values that signal the network had overfit its training set. The work gives trainers a way to spot collapse without needing test data.

Catalog record

CREATORHari K. Prakash, Charles H. Martin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04434

Browse the full library · About the Tower of Babel