When Data Falls Short: Grokking Below the Critical Threshold

Vaibhav Singh, Eugene Belilovsky, Rahaf Aljundi · 2025

Read When Data Falls Short: Grokking Below the Critical Threshold by Vaibhav Singh, Eugene Belilovsky, Rahaf Aljundi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04760.txt.

Description

Vaibhav Singh and two colleagues study grokking when training data falls below the critical size. Grokking is delayed generalization: a model overfits first, then suddenly grasps the pattern. Below a data threshold, that jump never becomes visible.

Knowledge distillation from an already-grokked model can trigger the jump on a new distribution. It works even when the new data sits below the usual threshold. The team also trains on two distributions jointly to compare plain practice. Their findings matter for deployed models that must adapt to shifted data while examples stay scarce.

Catalog record

CREATORVaibhav Singh, Eugene Belilovsky, Rahaf Aljundi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04760

Browse the full library · About the Tower of Babel