When Data Falls Short: Grokking Below the Critical Threshold
Vaibhav Singh, Eugene Belilovsky, Rahaf Aljundi · 2025
Read When Data Falls Short: Grokking Below the Critical Threshold by Vaibhav Singh, Eugene Belilovsky, Rahaf Aljundi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04760.txt.
Description
Vaibhav Singh and two colleagues study grokking when training data falls below the critical size. Grokking is delayed generalization: a model overfits first, then suddenly grasps the pattern. Below a data threshold, that jump never becomes visible.
Knowledge distillation from an already-grokked model can trigger the jump on a new distribution. It works even when the new data sits below the usual threshold. The team also trains on two distributions jointly to compare plain practice. Their findings matter for deployed models that must adapt to shifted data while examples stay scarce.
Catalog record
| CREATOR | Vaibhav Singh, Eugene Belilovsky, Rahaf Aljundi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-04760