NeuralGrok: Accelerate Grokking by Neural Gradient Transformation
Xinyu Zhou, Simin Fan, Martin Jaggi, Jie Fu * · 2025
Read NeuralGrok: Accelerate Grokking by Neural Gradient Transformation by Xinyu Zhou, Simin Fan, Martin Jaggi, Jie Fu * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17243.txt.
Description
Xinyu Zhou and co-authors tackle grokking, a training puzzle in which a model masters a task long after it has memorized the training data, in a 2025 paper. Their method, NeuralGrok, learns how to reshape the model's gradients during training. An extra small module adjusts each part of the gradient by how much it helps true learning.
A two level training scheme guides that module. In tests on arithmetic tasks, models using the method generalized much sooner, above all on hard problems. The method also kept training steadier and steadily lowered model complexity. Standard weight decay, by contrast, sometimes caused instability. The authors add a new complexity measure to explain why the reshaping helps.
Catalog record
| CREATOR | Xinyu Zhou, Simin Fan, Martin Jaggi, Jie Fu * |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17243