NeuralGrok: Accelerate Grokking by Neural Gradient Transformation

Xinyu Zhou, Simin Fan, Martin Jaggi, Jie Fu * · 2025

Read NeuralGrok: Accelerate Grokking by Neural Gradient Transformation by Xinyu Zhou, Simin Fan, Martin Jaggi, Jie Fu * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17243.txt.

Description

Xinyu Zhou and co-authors tackle grokking, a training puzzle in which a model masters a task long after it has memorized the training data, in a 2025 paper. Their method, NeuralGrok, learns how to reshape the model's gradients during training. An extra small module adjusts each part of the gradient by how much it helps true learning.

A two level training scheme guides that module. In tests on arithmetic tasks, models using the method generalized much sooner, above all on hard problems. The method also kept training steadier and steadily lowered model complexity. Standard weight decay, by contrast, sometimes caused instability. The authors add a new complexity measure to explain why the reshaping helps.

Catalog record

CREATORXinyu Zhou, Simin Fan, Martin Jaggi, Jie Fu *
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-17243

Browse the full library · About the Tower of Babel