The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold

Tiberiu Mușat · 2025

Read The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold by Tiberiu Mușat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01938.txt.

Introduction

Tiberiu Musat explains grokking, the strange training pattern where a neural network memorizes its training data perfectly, sits at zero loss for a long time, and only then suddenly generalizes. Earlier work tied the delay to representation learning driven by weight decay, but the dynamics stayed murky.

Musat frames post-memorization training as constrained optimization: once the loss is zero, gradient descent effectively minimizes the weight norm while staying on the zero-loss manifold, the surface of perfect fits. He proves this formally in the limit of infinitesimal learning rates and weight decay. An approximation then decouples a subset of parameters from the rest, yielding a closed-form expression for the post-memorization dynamics of the first layer in a two-layer network. Simulations using the predicted gradients reproduce both the delayed generalization and the representation learning that define grokking.

Catalog record

CREATORTiberiu Mușat
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01938

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold Tiberiu Mușat Abstract Grokking is a puzzling phenomenon in neural networks where full generalization occurs only after a substantial delay following the complete memorization of the training data. Previous research has linked this delayed generalization to representation learning driven by weight decay, but the precise underlying dynamics remain elusive. In this paper, we argue that post-memorization learning can be understood through the lens of constrained optimization: gradient descent effectively minimizes the weight norm on the zero-loss manifold. We formally prove this in the limit of infinitesimally small learning rates and weight decay coefficients. To further dissect this regime, we introduce an approximation that decouples the learning dynamics of a subset of parameters from the rest of the network. Applying this framework, we derive a closed-form expression for the post-memorization dynamics of the first layer in a two-layer network. Experiments confirm that simulating the training process using our predicted gradients reproduces both the delayed generalization and representation learning characteristic of grokking. 1 Introduction Neural networks have achieved great success, but their mechanisms remain far from being fully understood. Doshi-Velez & Kim 2017 argue that understanding the inner workings of neural networks is crucial for the development of AI systems with increased safety and reliability. Moreover, understanding the learning dynamics of neural networks could also help us improve their performance and efficiency: it is easier to design better learning algorithms when we understand the limitations of the existing ones. Furthermore, insights into artificial neural networks may also enhance our understanding of biological neural networks due to their fundamental similarities ( Sucholutsky et al. 2023 ; Kohoutová et al. 2020 ) . This work aims to clarify the learning dynamics underlying a particularly puzzling phenomenon termed grokking . Under specific training conditions, neural networks achieve generalization on the test data only after an extended period following the complete memorization of the training data. This behavior was first observed in synthetic problems such as modular addition ( Power et al. 2022 ) , but was later shown to also happen in real-world datasets ( Liu et al. 2022b ; Humayun et al. 2024 ) . In the specific problem of modular addition, interpretability research has revealed that neural networks achieve generalization by placing the embedding vectors on a circle ( Gromov 2023 ; Zhong et al. 2024 ) . The circular structure of the embedding layer enables the network to perform a symmetric algorithm that generalizes perfectly to unseen data. It is currently known that circular representations emerge gradually during the post-memorization phase ( Nanda et al. 2023 ) and that weight decay is mainly responsible for driving the delayed generalization ( Liu et al. 2022b ) , but the precise dynamics remain unclear. Moreover, the role of the embedding layer in the modular addition task is a striking example that that generalization in neural networks often hinges on representation learning within specific network components. In such cases, it can be highly beneficial to simplify the complex learning dynamics of a deep network by isolating and analyzing only the component of interest. Recent work has already begun to explore such approximations, also referred to as effective theories ( van Rossem & Saxe 2024 ; Liu et al. 2022a ; Musat 2024 ; Mehta et al. ) . 2 Our Contributions In this work, we aim to answer the following questions: Q1. What is the exact role of weight decay in the post-memorization learning dynamics? Q2. Can we isolate the dynamics of the embedding layer from the rest of the network? We answer Q1 in Section 4 by proving that, after memorization is achieved, the learning dynamics approximately follow the minimization of the weight norm, constrained to the zero-loss level set. We answer Q2 in Section 5 by proposing an approximation for the isolated learning dynamics of any parameter subset as the minimization of a specific cost function. We then combine these insights in Section 6 to study the post-memorization learning dynamics of a two-layer network, deriving a closed-form expression for the cost function of the first layer (the embedding layer). Finally, in Section 7 , we validate our theoretical insights on a modular addition task, showing that our approximations reproduce the delayed generalization and circular representations characteristic of grokking. 3 Intuitions from Toy Models Before presenting our theoretical findings, we give an intuition for our results by discussing and visualizing how they relate to a few highly simplified models. Figure 1: A two-parameter linear model \hat{y}=w_{1}x_{1}+w_{2}x_{2} groks simple addition when trained with just one sample: x_{1}=x_{2}=1,\ y=2 (corresponding to 1+1=2 ). We plot three training runs with different weight decay coefficients \lambda . After quickly achieving (almost) zero loss, learning is entirely driven by the minimization of the weight norm. 3.1 Grokking Addition We begin by discussing how a linear model can grok addition from just 1+1=2 . We use a single-layer linear model with two inputs and two weights: \hat{y}=w_{1}x_{1}+w_{2}x_{2} . We train this model with mean-squared error loss using just one sample: x_{1}=x_{2}=1,\ y=2 . We use three different values of weight decay, \lambda\in\{\,0.01,0.1,0.2\,\} . We initialize our model with w_{1}=-1 and w_{2}=1 . We aim to show that our model can learn to perform standard addition, despite being trained with a single sample. Test accuracy is measured on a set of 100 randomly generated samples, where x_{1} and x_{2} are sampled from a normal distribution, and y=x_{1}+x_{2} . We show our results in Figure 1 . We can see that our model reproduces grokking: training loss becomes very low after just a 10 steps, while test loss takes a few hundred steps. Additionally, the model achieves lower loss with smaller \lambda , but takes longer to generalize. Interpretation. From Figure 1 (a, b), we can see that learning follows two phases. In the first phase, driven by loss minimization, the model achieves a low loss by learning (w_{1},w_{2})\approx(0,2) . In the second phase, learning is entirely driven by weight decay. The model follows norm minimization, while maintaining (almost) zero loss, eventually reaching (w_{1},w_{2})\approx(1,1) . Note that, for smaller \lambda , the model remains closer to the zero-loss line, but generalization also takes longer. 3.2 Norm Minimization While the previous example illustrates our theoretical framework, it does not capture its full generality. It is unsurprising that applying weight decay encourages a reduction in norm. However, the central claim of this paper is significantly stronger: we argue that the learning dynamics under weight decay do not merely follow some norm-decreasing direction, but rather evolve along the direction that optimally minimizes the norm, subject to remaining on the zero-loss manifold . Another way to view this is the following: once the model achieves perfect memorization, learning effectively follows gradient descent on the weight norm, constrained to the zero-loss manifold. To offer a better intuition, we show how a linear model can grok three-number addition. We train a three-parameter linear model \hat{y}\,=\,w_{1}x_{1}+w_{2}x_{2}+w_{3}x_{3} with just one sample: x_{1}=x_{2}=x_{3}=1,\ y=3 . As in the previous section, this model exhibits grokking: after quickly achieving zero training loss, the model slowly reaches the generalizing solution (w_{1},w_{2},w_{3})\approx(1,1,1) . We perform four training runs with different initializations and visualize the resulting

Read the complete text