A Stochastic–Geometric Theory of Scaling Laws in Grokking
Róisín Luo, Christian Gagné, Jonas Ngnawé, Ihsan Ullah, Karyn Morrissey · 2026
Read A Stochastic–Geometric Theory of Scaling Laws in Grokking by Róisín Luo, Christian Gagné, Jonas Ngnawé, Ihsan Ullah, Karyn Morrissey (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30388.txt.
Introduction
Róisín Luo, Christian Gagné, Jonas Ngnawé, Ihsan Ullah, and Karyn Morrissey explain why neural networks sometimes generalize long after memorizing their training data. That delayed jump is called grokking. Models fit training data early, then test well much later. The mechanism behind the delay has stayed murky. The authors locate it in the geometry of solution space. Adam's dynamics with weight shrinkage create that geometry.
Reachable solutions form a shell-and-core arrangement, they argue. Random starting points sit on a thin outer spherical shell. A memorization shell encloses the core within. Generalization solutions occupy the core. Stopping-time theory analyzes when trajectories escape the memorization shell. The escape moment is the grokking transition time. Scaling laws follow for learning rate and batch size. The weight-decay scaling is derived as well. Experiments validate the predicted laws across tasks. The formulas also recover results from earlier literature.
Catalog record
| CREATOR | Róisín Luo, Christian Gagné, Jonas Ngnawé, Ihsan Ullah, Karyn Morrissey |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 164 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30388
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Stochastic–Geometric Theory of Scaling Laws in Grokking Róisín Luo, Christian Gagné, Jonas Ngnawé, Ihsan Ullah, Karyn Morrissey Abstract Delayed generalization ( i.e. grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell–core topological configuration of the reachable solution space induced by Adam’s optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model’s parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and ℓ 2 \ell_{2} regularization coefficient, which are further validated through experiments and shown to recover results from prior literature. 1 Introduction Neural networks trained on noise-free, highly structured learning tasks have been observed to exhibit an epiphany phenomenon known as delayed generalization, or grokking ( 19 ) . In these settings, models rapidly achieve near-zero training loss, often early in training, yet fail to generalize for an extended period before abruptly transitioning to strong test performance. This behavior has been reported across a range of tasks with exact underlying structure ( 6 ) , such as modular arithmetic ( 19 ; 12 ; 22 ) , algorithmic reasoning ( 17 ) , and group-theoretic learning ( 20 ; 18 ) . Moreover, prior work has shown that the onset of grokking is highly sensitive to hyperparameter choices ( 22 ) , including the fraction of training data, learning rate, batch size, and regularization coefficient ( 19 ; 12 ) . Despite extensive empirical investigation, however, its underlying mechanism remains poorly understood. (a) Weight-Shrinkage Grokking (b) Trajectories w/ S_{5} (c) Trajectories w/ \mathbb{Z}_{127} Figure 1: Optimization-Induced Shell–Core Topology and Empirical Evidence. Left ( 1(a) ): grokking dynamics under weight-shrinkage regularization induce a shell–core topology in parameter space, where initialization points \theta_{0}\sim\mathcal{N}(0,\sigma^{2}I_{p}) concentrate on a hyperspherical shell \Theta with radius \rho_{\Theta}\approx\sigma\sqrt{p} and thickness \sigma/\sqrt{2} , enclosing the memorization shell M\setminus G and the generalization core G ; the radii \rho_{G} , \rho_{M} , and \rho_{\Theta} are determined by the task and optimization configuration. Middle–right ( 1(b) – 1(c) ): empirical evidence on S_{5} and \mathbb{Z}_{127} using isometric manifold learning ( e.g. , MDS) to visualize ten optimization parameter trajectories, where \theta_{0} , \theta_{\tau_{M}} , and \theta_{\tau_{G}} denote initialization, first memorization, and first generalization states. The projections show that Adam-reachable M and G are approximately symmetric embedded in ambient space \mathbb{R}^{p} . Empirical findings consistently suggest that training dynamics in the grokking regime exhibit three-stage behavior ( 19 ; 17 ; 9 ) . In the initial stage, the model rapidly interpolates the training set and reaches the memorization regime. In the second stage, the trajectory remains near the memorization solutions for an extended period before escaping and eventually reaching the generalization regime. The length of this extended period scales with the task, learning rate, batch size, and regularization coefficient. In the third stage, drift and diffusion equilibrate in the optimization-induced dynamics, and the trajectory settles within the generalization solution region centered at its minimizer. Under the regime with \ell_{2} weight-shrinkage regularization, these observations imply the existence of two distinct classes of solutions: memorization solutions , which only interpolate the training data and does not generalize, and generalization solutions , which additionally achieve low loss with respect to the underlying data distribution. Their topological configuration is thus induced by optimization dynamics, and characterized into three stages: the early stage , the late stage , and the equilibrium , as shown in Figure 1(a) . The early-stage dynamics are driven by deterministic drift from large gradients, which dominates the optimization trajectory and rapidly drives it toward the memorization manifold. At this stage, the gradient fluctuations are anisotropic. The late-stage dynamics are driven more slowly by decaying but nonzero gradients, together with isotropic gradient fluctuations, which guide the optimization trajectory across the trivial memorization solutions. Once the trajectory reaches the generalization set, the drift and diffusion in the optimization dynamics equilibrate and confine the trajectory within the generalization manifold, referred to as the equilibrium . Figure 1(b) – 1(c) provide empirical evidence consistent with this optimization-induced topological configuration by visualizing optimization trajectories via isometric manifold learning. This optimizer-induced topological configuration, as illustrated in Figure 1(a) , gives rise to grokking and forms a shell–core configuration: initialization solutions concentrate on a thin hyperspherical shell, which encloses a shell of memorization solutions containing a compact core of generalization solutions. Detailed empirical results of Adam–induced shell–core radii are provided in Appendix A.2 . We theoretically analyze the geometry of this topological configuration and derive scaling laws for the manifold radii and the solution transition time from memorization to generalization through a stopping-time analysis ( 4 ) of stochastic differential equations (SDEs) ( 23 ) . The remainder of the paper is organized as follows. Section 3 presents initialization concentration, and Section 4.1 establishes the joint-state SDE for Adam ( 8 ) designed to analyze grokking dynamics. Building on this formulation, Section 4.2 derives the scaling law for the memorization-manifold radius. Under the late-stage reduction observations of grokking, Section 4.3 derives a closed-form preconditioned radius SDE for the evolution of manifold radii. Sections 4.4 – 4.5 derive the scaling laws for the generalization-manifold radius and the solution transition time from memorization to generalization. Finally, Section 5 validates the theory through experiments on modular arithmetic and group-theoretic learning tasks, and by comparison with the grokking literature. The contributions are summarized below: 1. Optimization-Induced Shell–Core Topology. We introduce an optimizer-induced shell–core prior for grokking, where initialization concentrates on a thin hyperspherical shell enclosing memorization solutions, which in turn contain a compact core of generalization solutions. 2. Stochastic–Geometric Characterization. Leveraging the topological prior, we formulate grokking as a stochastic transition problem and characterize the geometry by the first stopping time at which stochastic gradient flows escape the memorization manifold and reach the generalization manifold. 3. Empirical, Literature, and Symbolic-Algebra Validation. We validate our theory through experiments on group-theoretic learning and modular arithmetic tasks, and by comparison with the literature, recovering known scaling laws for the learning rate, batch size, and regularization coefficient. We additionally machine-check all closed-form and asymptotic results appearing in the proofs by a symbolic algebra system (SAS) with Julia. 2 Notations Data and Network. Let D_{\mathrm{tr}}:=\{(x_{i},y_{i})\}_{i=1}^{n} denote a finite training set of size n , where each pair (x_{i},y_{i})\in\mathcal{X}\times\mathcal{Y} is drawn i.i.d. from the empirical distribution \widehat{P}_{\mathcal{XY}} , with underlying true distribution P_{\mathcal{XY}} . Let f_{\theta}:\mathcal{X}\to\mathcal{Y} be a model parameterized by \theta\in\mathbb{R}^{p} . Throughout, we use X_{t} to denote the time-indexed value of a variable X at time t ; for example, \theta_{t} denotes the parameter at time t . Instance, Batch, Empirical, and True Loss. Let \ell_{f}(x,y;\theta) denote