The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology

Alper YILDIRIM · 2026

Read The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology by Alper YILDIRIM (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05228.txt.

Introduction

Alper Yıldırım tests grokking, late generalization of Transformers trained on cyclic modular addition, through architectural change rather than post-hoc analysis. He spots two shape factors that prolong memorization: unbounded form size and data-driven attention routing. A bounded spherical topology, with residual stream and fixed temperature, cuts grokking onset time more than twentyfold with no weight decay. Uniform attention ablation succeeds fully across all seeds with no grokking delay. A non-commutative permutation control shows no such speedup, pointing to design priors aligned with task symmetries.

Catalog record

CREATORAlper YILDIRIM
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51686
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05228

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology Alper YILDIRIM Abstract Mechanistic interpretability typically relies on post-hoc analysis of trained networks. We instead adopt an interventional approach: testing hypotheses a priori by modifying architectural topology to observe training dynamics. We study grokking—delayed generalization in Transformers trained on cyclic modular addition ( ℤ p \mathbb{Z}_{p} )—investigating if specific architectural degrees of freedom prolong the memorization phase. We identify two independent structural factors in standard Transformers: unbounded representational magnitude and data-dependent attention routing. First, we introduce a fully bounded spherical topology enforcing L 2 L_{2} normalization throughout the residual stream and an unembedding matrix with a fixed temperature scale. This removes magnitude-based degrees of freedom and qualitatively alters the training dynamics: models generalize immediately, bypassing the prolonged memorization phase characteristic of grokking in standard Transformer baselines, both with and without weight decay. Second, a Uniform Attention Ablation overrides data-dependent query-key routing with a uniform distribution, reducing the attention layer to a Continuous Bag-of-Words (CBOW) aggregator. Despite removing adaptive routing, these models achieve 100 % 100\% generalization across all seeds and bypass the grokking delay entirely. To evaluate whether this acceleration is a task-specific geometric alignment rather than a generic optimization stabilizer, we use non-commutative S 5 S_{5} permutation composition as a negative control. Enforcing spherical constraints on S 5 S_{5} does not accelerate generalization. This suggests eliminating the memorization phase depends strongly on aligning architectural priors with the task’s intrinsic symmetries. Together, these findings provide interventional evidence that architectural degrees of freedom substantially influence grokking, suggesting a predictive structural perspective on training dynamics. Keywords: Grokking, Mechanistic Interpretability, Inductive Bias, Architectural Intervention, Transformer, Representational Geometry 1 Introduction Neural networks often exhibit complex and unintuitive learning dynamics. One of the most striking examples is grokking: a delayed phase transition in which a model achieves near-perfect training accuracy while test accuracy remains low, followed by a sudden transition to full generalization after prolonged optimization. Originally observed in small models trained on algorithmic tasks, grokking has become a central phenomenon for studying the relationship between optimization, generalization, and implicit bias. In parallel, mechanistic interpretability aims to uncover the internal algorithms learned by neural networks. For example, prior work has shown that Transformers trained on modular addition eventually implement structured representations related to discrete Fourier features ( 10 ) . However, much of this work is post-hoc: researchers analyze models only after grokking has occurred, inferring mechanisms from frozen weights. In this work, we take a complementary interventional approach. Rather than analyzing a trained model, we modify architectural structure before training to test whether specific representational degrees of freedom influence grokking dynamics. Our central hypothesis is that standard Transformer architectures possess degrees of freedom that exceed the minimal symmetry requirements of commutative, periodic tasks such as modular addition. As demonstrated by 18 , this excess capacity allows standard networks to converge on diverse, sometimes messy, piecewise solutions (the “Pizza” algorithm) rather than the elegant, continuous Fourier solution (the “Clock” algorithm). We hypothesize that these additional degrees of freedom enable memorization-heavy solution pathways that delay the emergence of invariant representations ( 18 ; 7 ; 12 ) . We isolate and study two independent architectural factors: Path A: The Magnitude Degree of Freedom. In standard Transformers, information can be encoded in both the direction and magnitude of residual stream vectors. Prior work has suggested that unconstrained magnitude growth can influence training dynamics in grokking settings ( 12 ) . To examine the role of this degree of freedom, we introduce a Fully Bounded Spherical Topology that enforces strict L_{2} normalization throughout the residual stream and applies a normalized unembedding matrix with a fixed temperature scale. This intervention removes the model’s ability to encode information in vector norm and constrains logits to a bounded cosine geometry. Empirically, this modification substantially accelerates generalization, reducing grokking onset from \sim 54,160 epochs in standard LayerNorm baselines to \sim 2,480 epochs without weight decay. Path B: The Routing Degree of Freedom. Transformers also possess flexible, data-dependent attention routing through learned query-key interactions. However, recent theoretical results demonstrate that modular addition can be implemented using uniform token aggregation ( 5 ) . Motivated by this, we introduce a Uniform Attention Ablation that overrides learned query-key scores with a fixed uniform distribution, eliminating adaptive routing and reducing the attention mechanism to a Continuous Bag-of-Words (CBOW) aggregator. Despite removing routing flexibility, LayerNorm models with uniform attention achieve 100% peak test accuracy across 10 independent seeds and do not exhibit a prolonged grokking delay. Summary of Contributions. Through controlled architectural interventions, we show that (1) constraining representational magnitude and (2) removing data-dependent routing each independently and substantially reduce grokking delay in modular addition. These results demonstrate that grokking dynamics are highly sensitive to architectural degrees of freedom. Furthermore, using non-commutative S_{5} permutation composition as a negative control, we observe that symmetry-aligned architectural constraints do not eliminate delayed generalization in tasks lacking commutativity, suggesting task-specific effects. Overall, our findings support a predictive, geometry-aware approach to studying training dynamics, in which architectural priors are aligned with task symmetry to probe the mechanisms underlying delayed generalization. 2 Related Work 2.1 Grokking as Geometric Reorganization Recent empirical studies suggest that the delayed generalization characteristic of grokking has been interpreted as a geometric phenomenon rather than a purely optimization-driven one. Observational work by 17 demonstrated that delayed generalization coincides with a rapid decrease in the effective radius and dimensionality of the representation manifold. They found that changes in this representational geometry are more tightly linked to the onset of generalization than standard optimization metrics. Similarly, 7 characterized grokking as an asynchronous “construct-then-compress” dynamic. By tracking Jacobian transformations across local neighborhoods, they showed that standard networks spend the majority of their memorization phase building disjoint representations, only generalizing once these features are compressed into a globally coherent algorithmic geometry. This geometric bottleneck extends to higher-order reasoning tasks as well. 8 demonstrated that the emergence of analogical functors in Transformers relies heavily on structural alignment in the embedding space, which they quantified via the minimization of Dirichlet energy. Crucially, they noted that this alignment is highly sensitive to the presence of weight decay, reinforcing the necessity of regularization in forcing networks toward structured, low-energy geometric states. Our work builds directly upon this observational foundation. While prior studies passively measure the slow emergence of geometric compression ( 17 ; 7 ) , we treat geometry as an independent variable. By architecturally enforcing an L_{2} spherical constraint, we structurally restrict the representational manifold from initialization, potentially reducing the need for an extended unconstrained construction phase. This geometric bottleneck is intimately tied to the optimization objective itself. Recent theoretical work by 9 mathematically shows that the emergence of structured algebraic circuits—such as Fourier features for modular addition and irreducible representations for finite groups—is driven by the implicit margin-maximization bias of gradient descent under cross-entropy loss. In standard unconstrained networks, the prolonged grokking delay corresponds to the time required for this implicit bias to slowly sculpt the optimal mathematical geometry out of an unstructured parameter space. Our interventional methodology complements this analytical framework: by architecturally enforcing an L_{2} spherical constraint, we explicitly restrict the representational manifold to the continuous geometry required by the target

Read the complete text