The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression

Yongzhong Xu · 2026

Read The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression by Yongzhong Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07380.txt.

Introduction

Xu follows the spectral edge, the top direction of parameter-update grams, through grokking in two sequence tasks. Before grokking, the edge is driven by gradients and does real work. At grokking, gradient and weight decay line up, and the edge turns into a compression axis. It is flat to nudges yet vital to remove, over 4000 times more potent than random directions. Three broad classes emerge, forecast by a gap-flow equation, and nonlinear probes show facts are re-encoded, not lost.

Catalog record

CREATORYongzhong Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28790
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.07380

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression Yongzhong Xu Abstract The spectral edge—the dominant direction of the Gram matrix of parameter updates—has been shown to track phase transitions in neural network training. But what drives the spectral edge, and why does it matter causally? We decompose the spectral edge into its gradient and weight-decay components across two sequence tasks (Dyck-1 balanced parentheses and SCAN compositional generalization) and discover a sharp two-phase lifecycle. Before grokking , the edge is gradient-driven (88–98% gradient energy) and carries task-relevant functional content. At grokking , gradient and weight decay align along the edge—both push the same direction—and the edge transitions to a compression mode (0.2–5% gradient, 95–99.8% weight decay). This alignment is the microscopic signature of the phase transition. The post-grok edge presents a paradox: perturbing along it changes almost nothing (max KL ≈ 0.00005 \approx 0.00005 ; Hessian curvature 0.08 0.08 ), yet removing it is catastrophic ( Δ ​ acc = − 0.58 ± 0.09 \Delta\text{acc}=-0.58\pm 0.09 ; > > 4000 × \times worse than removing random directions). The resolution is geometric: the model’s function depends on the edge’s orientation (locked by the spectral gap) but not on displacement along it (flat curvature). The edge defines a stable axis that weight decay compresses without disrupting. Three universality classes emerge: functional edges (modular arithmetic: edge carries distinct Fourier modes), mixed edges (Dyck: edge retains partial functional content), and compression edges (SCAN: edge is functionally empty, frozen at 8 ∘ rotation). The class is predicted by the balance of gradient driving vs. weight-decay damping in the gap flow equation. Six causal experiments establish the mechanism: grad-WD decomposition, ablation with random controls, ε \varepsilon -sweep perturbation curves, Hessian curvature, nonlinear probes (MLP R 2 = 0.99 R^{2}=0.99 where linear R 2 = 0.86 R^{2}=0.86 ), and a weight-decay intervention showing that removing WD post-grok reverses compression ( R linear 2 : 0.85 → 0.99 R^{2}_{\text{linear}}:0.85\to 0.99 ) while preserving the learned algorithm (accuracy 0.97). All findings replicate across 3 seeds. 1 Introduction Training dynamics of neural networks are highly structured despite the enormous dimensionality of parameter space. The spectral edge thesis ( Xu 2026g ) makes this precise: the Gram matrix of rolling-window parameter updates develops an intra-signal eigenvalue gap that separates a few dominant directions from the bulk. These dominant directions—the spectral edge—concentrate the variance of training, and phase transitions in learning coincide with gap events ( Xu 2026h ; Xu 2026a ; Xu 2026c ) . The thesis establishes that the spectral edge exists and tracks learning. This paper asks why : what drives the spectral edge, and what gives it causal importance? 1.1 The Central Claim Claim (The Two-Phase Lifecycle). During training with weight decay, the dominant singular vector v_{1} of the Gram matrix of parameter updates (defined in § 2 ) undergoes a sharp transition: 1. Learning phase : v_{1} is gradient-driven, functionally active, and carries task-relevant information. 2. Transition : at grokking, gradient and weight decay align along v_{1} (both push the same direction). 3. Compression phase : v_{1} becomes weight-decay-driven, functionally flat, but causally essential—the model’s function depends on the edge’s orientation but is invariant to displacement along it. Everything in this paper supports this claim through multiple projections of the same underlying mechanism: • In weight space : the grad-WD decomposition (§ 3 ). • In function space : the ablation paradox and perturbation flatness (§ 4 ). • In representation space : nonlinear re-encoding and the probe R^{2} inversion (§ 5 ). • In the gap flow equation : the Term 2/Term 3 balance predicts universality classes (§ 7 ). 1.2 Relation to Prior Work The spectral edge thesis ( Xu 2026g ) derives gap dynamics from three axioms and confirms 19/20 quantitative predictions across six model families spanning 150K to 124M parameters (TinyStories 51M, GPT-2 124M, and grokking experiments on Dyck, SCAN, and modular arithmetic). A companion paper ( Xu 2026h ) shows that edge directions are functional modes —structured perturbation patterns that collapse to Fourier frequencies for modular arithmetic. The commutator analysis ( Xu 2026c ; Xu 2026b ) tracks \left\|[W_{Q},W_{K}]\right\|_{F} and shows it peaks before grokking, with superlinear lead times across Dyck, SCAN, and modular arithmetic. All three works describe the spectral edge as a geometric object : eigenvalue gaps, singular vectors, commutator norms. This paper adds a dynamical decomposition : the edge is not one thing—it transitions from a gradient-driven learning direction to a weight-decay-driven compression axis at the moment of grokking. This resolves the open question of why the spectral gap continues to widen post-grok even though learning has already occurred: the post-grok gap is maintained by weight decay compression, not by ongoing learning. 2 Setup 2.1 Tasks Dyck-1 depth prediction. A 2-layer causal Transformer ( d_{\text{model}}=128 , 4 heads, {\sim} 150K params) predicts stack depth (0–12) at each position in balanced parentheses sequences. Training: 50 sequences; test: 5000. The computation is a cumulative sum: \text{depth}(t)=\sum_{i\leq t}s_{i} where s_{i}=+1 (open) or -1 (close). SCAN. A 6-layer encoder–decoder Transformer ( d_{\text{model}}=256 , 4 heads, {\sim} 1.5M params) translates commands (“jump left twice”) to actions. Training: 2048 pairs; test: 500. The computation factors compositionally: verb \times direction \times repetition. Grokking. Both tasks are trained with AdamW ( \beta_{2}=0.98 ), weight decay \omega\in\{0,1\} , 3 seeds each. With \omega=1 : Dyck groks at steps 600–1400, SCAN at 2500–4000. With \omega=0 : no grokking in any run. Hit rate: 6/6 grok with WD, 0/6 without, per task. Combined with the thesis’s modular arithmetic runs ( Xu 2026g ) , this gives 24/24 grokking with WD and 0/24 without, across four task families. 2.2 Spectral Edge Construction Following the thesis, we compute the trajectory matrix X(t)\in\mathbb{R}^{W\times p} from W=5 consecutive parameter update deltas (restricted to attention weights), extract singular values \sigma_{1}\geq\cdots\geq\sigma_{W} and right singular vectors v_{1},\ldots,v_{W}\in\mathbb{R}^{p} . The spectral gap g_{23}=\sigma_{2}^{2}-\sigma_{3}^{2} compresses 33\times (Dyck) and 43\times (SCAN) during grokking, with k^{*}=1 universally—replicating the thesis. 3 The Mechanism: Gradient–Weight-Decay Alignment 3.1 Decomposition AdamW’s parameter update decomposes exactly: \Delta\theta=\underbrace{\Delta\theta_{\text{grad}}}_{\text{Adam-processed gradient}}+\underbrace{\Delta\theta_{\text{wd}}}_{=\;-\eta\omega\theta}. We project each component onto the Gram singular vectors v_{k} and measure the fraction of update energy from each source. Figure 1 : (A) Weight decay controls the compression–accessibility tradeoff: increasing \omega raises accuracy and attention entropy but lowers linear probe R^{2} (representations become more abstract). (B) The gradient-to-WD transition on v_{1} : both tasks flip from gradient-dominated (blue) to WD-dominated (orange) at grokking. SCAN’s transition is sharper (99.8% WD). (C) Hessian curvature: the grokked edge (solid) is flat (blue shading); the memorized bulk (dashed) is sharply curved (red shading, up to v^{\top}\!Hv=1.84 ). 3.2 The Flip Figure 1 B shows the main result. Before grokking, v_{1} is gradient-driven: 97.6% (Dyck) and 88.7% (SCAN) of the update energy along v_{1} comes from the gradient component. At grokking, this flips: gradient drops to 5.3% (Dyck) and 0.2% (SCAN), with weight decay providing the remainder. Crucially, gradient and weight decay are aligned along v_{1} —they push the same direction. On bulk directions ( v_{3} , v_{4} ), they oppose (the normal dynamics: gradient descends, WD regularizes). The alignment on v_{1} is the phase transition signature: at grokking, the optimizer’s gradient and its regularizer agree on the edge

Read the complete text