From SGD to Spectra: A Theory of Neural Network Weight Dynamics

Brian Richard Olsen, Sam Fatehmanesh, Frank Xiao ∗, Adarsh Kumarappan ∗, Anirudh Gajula · 2025

Read From SGD to Spectra: A Theory of Neural Network Weight Dynamics by Brian Richard Olsen, Sam Fatehmanesh, Frank Xiao ∗, Adarsh Kumarappan ∗, Anirudh Gajula (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12709.txt.

Introduction

Brian Richard Olsen and four colleagues derive a mathematical theory of how neural network weights evolve during training, in a 2025 paper. Networks are trained by stochastic gradient descent, a noisy step-by-step rule. Why trained weight matrices develop their observed spectral shapes, a dense bulk plus a long tail, lacked a rigorous explanation.

The team builds a continuous-time stochastic equation framework for the training process. Within it, squared singular values follow Dyson Brownian motion, complete with repulsion between neighboring values. Stationary distributions come out as gamma-type densities with power-law tails, matching the bulk-plus-tail pattern seen in practice. Controlled experiments on transformer and multilayer networks confirmed the forecasts closely. The authors present the framework as a first rigorous bridge from step-by-step training noise to the final geometry of learned weights.

Catalog record

CREATORBrian Richard Olsen, Sam Fatehmanesh, Frank Xiao ∗, Adarsh Kumarappan ∗, Anirudh Gajula
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12709

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From SGD to Spectra: A Theory of Neural Network Weight Dynamics Brian Richard Olsen, Sam Fatehmanesh, Frank Xiao ∗, Adarsh Kumarappan ∗, Anirudh Gajula Abstract Deep neural networks have revolutionized machine learning, yet their training dynamics remain theoretically unclear—we develop a continuous-time, matrix-valued stochastic differential equation (SDE) framework that rigorously connects the microscopic dynamics of SGD to the macroscopic evolution of singular-value spectra in weight matrices. We derive exact SDEs showing that squared singular values follow Dyson Brownian motion with eigenvalue repulsion, and characterize stationary distributions as gamma-type densities with power-law tails, providing the first theoretical explanation for the empirically observed ‘bulk+tail’ spectral structure in trained networks. Through controlled experiments on transformer and MLP architectures, we validate our theoretical predictions and demonstrate quantitative agreement between SDE-based forecasts and observed spectral evolution, providing a rigorous foundation for understanding why deep learning works. † † affiliationnotice: Equal contribution 1 Introduction Deep neural networks have fundamentally transformed machine learning, achieving unprecedented performance across diverse domains ( Krizhevsky et al. 2012 ; Vaswani et al. 2023 ; Jumper et al. 2021 ) . Yet despite their empirical success, our theoretical understanding of how neural networks learn remains remarkably incomplete ( Zhang et al. 2017 ; Neyshabur et al. 2017 ) . Central to this understanding is the evolution of weight matrices, whose spectral properties—the distribution and dynamics of singular values—provide deep insights into optimization dynamics, generalization behavior, and implicit regularization ( Pennington et al. 2017 ; Martin & Mahoney 2018 ) . At initialization, weight matrices exhibit well-characterized random matrix statistics described by the Marchenko-Pastur law, which characterizes the eigenvalue distribution of large random matrices, and related results from random matrix theory (RMT). However, training fundamentally alters these spectral properties, producing empirically observed ‘bulk+tail’ structured distributions that correlate strongly with generalization performance ( Papyan 2020 ; Martin & Mahoney 2021 ) . Existing theoretical frameworks fail to explain this transformation: while RMT describes initial conditions and stochastic differential equations (SDEs) can model SGD, current analyses focus on scalar parameters or low-rank models, failing to capture the full matrix-valued dynamics ( Mandt et al. 2018 ) . Most critically, no unified framework connects the microscopic stochastic dynamics of SGD to the macroscopic spectral evolution observed empirically. In this paper, we bridge this gap by developing a continuous-time, matrix-valued SDE framework with carefully designed small-scale experiments that captures the full dynamics of singular value evolution under SGD. Our key contributions are: 1. We derive exact SDEs for individual singular values under isotropic SGD noise, connecting the microscopic parameter updates to macroscopic spectral dynamics. Under the assumption of negligible gradients, we show that squared singular values follow a Dyson Brownian motion with \beta=1 , explaining the eigenvalue repulsion and spectral spreading. 2. We characterize the stationary spectral distribution in the non-negligible gradient regime using mean-field theory. We prove that the limiting distribution follows a gamma-type density with power-law tails, recovering the empirically observed ‘bulk+tail’ structure in trained networks and providing the first theoretical explanation for this empirical phenomenon. 3. Through experiments on transformer ( Vaswani et al. 2023 ) , vision transformer ( Dosovitskiy et al. 2020 ) , and MLP ( Rumelhart et al. 1986 ) architectures, we demonstrate quantitative agreement between our SDE-based predictions and spectral evolution and propose an algorithm to forecast singular value dynamics from minimal gradient information. We show that SGD’s stochastic noise acts like a “spectral sculptor”—initially spreading eigenvalues apart via repulsion, then concentrating them into beneficial empirically observed ‘bulk+tail’ structured patterns that enable generalization, connecting microscopic mini-batch randomness to macroscopic spectral evolution. Our work demonstrates how small-scale experimentation can unlock fundamental insights with implications for initialization strategies, optimization algorithm design, and understanding why deep learning works. 2 Related Works Spectral Analysis of Neural Network Weights. RMT establishes that at initialization, weight matrices follow Wigner’s semicircle law ( Wigner 1955 ) and the Marčenko–Pastur distribution ( Marčenko & Pastur 1967 ) , with edge statistics governed by Tracy-Widom distributions ( Tracy & Widom 1996b ) . Training induces pronounced deviations: singular-value spectra become highly anisotropic ( Saxe et al. 2014 ) , evolving ‘bulk+tail’ structured distributions linked to class structure ( Papyan 2020 ) and implicit regularization ( Martin & Mahoney 2021 ) . Martin and Mahoney ( Martin & Mahoney 2021 ) identified 5+1 phases of spectral evolution and showed that batch size affects spectral properties, with smaller batches leading to stronger implicit self-regularization. Extensions to empirically observed ‘bulk+tail’ structured matrix ensembles ( Auffinger et al. 2008 ) provide theoretical foundations past Gaussian universality classes. SGD as Stochastic Dynamics. SGD can be approximated by stochastic differential equations (SDEs), with constant-rate SGD behaving like an Ornstein–Uhlenbeck process ( Mandt et al. 2017 ) and anisotropic noise structures enabling escape from sharp minima ( Zhu et al. 2019 ) . Weight updates have been mapped to Dyson Brownian motion ( Aarts & de Wit 2024 ) , explaining eigenvalue repulsion as a Coulomb-gas phenomenon. The mathematical foundation relies on Itô calculus for matrix functions ( G. W. Stewart & Ji-guang Sun 1990 ) and Fokker-Planck equations for interacting particle systems ( Risken & Frank 1996 ) . The implicit regularization effects of gradient descent have been explored ( Neyshabur et al. 2015 ) . 3 Methodology Our approach transitions from discrete microscopic SGD dynamics to continuous macroscopic spectral evolution. We produce notation for the training of neural networks via a spatiotemporal interpretation of the evolution of the weight matrices with stochasticity: dW(x,t)=-\eta\frac{\partial\mathcal{L}}{\partial W(x,t)}dt+\sqrt{2\eta D_{W}(x,t)}\,d\mathcal{W}_{W}(x,t) , where \frac{\partial\mathcal{L}}{\partial W(x,t)} is the gradient of the loss, \eta is the learning rate, D is an effective diffusion constant described by d\mathcal{W}_{W} , and d\mathcal{W}_{b} are independent matrix/vector-valued Wiener processes, capturing the stochastic dynamics of SGD. We split our analysis into two cases: negligible gradient in loss and non-negligible gradient in loss. For the first limit (negligible gradient in loss, or \frac{\partial\mathcal{L}(W)}{\partial W}\approx 0 ), we perform SVD/eigenvalue decomposition and use Ito Calculus (see Appendix 6.1 ) to arrive at the result of Theorem 3.1 : Theorem 3.1 (Stochastic Dynamics of Singular Values) . Let W\in\mathbb{R}^{m\times n} evolve via stochastic gradient descent with noise. Then, the singular values \sigma_{k}(W) follow the SDE: \displaystyle d\sigma_{k}(t)=\left[-\eta u_{k}^{T}(\nabla_{W}\mathcal{L})v_{k}+\eta D\left(\frac{m-n+1}{2\sigma_{k}}+\sum_{j\neq k}\frac{\sigma_{k}}{\sigma_{k}^{2}-\sigma_{j}^{2}}\right)\right]dt+\sqrt{2\eta D}d\beta_{k}(t) where u_{k},v_{k} are the singular vectors and D is the effective diffusion strength. This theorem shows that under SGD noise, singular values behave like interacting particles that repel each other (the \sum_{j\neq k} terms), explaining why the spectrum spreads out during training rather than collapsing. This SDE can then be mapped to Dyson-Brownian Motion processes (see Appendix 6.6 whose statistics are described by the Marčenko–Pastur ( Lemma 6.9 ) and Tracy-Widom distributions respectively ( Lemma 6.11 ). After deriving these microscopic underpinnings, we may consider the second case (non-negligible gradient in loss, or \frac{\partial\mathcal{L}(W)}{\partial W}\neq 0 ) and transition into a macroscopic limit by considering the empirical spectral density distribution \rho(\lambda,t) such that: \rho(\lambda,t)=\frac{1}{r}\sum_{k=1}^{r}\delta(\lambda-\lambda_{k}(t)) As r\to\infty , we assume that \rho(\lambda,t) converges to a deterministic density function, normalized such that \int\rho(\lambda,t)d\lambda=1 . To study the dynamics of the squared singular values \{\lambda_{j}\} then, we adopt a mean-field perspective by assuming the effective influence of the complex loss function \mathcal{L}(W) is captured by a potential \mathcal{L}_{MF} that depends only

Read the complete text