Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks Guillaume Corlouer *, Avi Semler, Alexander Strang, Alexander Gietelink Oldenziel Abstract Deep linear networks (DLNs) are used as an analytically tractable model of the training dynamics of deep neural networks. While gradient descent in DLNs is known to exhibit saddle-to-saddle dynamics, the impact of stochastic gradient descent (SGD) noise on this regime remains poorly understood. We investigate the dynamics of SGD during training of DLNs in the saddle-to-saddle regime. We model the training dynamics as stochastic Langevin dynamics with anisotropic, state-dependent noise. Under the assumption of aligned and balanced weights, we derive an exact decomposition of the dynamics into a system of one-dimensional per-mode stochastic differential equations. This establishes that the maximal diffusion along a mode precedes the corresponding feature being completely learned. We also derive the stationary distribution of SGD for each mode: in the absence of label noise, its marginal distribution along specific features coincides with the stationary distribution of gradient flow, while in the presence of label noise it approximates a Boltzmann distribution. Finally, we confirm experimentally that the theoretical results hold qualitatively even without aligned or balanced weights. These results establish that SGD noise encodes information about the progression of feature learning but does not fundamentally alter the saddle-to-saddle dynamics. 1 Introduction Stochastic gradient descent (SGD) and its variants are optimization algorithms widely used to train deep neural networks. Classical statistical learning theory struggles to account for the observed ability of deep neural networks to generalize beyond their training dataset [ 35 ] . Therefore, it is proposed that there must be an implicit bias in the learning algorithm that emerges from the interaction of data, architecture and optimizer. Understanding this bias may be important for AI alignment and safety [ 18 , 3 ] . SGD differs from gradient descent by computing the update direction based on a randomly sampled subset of the data that varies between updates, rather than using a fixed dataset for all of training. One can distinguish the contribution of two terms during SGD. The first term is the gradient, and the second term is the stochasticity, which comes from the randomness in approximating the gradient. It is unclear whether the stochasticity is crucial in shaping the implicit bias, or if it is merely computationally convenient: for example, 23 shows that for high-dimensional convex optimization the noise is not important, and 31 provides empirical evidence that the regimes under which stochasticity is not relevant for generalization. However, 25 shows that for diagonal linear networks it promotes sparsity. Deep linear networks (DLNs) are a simple class of neural networks, consisting only of matrix multiplication operations. Despite only expressing linear functions, their training dynamics are nonlinear and exhibit many of the interesting phenomena that occur in architectures with nonlinearities [ 22 ] . We choose DLNs as the setting for this work because it makes precise mathematical analysis tractable. A key result in this literature is that gradient flow on DLNs, under small initialization, proceeds through a saddle-to-saddle regime. In this regime, the network traverses a sequence of saddle points, learning the singular values of the target ("teacher") matrix in decreasing order of magnitude [ 15 ] . This stage-wise, time-scale-separated dynamics has been characterized exactly for gradient flow [ 28 ] , and extensions to stochastic gradient flow in diagonal and rank-one linear networks have been explored [ 25 , 19 ] . However, the impact of using a continuous model of SGD rather than gradient flow has not been analytically characterized for fully connected DLNs. 1.1 Contributions We study the training dynamics of SGD modeled as a stochastic differential equation (SDE) on deep linear networks. More specifically, assuming that the weights are balanced and aligned during training, we model SGD using its continuous limit as an Itô SDE and decompose it into a system of one-dimensional SDEs. We focus on the saddle-to-saddle regime 15 , during which the singular values of a teacher matrix are learned in parallel and at different time scales. This extends the gradient-flow analysis of 28 to a stochastic setting. Our main contributions are: 1. Exact SGD noise covariance. We derive a closed-form expression for the gradient noise covariance matrix of SGD in DLNs, both with and without label noise. This expression is state-dependent and anisotropic, and it decomposes cleanly into a data-mismatch term and a label-noise term. 2. Modewise diffusion predicts feature learning. Under the balanced and aligned assumptions, we show that the diffusion coefficient along a given mode peaks before that mode is fully learned, then decays to zero once the mode has been fully learned (shown in Figure 1 ). This establishes that SGD noise carries information about the progression of feature learning. 3. Stationary modewise distributions. We characterize the stationary distribution of the modewise SDE via detailed balance. In the absence of label noise, the stationary distribution collapses to a Dirac mass, thus matching the gradient flow solution. With label noise, it is approximately Boltzmann. 4. State-dependent noise is a more accurate model. We also find that a continuous model of SGD with state-dependent noise is a more accurate model of SGD than the isotropic homogeneous noise (Langevin), which is commonly assumed in the literature during the feature learning regime and for the end-of-training distribution. Together, these results show that SGD noise encodes information about the stage of learning, but does not qualitatively alter the saddle-to-saddle structure: modes are still learned in order of decreasing singular value magnitude, with SGD primarily affecting the timescale of each transition. We also verify experimentally that the qualitative predictions hold even when the balanced and aligned assumptions are relaxed. 1.2 Related work 1.2.1 Implicit biases of SGD noise Some previous work argues that SGD noise matters for generalization. More specifically, gradient noise induces an implicit bias in SGD that attracts dynamics towards invariant sets of the parameter space corresponding to simpler subnetworks. This manifests as a noise-induced drift that pulls parameters toward zero, making neurons vanish or become redundant [ 6 ] . In diagonal linear networks, stochastic gradient flow has an implicit bias toward sparser solutions that is not present in gradient flow, suggesting that stochasticity matters for generalization [ 25 ] . In non-linear deep neural networks, during loss stabilization, the combination of gradient and noise also induces a bias towards sparser solutions when the learning rate is large [ 2 ] . Furthermore, some implicit biases of SGD can be made explicit by showing that SGD achieves the same performance as GD with an explicit regularization term that penalizes large batch-gradient updates [ 10 ] . In linear networks, stochastic gradient flow appears to be less dependent on initialization than gradient flow and induces an additional bias towards simpler solutions beyond the simplicity bias of gradient flow [ 29 ] . Other work has investigated the structure of SGD noise, which is state-dependent (i.e., it differs between points in parameter space) and anisotropic (the distribution of the noise is not rotationally invariant). The structure of the noise could matter for generalization and understanding the training dynamics of SGD. For example, in deep linear networks, SGD structured noise does not allow jumps from lower-rank to higher-rank weight matrices, while the noise from a Langevin process has a non-zero probability of jumping back to higher rank solutions [ 32 ] . Furthermore, the structured noise of SGD is sensitive to geometry and can induce an implicit bias towards flatter minima [ 34 ] . In particular, critical points of the loss landscapes of deep neural networks are typically highly degenerate [ 27 ] , and SGD noise is sensitive to these degeneracies by slowing down along degenerate directions. This slowing effect is not present with Langevin dynamics [ 8 ] . In addition to being structured, SGD noise can also be autocorrelated (colored noise). In a dynamical mean field theoretic model, SGD noise can converge to a non-equilibrium steady-state solution, where noisier regimes are associated with solutions that are more robust due to having wider decision boundaries [ 21 ] . However, other work suggests that the implicit biases arising from stochasticity do not matter in some regimes. In particular, the Golden Path hypothesis states that the population loss of gradient descent is upper bounded by the population loss of SGD for a given trajectory with fixed initialization in the online learning regime in which new batches are sampled at each time step. Empirical evidence for this Golden Path hypothesis has been found by showing that switching from high noise to low noise during training leads to convergence to the same solution as when using only low noise, for convolutional neural networks and transformers [ 31 ] . 23 prove that a Golden Path hypothesis holds for convex quadratic loss landscapes in high dimensions, using a novel continuous model of SGD [ 24 , 21 ] . 1.2.2 Regimes of learning in Deep Linear Networks Deep linear networks (DLNs) serve as an analytically tractable toy model that can shed light on the training dynamics of non-linear deep neural networks. DLN training has a rich non-linear dynamics and a non-convex high-dimensional loss landscape despite the expressivity of DLNs being limited to linear functions [ 22 ] . A particularly important result is the exact solution of gradient flow on DLNs [ 28 ] during the feature learning regime. In this regime, the training dynamics undergoes a separation of time-scales in which a DLN learns the singular values of the teacher matrix in decreasing order of size. This feature learning regime corresponds to a saddle-to-saddle dynamics in which gradient flow traverses the loss landscape through a series of saddle points in which the loss is stabilized until the flow can escape to the next saddle, which increases the rank of the solution by one [ 15 ] . This regime of saddle-to-saddle dynamics (also called “rich” regime) contrasts with the “lazy” regime in which the neural tangent kernel (NTK) at initialization––a linear operator––determines the time evolution of the network’s function [ 14 ] . The regime of training is determined by hyperparameters such as the variance of the parameters at initialization or the width of the neural network [ 9 ] . Transitions between regimes are possible: for example, the grokking phenomenon has been hypothesized to be a transition from a lazy to a rich regime [ 16 ] . In the limit of large depth, width, and amount of data (with constant ratios between these quantities), the generalization error of gradient flow has been characterized under different parametrizations with dynamical mean field theory, which enables theoretical predictions about gains from increased width and scaling laws in the training curve for some structured data [ 4 ] . Importantly, the training dynamics of gradient flow in DLNs is well understood [ 1 ] , and extensions to stochastic gradient flow have been explored in diagonal linear networks [ 25 ] and rank-one linear networks [ 19 ] . Additionally, SGD noise anisotropy causes the weights’ fluctuations during training to be inversely proportional to the flatness of the loss landscape in two-layer DLNs [ 11 ] . However, the training dynamics of SGD in fully connected DLNs remain to be understood in the rich (saddle-to-saddle) regime. 1.2.3 Steady-state distribution of SGD Another facet of the training dynamics is understanding the convergence properties of SGD, and specifically its end-of-training distribution. Under the assumptions that SGD is well approximated by Langevin dynamics, i.e. that SGD noise is white noise (Gaussian, with constant isotropic covariance) and that the loss is non-degenerate, then SGD approximates Bayesian inference and its limiting distribution is a Boltzmann distribution [ 20 , 33 ] . However, SGD noise is anisotropic and state-dependent, and the loss of neural networks is highly degenerate, which induces differences from the Bayesian approximation. For example, unlike a Bayesian learner, SGD can get stuck along a degenerate direction of a critical submanifold of the loss landscape [ 8 ] . Additionally, degeneracies and noise anisotropy can induce a non-equilibrium steady-state distribution with circular currents where the weights oscillate around critical points [ 5 , 17 ] . The end of distribution of SGD can be better understood if we model SGD as optimizing a competition between an energy and an entropy term corresponding to a Helmholtz Free Energy functional [ 26 , 5 ] . Another intriguing phenomenon is the anomalous diffusion of SGD. Specifically, we can observe sub-diffusive behavior of SGD where the mean square displacement of the weights is slower than what would be expected under Brownian motion. At the level of the distribution of SGD trajectories, this can be modeled by a time-fractional Fokker-Planck equation [ 13 ] . (a) Modes being learned (b) Diffusion along modes Figure 1: Predicting when modes are learned using the predicted time of maximum diffusion, in a 4-layer linear network trained with SGD. (a) The modes are learned in order of magnitude, with the time of learning being predicted by the time of maximum diffusion. (b) The diffusion along a mode peaks while a mode is being learned, and our theoretical prediction (see Equation 7 ) matches what is observed. The vertical lines in (a) correspond to the peaks (marked with \bigstar in (b)) of the theoretical prediction for the diffusion (see Equation 9 ). 2 Preliminaries 2.1 Online SGD update Let \mathcal{X}\subset\mathbb{R}^{d_{0}} be a set of inputs, and \mathcal{Y}\subset\mathbb{R}^{d_{L}} be a set of possible outputs. We consider a joint distribution p_{X,Y} over \mathcal{X}\times\mathcal{Y} , with marginal p_{X} and conditional output distribution p_{Y\mid X} . We write X\sim p_{X} for a random input and Y\mid X\sim p_{Y\mid X}(\,\cdot\mid X) for the associated output. A deep neural network architecture is a function 1 1 1 More specifically a composition of linear and non-linear functions, loosely abstracting the function of biological neurons f:\Theta\times\mathcal{X}\to\mathcal{Y} with parameters \theta\in\Theta\subseteq\mathbb{R}^{d} which can be trained to learn the expected output \mathbb{E}[Y|X=x] given an input x by minimizing the mean-squared error (MSE) over the data distribution: L(\theta)=\mathbb{E}_{X,Y}[\ell(\theta;Y,X)],\qquad\ell(\theta;y,x)=\tfrac{1}{2}\|y-f(\theta;x)\|^{2}. Because this is often intractable to minimize directly, we instead calculate the empirical batch loss and its gradient on a finite batch B=\{(x_{i},y_{i})\}_{i=1}^{b} , sampled independently from the distribution: L_{B}(\theta)=\frac{1}{b}\sum_{i=1}^{b}\ell(\theta;y_{i},x_{i}). In online 2 2 2 This contrasts with offline SGD, where a finite dataset is sampled from the distribution, and then all batches are sampled from the finite dataset SGD, the loss is minimized by initializing the neural network parameters as \theta_{0} , and then repeatedly sampling batches and updating the parameters using the empirical batch gradient g_{B}(\theta):=\nabla L_{B}(\theta) . Given a batch B_{k}\subset(\mathcal{X}\times\mathcal{Y})^{b} of size b , the discrete-time SGD update with learning rate \eta_{k}>0 is given by: \theta_{k+1}=\theta_{k}-\eta_{k}\,g_{B_{k}}(\theta_{k}). Observe that the batch gradient is an unbiased estimator of the population gradient: \mathbb{E}_{B}[g_{B}(\theta)]=\nabla L(\theta) . 2.2 Continuous limit of SGD (constant step size) Define the one-sample gradient noise \xi(\theta;X,Y):=g_{(X,Y)}(\theta)-g(\theta) , a random variable denoting the difference between the batch gradient using (X,Y) , and the population gradient g(\theta):=\nabla L(\theta) . Its covariance matrix is \Sigma(\theta):=\mathbb{E}_{X,Y\sim p_{X,Y}}\big[\xi(\theta;X,Y)\,\xi(\theta;X,Y)^{\top}\big]. For batches of size b , the batch-gradient covariance satisfies. \Sigma_{b}(\theta)=\frac{1}{b}\Sigma(\theta), and we see that the batch-gradient noise covariance \Sigma_{b} is proportional to the one-sample gradient noise covariance. Under the usual martingale functional CLT assumptions (uniform convergence of conditional quadratic variation and Lindeberg condition; see Appendix A.2 ), the piecewise-constant interpolation of \{\theta_{k}\} with constant \eta converges (as \eta\to 0 ) to the Itô SDE d\theta_{t}=-\,g(\theta_{t})\,dt+\sqrt{\eta\,\Sigma_{b}(\theta_{t})}\,dW_{t}, (1) with W_{t} a Wiener process. Equation ( 1 ) is the continuous-time model of SGD that we will use throughout, and will refer to as anisotropic Langevin dynamics . 2.3 Deep linear networks (DLNs) and mode dynamics during gradient flow Let x\in\mathbb{R}^{d_{0}} and W_{l}\in\mathbb{R}^{d_{l}\times d_{l-1}} . A depth- L DLN is defined by the following linear input-output map: f(x)=W_{L}W_{L-1}\cdots W_{1}x=Wx\in\mathbb{R}^{d_{L}} Data is generated by a teacher M\in\mathbb{R}^{d_{L}\times d_{0}} via Y=MX with whitened Gaussian inputs X such that \mathbb{E}[XX^{\top}]=I_{d_{0}} . We will also sometimes consider some label noise in addition to the teacher matrix, i.e., Y=MX+\xi_{q} where \xi_{q}\sim N(0,\sigma_{q}I) . Let the SVD be M=USV^{\top} , with left and right singular values (u_{\alpha},v_{\alpha}) associated to the singular value s_{\alpha} . A standard approach in the theory of deep linear networks is to decompose the training dynamics onto modes , i.e., onto a particular left and right singular value of the target function. Define the mode and cross-mode amplitude of the student: w_{\alpha}:=u_{\alpha}^{\top}\!\big(W_{L}\cdots W_{1}\big)v_{\alpha},\qquad w_{\alpha\beta}:=u_{\alpha}^{\top}\!\big(W_{L}\cdots W_{1}\big)v_{\beta}. Intuitively, the mode amplitude measures the extent to which the network function has learned a singular value of the teacher function. For example, when a mode has been fully learned, we have w_{\alpha}=s_{\alpha} . Under balanced initialization ( W_{\ell+1}^{\top}W_{\ell+1}=W_{\ell}W_{\ell}^{\top} ), no label noise 3 3 3 See [ 1 ] for the case with label noise and orthogonality of distinct modes (i.e. all cross modes are zero), the gradient-flow (GF) dynamics (continuous-time limit of GD) on a depth-2 linear network decouples along modes (see 28 for more details): \dot{w}_{\alpha}\;=\;2\,(s_{\alpha}-w_{\alpha})\,w_{\alpha} (2) Despite the linearity of DLNs, the latter equation is non-linear in the mode amplitudes. The solution of ( 2 ) is logistic: w_{\alpha}(t)=\frac{s_{\alpha}}{1+\left(\frac{s_{\alpha}}{w_{\alpha}(0)}-1\right)e^{-2s_{\alpha}t}}, (3) showing stagewise learning : larger s_{\alpha} modes rise earlier and faster (characteristic timescale \sim 1/s_{\alpha} ). The training dynamics are nonlinear and, generically, exhibit saddle-to-saddle transients before reaching minimizers; non-strict saddles are prevalent in DLNs and also arise in nonlinear DNNs. (a) Discrete optimizers (b) Numerical simulation of continuous limits (c) Mode growth with gradient descent (d) Mode growth with SGD Figure 2: Saddle-to-saddle dynamics with different optimizers in a depth-6 linear network, with the sharp changes corresponding to increases in the numerical rank of the network. (a) Train loss plateaus with discrete optimizers (b) Train loss plateaus with numerical simulation of their continuous counterparts (c) Mode growth over training with gradient descent, showing that the 5 singular values of the teacher matrix are learned in descending order of magnitude (d) Mode growth over training for stochastic gradient descent, showing that the stagewise dynamics are retained, but modes take longer to be learned. (See Appendix H for details of the numerical simulations and experiments.) In this work, we consider a DLN of depth L with small initialization of its weights, i.e., in the feature-learning regime. We take the continuous limit of SGD 2.2 into a stochastic differential equation (SDE) with state-dependent and anisotropic Gaussian noise 1 . We study the dynamics of the mode amplitude during anisotropic Langevin dynamics under the assumption of aligned and balanced weights. 2.4 Assumptions We make the following assumptions: • Continuous-time model : We model SGD by the SDE in Equation 1 , with state-dependent anisotropic noise. • Whitened inputs : The input distribution p_{X} is Gaussian with the covariance matrix being the identity. 4 4 4 This can probably be relaxed by demanding that the input and input-output covariance matrices are diagonalizable in the same basis. • Online learning : Batches are sampled directly from the data distribution, rather than from a finite dataset. This can be seen as a large-sample limit of the offline, finite-dataset case; Appendix A discusses relaxation to the offline case. To derive decoupled modewise SDEs, we additionally assume: (A1) Balanced weights: The layer weight matrices satisfy W_{\ell+1}(t)^{\top}W_{\ell+1}(t)=W_{\ell}(t)W_{\ell}(t)^{\top} for all layers \ell and all times t in training. This condition is preserved under gradient flow. (A2) Aligned modes: The cross-mode amplitudes vanish, i.e., w_{\alpha\beta}(t):=u_{\alpha}^{\top}W(t)v_{\beta}=0 for all mode indices \alpha\neq\beta and all times t in training. Equivalently, the student W is diagonal in the teacher’s singular basis. These standard assumptions in the DLN literature, and Appendix I tests how well they hold in practice. We refer to (A1) – (A2) collectively as the balance and alignment assumptions . 3 Modewise state-dependent SDE for SGD on DLNs This section contains results about how the modes evolve under the continuous-time model of Equation 1 . We start by showing that the population loss gradient is a product of the Jacobian and the student-teacher gap M-W_{L}\dots W_{1} . Proposition 3.1 (Gradient structure) . For each layer l of a DLN, define the partial products W_{>l}:=W_{L}\cdots W_{l+1},\qquad W_{l}^{\top};\quad J_{l}(x):=x^{\top}W_{l}^{\top} The Jacobian with respect to all layers is the block matrix: \,J(\theta;x)=\big[\,J_{1}(x)\ \mid\ J_{2}(x)\ \mid\ \cdots\ \mid\ J_{L}(x)\,\big] Let \Delta:=M-W=M-W_{L}...W_{1} be the error term between the teacher matrix and the end-to-end linear map W:=W_{L}...W_{1} that the network implements. The population gradient of the mean-square error loss function is: G_{l}:=\nabla_{W_{l}}L=-W_{>l}^{\top}\Delta W_{l})\textup{vec}(\Delta)=-J_{l}^{\top}\delta The gradient vector of the population loss is given by g:=(g_{l})_{l} The gradient is zero if and only if the Jacobian of the implemented linear map W is orthogonal to the teacher-student gap term \delta . It is interesting to observe that the set of critical points associated with a given level set of the loss is highly degenerate. Indeed, the loss and the zero set of the gradient are both invariant under transforming the weight matrices of the hidden layers by some action of the general linear group, i.e., for a hidden layer l of width d_{l} , we have: W_{l}P^{-1}_{l}P_{l}W_{l-1}=W_{l}W_{l-1},\ \text{for}\ P\in GL_{d_{l}}(\mathbb{R}) Next, we derive the covariance matrix of the gradient noise. Proposition 3.2 . Let X\sim\mathcal{N}(0,I_{d}) be the input data, let \xi_{q} be the label noise with covariance \Sigma_{q} and zero mean, with \mathbb{E}[\xi_{q}X^{\top}]=0 , and \xi_{q}\perp X . 5 5 5 This amounts to modelling aleatoric uncertainty in the output We consider the stacked (vectorized) one-sample gradient noise across all layers and its covariance matrix \Sigma(\theta)\in\mathbb{R}^{d\times d} (with d=\sum_{l}\dim(\mathrm{vec}\,W^{l}) ) such that \Sigma(\theta)=\big[\Sigma_{lm}(\theta)\big]_{l,m=1}^{L} is the stacked covariance of the vectorized layerwise gradient noise. 6 6 6 We use \mathrm{vec}(\cdot) and the Kronecker product \otimes . Let \Delta:=M-W=M-W_{L}...W_{1} be the gap between the teacher matrix M and the product of the weight matrices W . For all l,m , we have: \boxed{\;\Sigma_{lm}(\theta)\;=\;\underbrace{J_{l}^{\top}(I\otimes\Delta)\,(I_{d_{0}^{2}}+C)\,(I\otimes\Delta)^{\top}J_{m}}_{\text{data-mismatch term}}\;+\;\underbrace{(W_{l}^{\top}\Sigma_{q}W_{>m})}_{\text{label-noise term}}\;,} where C is the commutation matrix satisfying C\,\mathrm{vec}(A)=\mathrm{vec}(A^{\top}) . A proof of this proposition can be found in the appendix B . The gradient noise covariance matrix is state-dependent and anisotropic. This means that the noise of SGD is structured and depends on the geometry of the loss landscape. In the absence of label noise, we see that the gradient noise covariance is zero at global minima (where \Delta=0 ) and at zeros of the Jacobian of the parameter-function map. To see the time-scale separation of feature learning under the SDE model of SGD, we decompose the SDE along the modes of the teacher matrix M . The next proposition provides a general form for the diffusion of SGD along specific modes. It relies on the same assumptions as the decomposition along modes during gradient flow in 28 . Proposition 3.3 (Mode and cross-mode diffusion) . Let v_{\alpha} and u_{\alpha} be a fixed pair of right and left singular vectors (respectively) of the teacher matrix M . Define the mode amplitude w_{\alpha}(\theta):=u_{\alpha}^{\top}W\,v_{\alpha} . The gradient of the mode amplitude with respect to the weight matrix W_{l} satisfies \;\partial_{W^{l}}w_{\alpha}\;=\;\big(W_{>l}^{\top}u_{\alpha}\big)\big(W_{l}^{\top}u_{\alpha})\,(W_{m (with l,m swapped for ll}^{\top}u_{\alpha})(v_{\alpha}^{\top}W_{0 gives the stationary condition \frac{dD_{\alpha}}{dw_{\alpha}}=0\;\;\Longleftrightarrow\;\;(2a+4)(w_{\alpha}-s_{\alpha})^{2}+4s_{\alpha}(w_{\alpha}-s_{\alpha})+a\,\sigma_{q}^{2}=0,\qquad a=\frac{4(L-1)}{L}. Solving for w_{\alpha} yields the critical point w_{\alpha}^{\star}=\frac{(a+1)\,s_{\alpha}-\sqrt{\,s_{\alpha}^{2}-\frac{a(a+2)}{2}\,\sigma_{q}^{2}\,}}{\,a+2\,}. (9) This maximum exists if and only if the discriminant is non-negative, that is \sigma_{q}^{2}\;\leq\;\frac{2s_{\alpha}^{2}}{\,a(a+2)\,}\;=\;\frac{s_{\alpha}^{2}L^{2}}{\,4(L-1)(3L-2)\,}. (10) Whenever ( 10 ) holds, the maximizer satisfies w_{\alpha}^{\star}0 is the teacher singular value for mode \alpha , \eta>0 is the learning rate, \beta>0 is the temperature, and \sigma_{q}\geq 0 is the label noise scale. Assume detailed balance for the Fokker–Planck equation induced by ( 11 ) and D_{\alpha}(w_{\alpha})>0 . Then any stationary density p_{\alpha}^{\star} satisfies p_{\alpha}^{\star}(w)\;\propto\;\frac{1}{D_{\alpha}(w)}\,\exp\!\left(\int_{0}^{w}\frac{2\,\mu_{\alpha}(z)}{D_{\alpha}(z)}\,dz\right). (16) Moreover, we have the following: (i) No label noise ( \sigma_{q}=0 ): D_{\alpha}(w)\sim Q\,(s_{\alpha}-w)^{2} and \mu_{\alpha}(w)\sim q\,(s_{\alpha}-w) as w\uparrow s_{\alpha} with constants Q,q>0 . The density ( 16 ) is non-normalizable unless it collapses to a Dirac mass, hence p_{\alpha}^{\star}\;=\;\delta(w-s_{\alpha}). (ii) With label noise ( \sigma_{q}>0 ): there is a unique smooth stationary density peaked near the zero of \mu_{\alpha}(w) . One has the small- \eta expansions \displaystyle w_{\alpha}^{\star}-s_{\alpha} \displaystyle=\frac{\eta}{2B}\,\,(L-1)\,s_{\alpha}^{-2/L}\,\sigma_{q}\;+\;O(\eta^{2}), (17) \displaystyle\operatorname{Var}_{p_{\alpha}^{\star}}(w_{\alpha}) \displaystyle=-\frac{D_{\alpha}(s_{\alpha})}{2\,\mu_{\alpha}^{\prime}(s_{\alpha})}\;+\;O(\eta^{2})=\frac{\eta\,L}{2B}\,\sigma_{q}^{2}\,s_{\alpha}^{\frac{2(L-1)}{L}}\;+\;O(\eta^{2}), (18) so p_{\alpha}^{\star} is approximately Gaussian with mean s_{\alpha}+O(\eta) (slightly above s_{\alpha} due to the Itô drift) and O(\eta) variance. The proof of this proposition can be found in the appendix F . The latter proposition shows that in the absence of label noise, the modewise distribution of parameters is similar for SGD and GD, i.e., a Dirac at a specific mode. In the presence of label noise, SGD appears to approximate a Boltzmann distribution at finite time, although we suspect that it converges to a Dirac in the long run. 5.1 Experimental results (a) Online SGD (b) Euler-Maruyama simulation of Equation 1 , with anisotropic and state-dependent noise (c) Isotropic Gaussian noise Figure 4: Comparison of the empirical distributions of the amplitude of the first mode w_{0} at the end of training for SGD, anisotropic Gaussian, and isotropic Gaussian noise. In the absence of label noise, SGD (a) concentrates entirely on the value of the top singular value of the teacher matrix, but anisotropic noise (b) does not have this behavior. However, the variance of the distribution for anisotropic noise is lower than for isotropic noise (c), and is thus more similar to SGD. In Figure 4 , we use histograms to visualize the distribution of the amplitude of the largest mode at the end of training. SGD, shown in 4(a) sharply peaks around the teacher’s mode amplitude (1.0). This matches the theoretical prediction of Proposition 5.2 . However, the Euler-Maruayama discretization of the anisotropic Langevin dynamics SDE ( Equation 1 ) exhibits a less sharply peaked distribution ( 4(b) ) than SGD does. We conjecture that the gap between the observed end-of-training distribution and the prediction of a Dirac distribution is because of discretization error; Appendix K provides evidence for this by showing that the variance of the mode amplitude’s distribution reduces when the simulation uses finer time steps. Figure 5: End-of-training distribution in the presence of label noise (variance 0.1) of the amplitude of the first mode w_{0} for SGD. The distribution changes from being concentrated at a point to having greater variance. Figure 5 shows the same histogram in the presence of label noise for SGD. In agreement with the prediction of part (ii) of Proposition 5.2 , the distribution is approximately Gaussian and is no longer concentrated at a point. 6 Discussion We derive the training dynamics of stochastic gradient descent in deep linear networks with balanced weights, aligned modes of the teacher matrix, and whitened inputs. We extended previous analyses of gradient flow in DLNs [ 28 ] to stochastic Langevin dynamics as a model of SGD. Furthermore, we considered an anisotropic and state-independent gradient noise covariance matrix of SGD. We gave an analytic expression of the gradient noise covariance matrix in deep linear networks and its behavior along specific modes of the DLN. We found that the modewise diffusion of SGD precedes the time at which a feature is going to be fully learned during the growth of mode amplitude away from its initialization. This observation shows that stochasticity encodes information about the progression of feature learning. Finally, we showed that the stationary modewise distribution of the stochastic Langevin process approaches that of discrete-time SGD (which is the same as GD in the absence of label noise), concentrating around the teacher singular values. Limitations First of all, we have modeled SGD with a continuous limit, ignoring the effect of a finite learning rate. However, this assumption might be overcome by using an effective potential instead of the loss function, using a central flow term accounting for the oscillations of gradient descent induced by a finite learning rate 7 . We also assumed that the data is i.i.d. and that SGD noise is not heavy-tailed. Further work could look into heavy-tailedness by using specific data distribution on which heavy-tailedness is important 12 . We also assumed that the weights are balanced and aligned assumptions, which are common when studying DLNs but the assumption of modes alignment is not always accurate (see figure 6 ). These assumptions are important, and further work could relax them to extend our analyses to take into account cross-modes interactions. Future directions. The framework developed here opens several avenues for further investigation. Beyond the linear setting, we would like to study how mode-wise diffusion manifests in non-linear architectures, such as two-layer ReLU networks, and to what extent we can track stagewise feature learning. Another direction would be to investigate the Golden Path hypothesis. This hypothesis states that in the online learning regime, in which new, fresh batches are sampled at each time step, stochasticity does not affect generalization and the function selected by the training process and instead is mere computational convenience 30 . While we have seen that SGD noise carries information about when a new feature is going to be learned, it might in itself not matter for learning a particular feature, as gradient flow is able to learn the same features in the same order as anisotropic Langevin dynamics, and the loss curves have similar qualitative stagewise behavior. Clarifying under which conditions the golden path hypothesis holds would be important, as it would enable us to simplify the theoretical analysis of deep neural networks’ training dynamics by restricting it to the study of gradient flow. 7 Acknowledgments We used ChatGPT and Claude as tools to assist in mathematical derivations and checking intermediate calculations. All proofs were verified by the first author. ChatGPT and Claude were used to assist with writing, editing, and LaTeX formatting. This work was funded by the Pivotal fellowship. Appendix A Setup A.1 Gradient noise Let B_{k} be a batch of b independent samples from the data distribution p_{\mathcal{X}\times\mathcal{Y}} . Define the gradient batch noise as: \displaystyle\xi(B_{k}) \displaystyle:=g(\theta_{k})-g_{B_{k}}(\theta_{k}) The batch gradient noise covariance matrix is defined as: \displaystyle\Sigma_{b} \displaystyle:=\mathbb{E}_{B_{k}}[\xi(B_{k})\xi(B_{k})^{\top}] Where the expectation is taken over all possible batches of size b . Let g_{i} be the 1-sample gradient of the loss function \ell(\theta;y_{i},x_{i}) over N samples (x_{i},y_{i}) . The empirical gradient noise covariance matrix is defined as: \displaystyle\hat{\Sigma}_{N} \displaystyle:=\frac{1}{N}\sum_{i=1}^{N}g_{i}g_{i}^{\top}-g_{N}g_{N}^{\top} Note that, using independence of batch sampling, the covariance matrix for batches of size b>1 is a scalar multiple of the covariance matrix in the 1-sample ( b=1 ) case: \Sigma_{b}=\operatorname{Cov}\!\left(\frac{1}{b}\sum_{i=1}^{b}g_{i}\right)=\frac{1}{b^{2}}\sum_{i=1}^{b}\operatorname{Cov}(g_{i})=\frac{1}{b}\Sigma_{1} Since batch gradient and 1-sample gradient, as well as their noise covariance matrices, are closely related, we will only consider the 1-sample gradient and noise covariance matrix (denoted \Sigma ). Let g(\theta_{k};Y_{k},X_{k}) be the 1-sample gradient at the iterate k . The SGD update rule can be written as a drift term and a noise term: \displaystyle\Delta\theta_{k+1} \displaystyle:=\theta_{k+1}-\theta_{k}=-\eta_{k}g_{N}(\theta_{k})+\eta_{k}\xi_{k} \displaystyle\xi_{k} \displaystyle:=g_{N}(\theta_{k})-g(\theta_{k};Y_{k},X_{k}) A.2 Continuous limit of SGD with constant learning rate Let \theta_{k} be the parameter at iterate k satisfying the SGD update rule. Let B_{t} be a Wiener process. We want to understand the conditions under which the parameter \theta_{k} is the solution of the Euler-Maruyama discretization of the following SDE: \displaystyle d\theta(t)=-g_{N}(\theta,t)dt+\sqrt{\eta\Sigma(\theta,t)}dB(t) I.e., we want: \theta(\eta k)=\theta_{k} for all k and \theta(t) is a solution of the latter SDE. Iterating the SGD update rule, we have: \displaystyle\theta_{k}=\theta_{0}-\sum_{i=0}^{k-1}\eta g_{N}(\theta_{i})+\sum_{i=0}^{k-1}\eta\xi_{i} Let t=\eta k and \Delta t=\eta . We have the following: \displaystyle\theta_{\frac{t}{\eta}}=\theta_{0}-\sum_{i=0}^{\frac{t}{\eta}-1}\Delta tg_{N}(\theta_{i})+\sqrt{\eta}\sum_{i=0}^{\frac{t}{\eta}-1}\sqrt{\Delta t}\xi(\theta_{i}) For a given t we have: \displaystyle M(t) \displaystyle:=\sqrt{\eta}\sum_{i=0}^{\frac{t}{\eta}-1}\xi(\theta,t_{i})\ \ \implies\ \text{Cov}(M(t_{i}))=\sqrt{\eta}\frac{t}{\eta}\Sigma=\sqrt{\eta}t\Sigma\quad\text{if all $\xi$ have same covariance matrix} Let \mathcal{F}_{k} be a filtration adapted to \xi_{k} . Since \mathbb{E}[\xi_{k}|\mathcal{F}_{k}]=0 , the process M(t) is a martingale. If there exists a continuous, symmetric, positive semi-definite matrix \Sigma(s) such that, uniformly: \displaystyle\eta\sum_{k=0}^{\frac{t}{\eta}-1}\mathbb{E}[\xi_{k}\xi_{k}^{\intercal}|\mathcal{F}k]\to_{p}\int_{0}^{t}\Sigma(s)ds Assume that the noise \xi_{k} satisfies the Lindeberg condition: \displaystyle\forall\epsilon>0,\quad\lim_{\eta\to 0}\eta\sum_{k=0}^{\frac{t}{\eta}-1}\mathbb{E}[||\xi_{k}||^{2}1_{||\xi_{k}||>\epsilon/\sqrt{\eta}}|\mathcal{F}_{k}]=0 Under these conditions, when taking \eta\to 0 , the functional central limit theorem ensures that the martingale M(t) converges to a Wiener process with covariance \Sigma(\theta,t) and the process \theta(t) whose Euler-Maruyama discretization is \theta_{k} satisfies the SDE 1 . To model SGD with the SDE 1 , one must verify that all the conditions for applying the FCLT are satisfied and take the limit \eta\to 0 . Appendix B Derivation of the Gradient–Noise Covariance \Sigma_{lm} In this section, we derive the covariance of the per-sample gradient noise in a deep linear network (DLN). This corresponds to proposition 3.2 in the main text. Assumptions and Setup We consider a depth- L deep linear network with weight matrices W_{i}\in\mathbb{R}^{d_{i}},\ i\in\{1,...L\} and end-to-end map W:=W_{L}W_{L-1}\cdots W_{1}\in\mathbb{R}^{d_{L}\times d_{0}} The data (Y,X)\in\mathbb{R}^{d_{L}\times d_{0}} is generated from a teacher model with additive label noise: Y=MX+\xi_{q}, where • The inputs X\in\mathbb{R}^{d_{0}} are i.i.d. whitened Gaussian, X\sim\mathcal{N}(0,I_{d_{0}}) . • The teacher map is M\in\mathbb{R}^{d_{L}\times d_{0}} . • The label noise \xi_{q}\in\mathbb{R}^{d_{L}} is independent of X , with \mathbb{E}[\xi_{q}]=0 , X\perp\xi_{q} and covariance \mathbb{E}[\xi_{q}\xi_{q}^{\top}]=\Sigma_{q} . The model error is denoted \Delta:=M-W. For notational convenience, we define partial products of the student weights: W_{>l}:=W_{L}W_{L-1}\cdots W_{l+1},\qquad W_{L}=I and W_{<1}=I . We also define A_{l}:=W_{l})^{\top}. Per–sample Gradient noise For one sample (X,Y) with Y=MX+\xi_{q} and \Delta:=M-W , the prediction error is: \varepsilon:=Y-WX The per-sample squared loss is \ell=\tfrac{1}{2}\|\varepsilon\|^{2} , and its gradient w.r.t. W_{l} is g_{l}:=\nabla_{W_{l}}\ell=-W_{>l}^{\top}\varepsilon X^{\top}W_{l}^{\top}(\varepsilon X^{\top}-\Delta)W_{l}dW_{l}W_{l} \displaystyle=W_{L}...W_{l+1},\quad W_{l}^{\intercal}U^{\alpha}V^{\intercal\alpha}W_{l,t}:=W_{L,t}\cdots W_{l+1,t},\qquad W_{0. (22) This yields the quadratic covariation d[\theta]_{t}\;=\;\eta\,\Sigma(\theta_{t})\,dt\quad\Longleftrightarrow\quad d\big[\mathrm{vec}\,W^{l},\mathrm{vec}\,W^{m}\big]_{t}\;=\;\eta\,\Sigma_{lm}(\theta_{t})\,dt. (23) Quadratic covariation. For column vector semimartingales X_{t},Y_{t} with continuous paths, the quadratic covariation (or bracket) is the matrix–valued, pathwise limit [X,Y]_{t}\;:=\;\lim_{|\Pi|\to 0}\sum_{k}\big(X_{t_{k+1}}-X_{t_{k}}\big)\big(Y_{t_{k+1}}-Y_{t_{k}}\big)^{\top}, taken along partitions \Pi of [0,t] . For continuous Itô processes, [X,Y]_{t} equals the bracket \langle X,Y\rangle_{t} , and if dX_{t}=H_{X}(t)\,dB_{t} and dY_{t}=H_{Y}(t)\,dB_{t} are written against a Brownian B_{t} , then d[X,Y]_{t}\;=\;H_{X}(t)\,H_{Y}(t)^{\top}\,dt. Applying this to ( 22 ) gives ( 23 ). First and second derivatives of the mode amplitude. Consider the scalar multilinear map f(W_{1},\dots,W_{L})\;=\;u_{\alpha}^{\top}W_{L}\cdots W_{1}v_{\alpha}. Its derivative with respect to W^{l} is the matrix A_{l,\alpha}(t)\;:=\;\partial_{W^{l}}f(W_{1,t},\dots,W_{L,t})\;=\;(W_{>l,t}^{\top}u_{\alpha})\,(W_{l,t}\,H_{l}\,\big(W_{l-1,t}\cdots W_{m+1,t}\big)\,H_{m}\,W_{m,\\[2.0pt] u_{\alpha}^{\top}\,W_{>m,t}\,H_{m}\,\big(W_{m-1,t}\cdots W_{l+1,t}\big)\,H_{l}\,W_{l.\end{cases} (25) Equivalently, in the stacked, vectorized coordinates \theta=(\mathrm{vec}\,W_{1};\dots;\mathrm{vec}\,W_{L}) , let a_{l,\alpha}(t):=\mathrm{vec}\,A_{l,\alpha}(t),\qquad a_{\alpha}(t):=\big(a_{1,\alpha}(t);\dots;a_{L,\alpha}(t)\big)\in\mathbb{R}^{P}, and for l>m , using the identification \mathrm{D}^{2}_{W^{l},W^{m}}f[H_{l},H_{m}]=\text{vec}(H_{l})^{\top}\partial_{\text{vec}(W_{l}),\text{vec}(W_{m})}w_{\alpha}\text{vec}(H_{m}) we have: \nabla^{2}_{l,m}w_{\alpha}(\theta_{t})=(W_{l-1:m+1}\otimes W_{>l}^{\top}u_{\alpha})(v_{\alpha}^{\top}W_{l}^{\!\top}u_{\alpha})\,(W_{l}^{\!\top}u_{\alpha}), with W_{l}^{\!\top}u_{\alpha}\|^{2}=\prod_{j\neq l}w_{j,\alpha}^{2}. Stacked SDE and Itô formula. Let the stacked parameter SDE be d\theta_{t}=-g(\theta_{t})\,dt+\sqrt{\eta}\,\sigma(\theta_{t})\,dB_{t},\quad\sigma\sigma^{\top}=\Sigma, with \theta=(\mathrm{vec}\,W_{1};\dots;\mathrm{vec}\,W_{L}) and block covariance \Sigma=\{\Sigma_{lm}\}_{l,m=1}^{L} . By multivariate Itô for the scalar w_{\alpha}(\theta) , dw_{\alpha}=\underbrace{\Big(-\sum_{l=1}^{L}\langle A_{l,\alpha},\,g_{l}\rangle\Big)}_{\mu_{\alpha}^{\rm grad}}\,dt\;+\;\underbrace{\frac{\eta}{2}\,\mathrm{tr}\!\big(\Sigma\,\nabla^{2}w_{\alpha}\big)}_{\mu_{\alpha}^{\rm Ito}}\,dt\;+\;\underbrace{\sqrt{\eta}\,a_{\alpha}^{\top}\sigma\,dB_{t}}_{\text{diffusion}},\qquad a_{\alpha}:=(a_{1,\alpha};\dots;a_{L,\alpha}). (30) 1.  Gradient drift \mu_{\alpha}^{\rm grad} (aligned \Rightarrow modewise GF) Population gradient blocks for squared loss with whitened inputs read g_{l}=W_{>l}^{\!\top}(W-M)W_{m ; the other case is symmetric) D^{2}_{W^{l},W^{m}}w_{\alpha}[H_{l},H_{m}]=u_{\alpha}^{\top}\,W_{>l}\,H_{l}\,(W_{l-1}\cdots W_{m+1})\,H_{m}\,W_{l}^{\top} , \Delta:=M-W and C the commutation matrix. Evaluate on the aligned manifold. On alignment, \Delta|_{\alpha}=(s_{\alpha}-w_{\alpha})u_{\alpha}v_{\alpha}^{\top} . Using \mathrm{vec}(uv^{\top})=v\otimes u , C(x\otimes y)=y\otimes x , and (x\otimes y)^{\top}(A\otimes B)(u\otimes v)=(x^{\top}Au)(y^{\top}Bv) , one finds the two contractions: (i) Data–mismatch term. \sum_{l\neq m}\!\big\langle(A_{l}\!\otimes\!B_{l}\Delta)(I+C)(A_{m}\!\otimes\!B_{m}\Delta)^{\top},\;H_{lm}\big\rangle_{F}=2\,(s_{\alpha}-w_{\alpha})\sum_{l\neq m}\prod_{j\neq l,m}w_{j,\alpha}^{2}. (ii) Label–noise term. \sum_{l\neq m}\!\big\langle(A_{l}A_{m}^{\top})\!\otimes\!(B_{l}\Sigma_{q}B_{m}^{\top}),\;H_{lm}\big\rangle_{F}=\sum_{l\neq m}\!\Big(\prod_{j\neq l,m}w_{j,\alpha}^{2}\Big)\;\big\langle u_{\alpha}u_{\alpha}^{\top},\;B_{l}\Sigma_{q}B_{m}^{\top}\big\rangle. Impose balance. Since \sum_{l\neq m}\prod_{j\neq l,m}w_{j,\alpha}^{2}=L(L-1)\,w_{\alpha}^{\frac{2(L-2)}{L}} , the Itô drift becomes \mu_{\alpha}^{\rm Ito}(w_{\alpha})=\frac{\eta}{2}\,\Big[2\,(s_{\alpha}-w_{\alpha})\,L(L-1)\,w_{\alpha}^{\frac{2(L-2)}{L}}\;+\;L(L-1)\,w_{\alpha}^{\frac{2(L-2)}{L}}\,\Gamma_{\alpha}\Big], i.e. \mu_{\alpha}^{\rm Ito}(w_{\alpha})=\eta\,(s_{\alpha}-w_{\alpha})\,L(L-1)\,w_{\alpha}^{\frac{2(L-2)}{L}}\;+\;\frac{\eta}{2}\,\,L(L-1)\,w_{\alpha}^{\frac{2(L-2)}{L}}\;\Gamma_{\alpha}, which is ( 6 ). Here \Gamma_{\alpha}:=\frac{1}{L(L-1)}\sum_{l\neq m}\big\langle u_{\alpha}u_{\alpha}^{\top},\;B_{l}\Sigma_{q}B_{m}^{\top}\big\rangle. 3.  Diffusion coefficient D_{\alpha} (mode-diagonal) The scalar diffusion along mode \alpha is D_{\alpha}=\eta\,a_{\alpha}^{\top}\Sigma\,a_{\alpha}=\eta\sum_{l,m}a_{l,\alpha}^{\top}\Sigma_{lm}\,a_{m,\alpha}. Orthogonality of different modes under alignment implies D_{\alpha\beta}=0 for \alpha\neq\beta (no cross-mode diffusion). Data–mismatch part. A direct application of the vector identities yields, for each (l,m) , a_{l,\alpha}^{\top}(A_{l}\!\otimes\!B_{l}\Delta)(I_{d_{0}^{2}}+C)(A_{m}\!\otimes\!B_{m}\Delta)^{\top}a_{m,\alpha}=2\,(s_{\alpha}-w_{\alpha})^{2}\Big(\prod_{j\neq l}w_{j,\alpha}^{2}\Big)\Big(\prod_{j\neq m}w_{j,\alpha}^{2}\Big). Summing and multiplying by \eta gives D_{\alpha}^{\rm data}=2\,\eta\,(s_{\alpha}-w_{\alpha})^{2}\Bigg[\sum_{l=1}^{L}\prod_{j\neq l}w_{j,\alpha}^{2}\Bigg]^{2}. Label–noise part. Similarly, a_{l,\alpha}^{\top}\big((A_{l}A_{m}^{\top})\!\otimes\!(B_{l}\Sigma_{q}B_{m}^{\top})\big)a_{m,\alpha}=\Big(\prod_{j\neq l}w_{j,\alpha}^{2}\Big)\Big(\prod_{j\neq m}w_{j,\alpha}^{2}\Big)\,\Gamma_{lm,\alpha},\quad\Gamma_{lm,\alpha}:=\big\langle u_{\alpha}u_{\alpha}^{\top},\;B_{l}\Sigma_{q}B_{m}^{\top}\big\rangle, hence D_{\alpha}^{\rm label}=\eta\sum_{l,m}\Big(\prod_{j\neq l}w_{j,\alpha}^{2}\Big)\Big(\prod_{j\neq m}w_{j,\alpha}^{2}\Big)\Gamma_{lm,\alpha}. Impose balance. Since \sum_{l}\prod_{j\neq l}w_{j,\alpha}^{2}=L\,w_{\alpha}^{\frac{2(L-1)}{L}} , we obtain D_{\alpha}^{\rm data}=2\,\eta\,\,L^{2}(s_{\alpha}-w_{\alpha})^{2}\,w_{\alpha}^{\frac{4(L-1)}{L}},\qquad D_{\alpha}^{\rm label}=\eta\,\,L^{2}\,\overline{\Gamma}_{\alpha}\,w_{\alpha}^{\frac{4(L-1)}{L}}, where \displaystyle\overline{\Gamma}_{\alpha}:=\frac{1}{L^{2}}\sum_{l,m}\Gamma_{lm,\alpha} . Therefore D_{\alpha}(w_{\alpha})=\eta\,\,L^{2}\Big(2\,(s_{\alpha}-w_{\alpha})^{2}+\overline{\Gamma}_{\alpha}\Big)\,w_{\alpha}^{\frac{4(L-1)}{L}}, which is ( 7 ). In the isotropic case \Sigma_{q}=\sigma_{q}^{2}I one has \Gamma_{lm,\alpha}=\sigma_{q}^{2} and hence \overline{\Gamma}_{\alpha}=\sigma_{q}^{2} . Appendix F Derivation of modewise stationary law under detailed balance, proposition 5.2 The Fokker–Planck equation for the modewise density p_{\alpha}(w,t) is \partial_{t}p_{\alpha}(w,t)=-\partial_{w}\!\big(\mu_{\alpha}(w)\,p_{\alpha}(w,t)\big)+\tfrac{1}{2}\,\partial_{w}^{2}\!\big(D_{\alpha}(w)\,p_{\alpha}(w,t)\big). (31) Define the probability current J(w,t):=\mu_{\alpha}(w)\,p_{\alpha}-\tfrac{1}{2}\,\partial_{w}\!\big(D_{\alpha}p_{\alpha}\big) . Under detailed balance (zero current in steady state), J=0 and ( 31 ) reduces at stationarity to the first-order ODE \tfrac{1}{2}\,(D_{\alpha}p_{\alpha}^{\star})^{\prime}-\mu_{\alpha}\,p_{\alpha}^{\star}\;=\;0. Divide by D_{\alpha}>0 and write it in linear form (\log p_{\alpha}^{\star})^{\prime}+(\log D_{\alpha})^{\prime}=\tfrac{2\mu_{\alpha}}{D_{\alpha}} . Integrating yields ( 16 ) (up to a normalization constant). For (i) \sigma_{q}=0 : as w\uparrow s_{\alpha} , one has \mu_{\alpha}(w)=L\,w^{a}(s_{\alpha}-w)+O(\eta),\qquad D_{\alpha}(w)=2\beta\,L^{2}\,(s_{\alpha}-w)^{2}w^{2a}. Hence \tfrac{2\mu_{\alpha}}{D_{\alpha}}\sim\tfrac{c}{(s_{\alpha}-w)} with c>0 , so the exponential in ( 16 ) diverges like (s_{\alpha}-w)^{-\gamma} with \gamma>0 and the prefactor 1/D_{\alpha} contributes another (s_{\alpha}-w)^{-2} . The resulting singularity is non-integrable at s_{\alpha} , which becomes an absorbing state forcing the stationary measure to collapse to \delta(w-s_{\alpha}) . For (ii) \sigma_{q}>0 : evaluate drift, slope, and diffusion at w=s_{\alpha} . \displaystyle\mu_{\alpha}(s_{\alpha}) \displaystyle=\mu_{\alpha}^{\rm grad}(s_{\alpha})+\mu_{\alpha}^{\rm Ito}(s_{\alpha})=0\;+\;\frac{\eta}{2B}\,L(L-1)\,s_{\alpha}^{b}\,\sigma_{q}, \displaystyle\mu_{\alpha}^{\prime}(s_{\alpha}) \displaystyle=\frac{d}{dw}\Big[L\,w^{a}(s_{\alpha}-w)\Big]_{w=s_{\alpha}}+O(\eta)=-L\,s_{\alpha}^{a}+O(\eta), \displaystyle D_{\alpha}(s_{\alpha}) \displaystyle=\beta\,L^{2}\,\sigma_{q}^{2}\,s_{\alpha}^{2a}. A first-order zero of \mu_{\alpha}(w) is obtained by linearizing: 0=\mu_{\alpha}(s_{\alpha})+\mu_{\alpha}^{\prime}(s_{\alpha})(w_{\alpha}^{\star}-s_{\alpha})+O(\eta^{2}) , hence w_{\alpha}^{\star}-s_{\alpha}=-\frac{\mu_{\alpha}(s_{\alpha})}{\mu_{\alpha}^{\prime}(s_{\alpha})}+O(\eta^{2}) This gives w_{\alpha}^{\star}-s_{\alpha}=\frac{\eta}{2B}\,(L-1)\,s_{\alpha}^{-2/L}\,\sigma_{q}+O(\eta^{2}). Variance and local linearization. To compute the variance of the stationary distribution, we linearize the scalar SDE in a small neighborhood of the stable fixed point w_{\alpha}^{\star} , defined by \mu_{\alpha}(w_{\alpha}^{\star})=0 . Setting x_{t}:=w_{t}-w_{\alpha}^{\star} , the dynamics expand as dx_{t}=\mu_{\alpha}^{\prime}(w_{\alpha}^{\star})\,x_{t}\,dt+\sqrt{D_{\alpha}(w_{\alpha}^{\star})}\,dB_{\alpha,t}+O(\eta), where higher-order terms such as x_{t}^{2} or x_{t}D_{\alpha}^{\prime}(w_{\alpha}^{\star}) are O(\eta) smaller than the leading terms because x_{t}=O(\sqrt{\eta}) in the stationary regime (since D_{\alpha}=O(\eta) ). Neglecting these subleading corrections yields an Ornstein–Uhlenbeck (OU) approximation. dx_{t}=\mu_{\alpha}^{\prime}(w_{\alpha}^{\star})\,x_{t}\,dt+\sigma\,dB_{\alpha,t},\qquad\sigma^{2}:=D_{\alpha}(w_{\alpha}^{\star}). For an OU process, Itô’s formula applied to x_{t}^{2} gives d(x_{t}^{2})=2\mu_{\alpha}^{\prime}(w_{\alpha}^{\star})\,x_{t}^{2}\,dt+\sigma^{2}\,dt+2\sigma\,x_{t}\,dB_{\alpha,t}, and taking expectations at stationarity ( d\,\mathbb{E}[x_{t}^{2}]/dt=0 ) yields 0=2\mu_{\alpha}^{\prime}(w_{\alpha}^{\star}),\operatorname{Var}(x_{t})+\sigma^{2}\quad\Longrightarrow\quad\operatorname{Var}(x_{t})=\frac{\sigma^{2}}{-2\mu_{\alpha}^{\prime}(w_{\alpha}^{\star})}=\frac{D_{\alpha}(w_{\alpha}^{\star})}{-2\,\mu_{\alpha}^{\prime}(w_{\alpha}^{\star})}. Since w_{\alpha}^{\star}=s_{\alpha}+O(\eta) and D_{\alpha}^{\prime}(s_{\alpha}),\mu_{\alpha}^{\prime\prime}(s_{\alpha})=O(\eta) , evaluating at s_{\alpha} instead of w_{\alpha}^{\star} introduces only O(\eta^{2}) corrections, so \operatorname{Var}_{p_{\alpha}^{\star}}(w_{\alpha})=\frac{D_{\alpha}(s_{\alpha})}{-2\,\mu_{\alpha}^{\prime}(s_{\alpha})}+O(\eta^{2})=\frac{\beta L^{2}\sigma_{q}^{2}s_{\alpha}^{2a}}{2Ls_{\alpha}^{a}}+O(\eta^{2})=\frac{\beta L}{2}\,\sigma_{q}^{2}\,s_{\alpha}^{a}+O(\eta^{2}), with a=\tfrac{2(L-1)}{L} . \square Appendix G Offline, finite-dataset case This appendix provides a sketch of how the theory can be adapted to the finite-dataset case, as opposed to the online learning case assumed elsewhere. G.1 Finite population correction The batch is sampled without replacement from a finite dataset \mathcal{D}_{N}=(x_{i},y_{i})_{i=1}^{N} , so the covariance matrix for batch sizes 11 correspond to the rich regime [ 15 ] , and by default we use \gamma=3 to be well within the regime. ( Figure 1 uses \gamma=2 so that the times of mode learning are more concentrated and easier to visualize.) H.3 Training Several optimizers are used: • Gradient descent : a large fixed dataset is used in computing a gradient. • SGD with batch size b : a subset of batch size b is randomly selected from the data distribution at every iteration to compute the gradient (this is online SGD, which our theoretical results are about). • Isotropic Langevin dynamics : the gradient is computed as in gradient descent, and an isotropic Gaussian noise term \xi\sim\mathcal{N}(\mathbf{0},\eta\,\Delta t\,I) is added, where \eta is the learning rate and \Delta t is the discretization time step. This corresponds to an Euler-Maruyama discretization of a corresponding Langevin dynamics SDE. • Anisotropic, state-dependent noise : the gradient is computed as in gradient descent, and noise sampled using the SGD noise covariance \Sigma(\theta) is added. The covariance matrix is recomputed at each step. The noise is therefore both anisotropic (the covariance matrix is not identity) and nonhomogeneous (the covariance matrix varies along the trajectory). This corresponds to an Euler-Maruyama simulation of Equation 1 . The default learning rate used is \eta=0.005 . SGD by default uses batch size 1. For the Euler-Maruyama simulations, we use \Delta t=10^{-4}\ll\eta for accuracy. Appendix I Testing assumptions This appendix describes experiments that test how well the balance and alignment assumptions, (A1) and (A2) , hold under standard online SGD training. (a) Depth 2 (b) Depth 4 Figure 6: Measuring magnitude of cross modes over SGD training . For both a depth-2 and a depth-4 linear network, we observe that the majority of cross modes are small for most of training, except for a small number of cross modes which peak at times corresponding to modes being learned. I.1 Alignment Figure 6 examines the assumption that cross modes can be neglected. We observe that for most of training, cross modes are negligible, except for a few that peak during the intervals when modes are learned. The fact that the 0-1 cross mode has the largest peak is perhaps related to the fact that mode 1 has a visible increase while mode 0 is being learned. (a) Standard initialization, depth 2 (b) Balanced initialization, depth 2 (c) Standard initialization, depth 4 (d) Balanced initialization, depth 4 Figure 7: Measuring balance from balanced and unbalanced initialization . (a) shows that from an unbalanced Gaussian initialization, there is not strong balance at the start of training, but as training continues balance increases. (b) When we enforce balance at initialization, it is approximately maintained, and for all of training the weights are significantly more balanced than at any point in the unbalanced-initialization trajectory. (c) and (d) show that this holds for deeper (depth-4) linear networks. The unnormalized lines are the numerator of the expression for r_{l} , and show little change. I.2 Balance We also test how well balance holds under non-balanced initializations. For this we track, for each non-final layer l , r_{l}:=\frac{\lVert W_{l}W_{l}^{\top}-W_{l+1}^{\top}W_{l+1}\rVert_{F}}{\lVert W_{l}W_{l}^{\top}\rVert_{F}+\lVert W_{l+1}^{\top}W_{l+1}\rVert_{F}}, (33) where small values of r_{l} correspond to approximately balanced weights (normalized to be invariant to weight scaling). Figure 7 shows that from non-balanced initializations, balance increases over SGD training of both a 2-layer and 4-layer linear network, but never reaches a level comparable to when balance is enforced at the start of training. To give a sense of the scale of the (im)balance, we also plot the unnormalized numerator of Equation 33 and also the Frobenius norm of each layer in Figure 8 . This reveals that the increase in balance from unbalanced initialization is mostly an increase relative to the increasing magnitude of the weights, and not an increase in absolute terms. (a) Standard initialization, depth 2 (b) Balanced initialization, depth 2 (c) Standard initialization, depth 4 (d) Balanced initialization, depth 4 Figure 8: Frobenius norms of weight matrices over SGD training . Frobenius norms of layers increase over training, accounting for most of the decrease in normalized balance measure r_{l} from standard initializations. Appendix J Varying hyperparameters This appendix shows the effect of varying hyperparameters on our experimental results connecting mode learning with modewise diffusion. J.1 Learning rate Figure 9 shows that the maximum diffusion along modes scales approximately linearly with the learning rate, in agreement with the functional form derived in Proposition 3.5 . Figure 9: Learning rate versus maximum diffusion for each mode. We observe a linear relationship between learning rate and the diffusion along each mode. J.2 Batch size Figure 10 shows that the maximum diffusion along modes scales approximately linearly with the reciprocal of the batch size, in agreement with the functional form derived in Proposition 3.5 . Figure 10: Batch size versus maximum diffusion for each mode. We observe a linear relationship between the reciprocal of batch size and the diffusion along each mode. J.3 Finite dataset This section shows that the results about the diffusion over time J.4 DLN architecture So far, all experiments shown are with a rectangular DLN architecture (rectangular means that all the weight matrices are square). Figure 11 shows that the formula in Proposition 3.5 for the modewise diffusion is also similar to what is observed in a DLN with hidden layers of size 24, double the size of the input and output (12). Figure 11: Empirical versus theoretical modewise diffusion for a non-rectangular DLN. The structure of the diffusion, including the location of the peaks and tending to zero once the mode is learned is maintained. Appendix K Discretization error and end-of-training distribution This appendix simulates the SDE in Equation 1 with different values of \Delta t for Euler-Maruyama. Smaller values of \Delta t correspond to lower discretization error. At lower levels of discretization error, we observe that the end-of-training mode distribution has lower variance ( Figure 12 ) and is thus closer to the prediction of Proposition 5.2 . This provides weak evidence in favor of our explanation for the mismatch between the prediction of Proposition 5.2 and the experiments shown in Figure 4 . Figure 12: Variance of the end-of-training distribution of the first mode versus simulation fineness. We vary the fineness (i.e., the \Delta t parameter) of the Euler-Maruyama simulation of the stochastic gradient flow SDE ( Equation 1 ). For finer simulations, the variance reduces.