Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model Yizhou Xu 1 , Pierfrancesco Beneventano 2 , Isaac Chuang 2 , Liu Ziyin 2,3 1 École Polytechnique Fédérale de Lausanne 2 Massachusetts Institute of Technology 3 NTT Research Abstract A large body of theory and empirical work hypothesizes a connection between the flatness of a neural network’s loss landscape during training and its performance. However, there have been conceptually opposite pieces of evidence regarding when SGD prefers flatter or sharper solutions during training. In this work, we partially but causally clarify the flatness-seeking behavior of SGD by identifying and exactly solving an analytically solvable model that exhibits both flattening and sharpening behavior during training. In this model, the SGD training has no a priori preference for flatness, but only a preference for minimal gradient fluctuations. This leads to the insight that, at least within this model, it is data distribution that uniquely determines the sharpness at convergence, and that a flat minimum is preferred if and only if the noise in the labels is isotropic across all output dimensions. When the noise in the labels is anisotropic, the model instead prefers sharpness and can converge to an arbitrarily sharp solution, depending on the imbalance in the noise in the labels spectrum. We reproduce this key insight in controlled settings with different model architectures such as MLP, RNN, and transformers. 1 Introduction The flatness and sharpness of the loss landscape have become a major object of study for understanding the optimization and generalization of neural networks ( 27 ; 33 ) , not to mention various algorithms based on these two mechanisms ( 22 ) . For example, flatness is one of the most common hypotheses for why neural networks generalize: even when many parameter settings fit the training data perfectly, solutions that lie in flatter regions of the loss landscape are often argued to be more stable and to generalize better. This (rather questionable) intuition has motivated a large literature connecting SGD’s step size and stochasticity to an implicit preference for flatter minima, typically operationalized through Hessian-based sharpness measures 33 ; 49 ; 52 ; 50 . At the same time, an apparently contradictory phenomenon has become increasingly prominent in modern training: sharpness often increases over the course of optimization, and eventually reaches the edge of stability (EoS), where the curvature is large, and training is only marginally stable 14 ; 4 . This coexistence of the two tendencies “SGD seeks flatness” and “SGD seeks sharpness” is what we will call the sharpness “paradox" – of course, this is not a paradox in the literal sense, but a reflection of our lack of understanding about whether SGD has an intrinsic tendency towards learning flatter or sharper solutions. Resolving this “paradox" is quite difficult, especially because of the existence of many confounding variables: sharpness trajectories are confounded by initialization (a flatter start will lead to sharpening and vice versa) and by run-to-run variability. In this work, we give an exact solution of the sharpness to a mathematical model under the so-called “minimal-fluctuation" constraint, which allows us to strip away these confounders and identify the precise cause of the flattening and sharpening within this model. In this model, the global minima form a highly degenerate manifold containing solutions with a wide range of sharpness values – including arbitrarily sharp ones – yet the learning dynamics of SGD selects a solution whose sharpness is unique. This lets us answer the implicit-bias question in its cleanest form: when many equally good minima exist, which curvature does SGD pick, and what factors determine that choice? Our main contributions are: 1. In the regime where SGD minimizes fluctuations, we identify an exactly solvable model of sharpness / flatness; 2. We precisely identify the causes of flatness-seeking behavior of SGD in this model: (a) When the label uncertainty is isotropic, the training always prefers the flattest of all solutions; (b) Otherwise, the label uncertainty spectrum \Sigma_{\epsilon} determines the converged sharpness, which is inversely proportional to the condition number of \Sigma_{\epsilon} . These predictions are tested in nonlinear models and realistic data distributions, showing that the messages carry beyond the analytical model we solved. The starting point of our theory is the recent theoretical advancement that shows that SGD training has a intrinsic preference for solutions that have a small gradient fluctuation 44 ; 46 ; 11 ; 59 , and our empirical success of in explaining real neural networks lends further support to the minimal-fluctuation theory. In the minimal-fluctuation theory, the crucial point -- and the source of the paradox -- is that fluctuation and sharpness are not the same object. They align only under special conditions, so minimizing fluctuation does not generally imply minimizing sharpness. The exact model we solve is a deep linear network trained on a linear teacher with noisy labels. In this model, the SGD-selected sharpness can be written explicitly in terms of the input covariance, the teacher map, the depth, and the label-noise 1 1 1 To clarify, throughout our work, we use the word “label noise” in the broad sense. It could mean a dynamically injected noise or could be an inherent static noise in the labels such that the model cannot perfectly interpolate the data. covariance across output dimensions. The key consequence of the theory is to reframe the sharpness debate in data-geometric terms: whether SGD appears flatness-seeking or sharpening with respect to GD is situation-dependent, and the controlling knob is the anisotropy of effective noise on the labels. Figure 1 : We reproduce the experiment from 58 , where a deep matrix factorization problem trained with SGD converges to the same sharpness, even if the initial sharpness depends strongly on the initialization scale. The theoretical line is the prediction of Theorem 1 . 2 Sharp or flat? Sharpness. A long-standing consensus in deep learning theory suggests that the implicit bias of Stochastic Gradient Descent (SGD) drives the model toward flat minima. This line of thought 27 ; 33 ; 22 argues that flat regions are more robust to parameter perturbations, which translates to superior generalization. A vast body of theoretical work seeks to explain such an implicit bias from the perspectives of gradient noise and dynamical stability 56 ; 49 ; 51 ; 50 ; 40 . However, recent research into the discrete dynamics of neural network training reveals that SGD does not always settle into the flattest possible regions. Instead, it often converges to regions that are significantly “sharp", a phenomenon known as the progressive sharpening 53 ; 29 ; 30 ; 14 ; 4 . Thus, the preference of SGD regarding the sharpness remains a paradox. Arbitrariness of initialization. However, due to the large number of confounding variables, it is difficult to say and understand when a flatter minimum or a sharper minimum is preferred. The first confounding variable is the arbitrariness of the solution. It is common for us to measure the evolution of sharpness over training steps and argue that the model prefers a sharper or flatter solution by looking at how sharpness changes during training. However, this perspective is biased by the arbitrariness of the initialization scheme we use. Imagine a landscape where the global minimum has a unique, nonzero level of sharpness, but initializations can have arbitrary levels of sharpness. Then, conditioning on the training trajectories that take the model to the global minimum, a flatter initialization implies that the training must, on average, sharpen the landscape, while a sharper initialization must, on average, flatten. In this case, it is difficult to tell whether the learning dynamics favor flatness or sharpness, as this largely depends on the initialization’s arbitrariness. The second confounding variable is the potential randomness of the outcome. It is also fully possible that every initialization (or sampling process) leads to a different local minimum with different values of sharpness, and it becomes difficult to compare across settings. Thus, ideally, the ideal theoretical model should remove this arbitrariness. Identifying a theoretical model where the models only converge to unique levels of sharpness is another great step towards understanding the flatness-seeking behavior. See Figure 1 for a demonstration of this effect. Intrinsic sharpness. Therefore, one can imagine the following scenario being the best for understanding the flatness-seeking behavior of learning algorithms: (a) construct a loss landscape with a degenerate global minima manifold, containing solutions with different levels of sharpness; (b) condition on the learning trajectories reaching a global minimum, and study how the learning dynamics determine the average level of sharpness. In this scenario, there are clearly two contributions to the sharpness of the learned solution: (a) the intrinsic sharpness required to learn the problem: for any given task and any given architecture, the minimal level of sharpness required to learn it should be nonzero; (b) the learning-algorithm-biased sharpness: when there is a manifold of solutions with different levels of sharpness values, which one among these is chosen by SGD. Conceptually, we can write (using T to denote some metric of sharpness): T=T_{\rm intrinsic}+\Delta_{\rm learning}. (1) The first term is present in any learning algorithm that can reach a global minimum, and so whatever effect that only changes the first term cannot be argued to be a “preference" of the learning algorithm. The second term is purely determined by the preference learning algorithm and the choice of its hyperparameters. What affects the second term can thus be called the “implicit flatness bias." Of course, it is equally important to study what determines the first term, which we will also answer in our model. Novel SGD-specific sharpening. Prior work on progressive sharpening and the edge-of(-stochastic)-stability documents that curvature can increase during training, but the underlying mechanism is fundamentally landscape-driven : analogous sharpening is observed for GD and Gradient Flow (GF) as well, and interventional studies attribute the effect to landscape slopes pointing towards sharper areas, rather than to the SGD-specific stochasticity ( 30 ; 14 ; 13 ; 4 ) . Our mechanism is qualitatively different. By contrast, SGD does induce a selection rule: it approximately minimizes the entropic objective ( 12 ), and therefore selects minimizers of the gradient-fluctuation functional S(\theta) over the minimizer manifold (Lemma 2 ). GF/GD are insensitive to the label-noise covariance \Sigma_{\epsilon} (they depend on label first moments, not on second moments), while SGD is not. In our solvable model, the minibatch-induced term selects among a degenerate manifold of global minima by minimizing gradient fluctuations, and this selection depends explicitly on the anisotropy of \Sigma_{\epsilon} . As a consequence, SGD can move along the minimizer manifold toward solutions that are strictly sharper—and can be arbitrarily sharp—even though much flatter global minima coexist. This is not “sharpening on the way to convergence”, but a noise-conditioned implicit bias at convergence, see also Fig. 4 . In our model, we will find that \Delta_{\rm learning} is the SGD-specific contribution arising from fluctuation minimization (vanishing in the isotropic-noise case and scaling with \kappa(\Sigma_{\epsilon}) in our model; Cor. 1 ). Consistent with this interpretation, a direct intervention that switches from SGD to full-batch GD arrests the sharpening drift (Fig. 4 ). Overall, this challenges, if not overturns, the common heuristic “SGD seeks flatness” narrative: in our model, SGD primarily seeks minimal fluctuations, and flatness emerges only in the contrived regimes where fluctuation and sharpness are aligned. 3 An exactly solvable model In this section, we first motivate and formally define the problem setting, and we then present the main theorem, whose proof is given in the appendix. Readers wishing to learn more about the intuition may start by reading the general theoretical framework outlined in Section 5 . We will discuss the various implications of the theory in the next main section. Notation: We use \theta to denote the set of all trainable parameters. When used without a subscript, \nabla denotes the gradient with respect to \theta . \lambda_{\max}(\Sigma) denotes the maximal eigenvalue of \Sigma . \nabla^{2}z denotes the Hessian matrix of a scalar function z . Throughout this work, we use \mathbb{E} to denote the average over the training set \mathcal{D}_{\rm train} , which is not necessarily a finite set. I_{d} is the identity operator for \mathbb{R}^{d} , and d is often omitted when the context is clear. Other notations are introduced in the context. 3.1 Model and loss function We study a D -layer deep linear network. Specifically, the model is defined as f_{\theta}(x)=W_{D}\cdots W_{1}x. (2) where \theta:=\{W_{k}\}_{k=1}^{D} denotes the collection of all the weight matrices, and x is the input data. The model is trained with the MSE loss on a regression task, where the data-wise stochastic loss is \ell(x,y,\theta)=\frac{1}{2}||y-W_{D}\cdots W_{1}x||^{2}, (3) where y is the label associated with x . In practice, we train this model with SGD, but the training dynamics of SGD on this model are difficult to solve analytically, and we thus adopt an alternative approach, which can be seen as answering the question of “if the training proceeds with SGD, then what kind of objective function is the algorithm actually minimizing?" This alternative problem, in contrast, can be exactly solved to its full generality. This alternative objective function will be discussed in Section 3.3 . The labels are generated by a linear teacher V with noise \epsilon y=Vx+\epsilon\in\mathbb{R}^{d_{y}}, (4) where \mathbb{E}[x]=0 and \mathbb{E}[\epsilon]=0 . We define the covariances \Sigma_{x}:=\mathbb{E}_{x}[xx^{T}] and \Sigma_{\epsilon}:=\mathbb{E}_{\epsilon}[\epsilon\epsilon^{T}] . Note that all input, label, V , and noise are allowed to have arbitrary dimensions as long as their dimensions are mutually consistent. Under this data generation model, the global minima of the empirical loss L satisfy: W_{D}...W_{1}x=Vx (5) for all x in the training set. As a key part of the analysis, note that the model has many symmetries (namely, loss-invariant directions), which we call the matrix rescaling symmetry. For an arbitrary invertible matrix A , and an arbitrary layer index i , we have \displaystyle f(W_{i},W_{i-1}) \displaystyle:=W_{D}\cdots W_{i}W_{i-1}\cdots W_{1}x (6) \displaystyle=f(W_{i}A,A^{-1}W_{i-1}). (7) This also implies that there are infinitely many global minima for the deep linear network. As we will show in the next section, this implies that a global minimum for L exists with arbitrarily high sharpness. 3.2 Sharpness For most of the manuscript, we use the following metric of sharpness: Definition 1 . The sharpness T(\theta):={\rm Tr}\mathbb{E}[\nabla^{2}\ell(x,y,\theta)] . T has been used in many prior works as a metric of sharpness and is often found to better represent the stability of the learning dynamics 35 ; 50 ; 58 , which is one of the primary reasons for studying the sharpness of the landscape. It also offers an upper and lower bound to the largest eigenvalue \lambda_{\max} of the Hessian: T/d\leq\lambda_{\max}\leq T. (8) Thus, T(\theta) serves as a reliable proxy for the stability of the solution. In the asymptotic limit where T\to\infty , \lambda_{\max} must also diverge. The necessity of studying SGD’s implicit bias is highlighted by the fact that the loss landscape contains paths to arbitrarily sharp minima. The following lemma shows that for the class of symmetries called exponential symmetries 58 , we can always find a global minimum with infinite sharpness: Lemma 1 . Assume that A is a symmetric matrix and that for any x,y,\theta , \ell(x,y,e^{\lambda A}\theta)=\ell(x,y,\theta) . Moreover, assume that A\mathbb{E}_{x,y}\nabla^{2}\ell(x,y,\theta)\neq 0 . Then, \limsup_{|\lambda|\to+\infty}|T(e^{\lambda A}\theta)|=+\infty . The matrix rescaling symmetry is a type of exponential symmetry as well and obeys Lemma 1 . That symmetry implies the existence of sharp solutions is first identified and proved in 18 for the case of simple rescaling symmetries, and lemma 1 can be seen as the most general form of this result. As an example of its meaning, consider its application to the deep linear network with D=2 . For \ell(x,y,\theta)=\frac{1}{2}||y-W_{2}W_{1}x||^{2} , we have T(W_{2},W_{1})=||W_{2}||_{F}^{2}{\rm Tr}\Sigma_{x}+d_{y}\text{Tr}(W_{1}^{T}W_{1}\Sigma_{x}), (9) In this case we have \ell(x,y,W_{2},W_{1})=\ell(x,y,\lambda W_{2},\lambda^{-1}W_{1}) but \lim_{|\lambda|\to+\infty}T(\lambda W_{2},\lambda^{-1}W_{1})=+\infty (10) for any W_{1},W_{2} . As the main theorem 1 below will prove, at the global minima, the deep linear network will (1) have a unique minimal sharpness value T_{\min} that only depends on the problem and will (2) converge to a unique sharpness value T\geq T_{\min} , this means that the actually learned level of sharpness can be decomposed into two contributions: T=T_{\min}+\Delta_{\rm SGD}, (11) where \Delta_{\rm SGD}\geq 0 , T_{\min} is the minimal level of sharpness that is required in order to fit the data distribution (given the architecture) and needs to be obeyed by any learning algorithm that is capable of reaching the global minimum, and \Delta_{\rm SGD} is the level of sharpness due to the implicit bias of SGD (namely, due to the fluctuation minimization effect that we will introduce in the next section). The more standard quantity \lambda_{\max} is also of interest because it directly relates to the recent popular problem of edge of stability (EoS). In general, while our theory allows one to compute \lambda_{\max} directly, its functional form is so complicated that we do not present it explicitly 2 2 2 Note that this is not a problem of our theory, but a problem of generally computing the largest eigenvalue of any matrix. . That being said, we will discuss in Section 4 a few cases where \lambda_{\max} can indeed be written in easily interpretable form. Figure 2 : Non-isotropic noise in the labels leads to progressive sharpening. Left : linear networks, with the minimal sharpness predicted by ( 15 ). Middle : Two layer ReLU networks trained under a teacher-student setting. Right : Two layer ReLU networks trained on MNIST. Each line is averaged over five trials shown with the standard error. See Appendix B for experiment details and more experiments. Figure 3 : Across all four models, and for both regression and classification tasks, we observe that the learning process yields a sharper solution as the noise in the labels becomes increasingly non-isotropic. We define relative sharpness as the ratio of the Hessian trace under non-isotropic noise to that under isotropic noise. Left : MSE loss. Right : Cross entropy loss. 3.3 Minimal Fluctuation Ansatz A line of works has shown that training with SGD at a finite learning rate and minibatch sampling is equivalent to solving the following perturbed optimization problem, which we refer to as the entropic loss 44 ; 46 ; 11 ; 59 : \displaystyle F_{\eta}(\theta) \displaystyle=\underbrace{\mathbb{E}_{x,y}\ell(x,y,\theta)}_{\text{learning, symmetry}}+\underbrace{\frac{\eta}{4}\mathbb{E}_{\mathcal{B}}||\mathbb{E}_{(x,y)\in\mathcal{B}}\nabla\ell(x,y,\theta)||^{2}}_{\text{due to discretization error, noise, $:=S(\theta)$}}, (12) where \mathcal{B} denotes a mini-batch and \eta is the learning rate. Using this modified entropic loss allows us to characterize the implicit bias of SGD via the following lemma. Lemma 2 (Minimal Fluctuation Lemma, informal) . In the limit \eta\to 0^{+} , the global minima of F_{\eta}(\theta) are \arg\min_{\theta\in\mathcal{M}}S(\theta), (13) where M=\{\theta\mid\mathbb{E}_{x,y}\ell(x,y,\theta)=\inf_{u}\mathbb{E}_{x,y}\ell(x,y,u)\} is the the global minima manifold. Since modern neural networks are typically over-parameterized, the loss landscape \mathbb{E}_{x,y}\ell(x,y,\theta) contains a manifold of global minima. The lemma suggests that SGD does not treat these minima equally but implicitly prefers those with minimal fluctuation S . Notably, this bias vanishes for full-batch gradient descent (GD), as the fluctuation term disappears at any global optimum. A key mechanism of the proof is that different symmetry directions along the matrix rescaling symmetry actually have different levels of fluctuation (often also corresponding to different levels of sharpness, as shown by Lemma 1 ) – thus, the preference of the learning algorithm for the minimal-fluctuation solution causes the learning algorithm to also moves towards a certain level of sharpness. 3.4 Main Result Let V^{\prime}:=\sqrt{\Sigma_{\epsilon}}V\sqrt{\Sigma_{x}} . Let its SVD decomposition be given by V^{\prime}=LSR . We assume the network width d\geq\text{rank}(V^{\prime}) 3 3 3 d is defined as the the smallest dimension of the output and input dimensions of W_{i} for all i\in[D] . When d<\text{rank}(V^{\prime}) , it learns the largest d eigenvalues, and the result is similar. . Under the SGD limit \eta\to 0^{+} , we obtain the following closed-form result for the sharpness: Theorem 1 . For all global minimum of ( 12 ), the spectrum of the Hessian is identical (determined by \Sigma_{x},\Sigma_{\epsilon} and V ), and the sharpness is given by \displaystyle T(\theta)=(D-1)\left({\rm Tr}S\right)^{(D-2)/D}({\rm Tr}\Sigma_{\epsilon}{\rm Tr}\Sigma_{x})^{1/D}{\rm Tr}[\Sigma_{\epsilon}^{-1}LSL^{T}]+d_{y}\left({\rm Tr}S\right)^{2(D-1)/D}\frac{({\rm Tr}\Sigma_{x})^{1/D}}{({\rm Tr}\Sigma_{\epsilon})^{(D-1)/D}}. (14) In addition, the minimal sharpness of the global minimum of \mathbb{E}_{x}\ell(x,\theta) is given by T_{\min}:=\min_{\theta}T(\theta)=D({\rm Tr}\hat{S})^{2(D-1)/D}(d_{y}{\rm Tr}\Sigma_{x})^{1/D}, (15) where \hat{S} is the singular values of V\sqrt{\Sigma_{x}} . Theorem 1 reveals a striking property: although there exist infinitely many global minima with varying (and even infinite) sharpness, SGD consistently selects a solution with a unique sharpness level. This suggests that the final sharpness is independent of initialization, provided the model converges to a global optimum. Moreover, the first term of ( 14 ) suggests that D-1 layers contribute to the sharpness in the same way, while the last layer (the second term of ( 14 )) contributes differently. At the minimal sharpness solution ( 15 ), however, all layers contribute in the same way. We will analyze the implications of this theorem in detail in Section 4 . 4 Insights A great merit of exactly solvable models is that every effect and phenomenon within the model can be precisely and causally understood. This section is thus devoted to naming and understanding the key observations and insights of Theorem 1 . We then present experiments on a diverse set of model architectures to demonstrate the qualitative relevance of the prediction. The proofs of the theoretical results are also presented in the appendix. Noise Imbalance as a Source of Sharpness. Corollary 1 . Suppose \Sigma_{\epsilon} and V\sqrt{\Sigma_{x}} have full rank. Let the condition number of \Sigma_{\epsilon} be \kappa(\Sigma_{\epsilon}):=\frac{\lambda_{\max}(\Sigma_{\epsilon})}{\lambda_{\min}(\Sigma_{\epsilon})} . Then, c_{1}D\sqrt{\kappa(\Sigma_{\epsilon})}\leq T(\theta)\leq c_{2}D\kappa(\Sigma_{\epsilon}) (16) at the global minimum of ( 12 ), where c_{1},c_{2} are constants depending on V\sqrt{\Sigma_{x}} and d_{x},d_{y} . Corollary 1 suggests that the sharpness mainly depends on the imbalance of noise in the labels, and does not depend much on the depth. As an illustrative special case, we can choose \Sigma_{x}=I , V=I ( d_{x}=d_{y}=d ), which gives V^{\prime}=\sqrt{\Sigma_{\epsilon}}=LSL^{T} , and thus \displaystyle T(\theta) \displaystyle=(D-1)({\rm Tr}[\Sigma_{\epsilon}^{1/2}])^{(D-2)/D}(d{\rm Tr}[\Sigma_{\epsilon}])^{1/D}{\rm Tr}[\Sigma_{\epsilon}^{-1/2}]+d^{(D+1)/D}{\rm Tr}[\Sigma_{\epsilon}^{1/2}]^{2(D-1)/D}{\rm Tr}[\Sigma_{\epsilon}]^{-(D-1)/D}, (17) which can be arbitrarily large. On the other hand, we have \min T(\theta)=Dd^{(D+2)/D} , which does not depend on \Sigma_{\epsilon} . Thus, the imbalance of the noise spectrum can lead to arbitrarily high sharpness. If we further choose \Sigma_{\epsilon}=\text{diag}(1,\sigma^{2}) , we have \displaystyle T(\theta) \displaystyle=(D-1)2^{1/D}(1+\sigma)^{(D-2)/D}(1+\sigma^{2})^{1/D}(1+1/\sigma)+2^{(D+1)/D}(1+\sigma)^{2(D-1)/D}(1+\sigma^{2})^{-(D-1)/D}. (18) One can check that it is symmetric in terms of \sigma\to\sigma^{-1} and has a unique minimum at \sigma=1 . We approximately have T_{\min}\approx D and \frac{T(\theta)}{T_{\min}}\approx\max(\sigma,1/\sigma) (19) for large D and very imbalanced noise in the labels. Thus, the gap between the sharpness of SGD and the minimal sharpness is approximately proportional to the imbalance of the noise in the labels and independent of the depth, as demonstrated by Corollary 1 . This could provide a theoretical explanation for the sharpening behavior observed in transformers. In token prediction, "stop words" (e.g., "the") have very low conditional entropy, while content words (nouns/verbs) have high entropy. This massive imbalance in token-level noise naturally drives learning toward sharper solutions. For the same reason, when there is class imbalance, the noise in the labels also tends to be imbalanced, and this will sharpen the solution. See Appendix A.10 for an example of the power-law covariance. Recovery of Minimal Sharpness under Isotropic Noise. Corollary 2 . The global minimum is the flattest if and only if \Sigma_{\epsilon}=I . The reason behind Corollary 2 is that for deep linear networks with \Sigma_{\epsilon}=I , the sharpness is exactly proportional to the fluctuation (the second order derivative in ( 24 ) disappears for linear networks), and thus minimizing fluctuation is equivalent to minimizing sharpness. Largest Eigenvalue Has a Similar Property Corollary 3 . Assume that VV^{T} commutes with \Sigma_{\epsilon} and V^{T}V commutes with \Sigma_{x} . Then, c_{1}D\sqrt{\kappa(\Sigma_{\epsilon})}\leq\lambda_{\max}(\mathbb{E}[H])\leq c_{2}D\kappa(\Sigma_{\epsilon}), (20) where H is the Hessian at the global minimum of ( 12 ) and c_{1},c_{2} are constants depending on V , \Sigma_{x} and d_{y} . Corollary 3 suggests that in a special case where all the matrices are aligned, the maximal eigenvalue of the Hessian at the global minimum of ( 12 ) is mainly determined by the noise imbalance and not by the depth. We expect that this should be true in general. For an illustrative example, we can choose V=\Sigma_{x}=I , \Sigma_{\epsilon}=\text{diag}(1,\sigma^{2}) . Then, we can obtain \displaystyle\lambda_{\max}(\mathbb{E}[H])=(D-2)2^{1/D}(1+\sigma)^{-2/D}(1+\sigma^{2})^{1/D}\max(1,\sigma^{-1})\max(1,\sigma) (21) \displaystyle+2^{1/D}(1+\sigma)^{(D-2)/D}(1+\sigma^{2})^{-(D-1)/D}\max(1,\sigma)+2^{-(D-1)/D}(1+\sigma)^{(D-2)/D}(1+\sigma^{2})^{1/D}\max(1,\sigma^{-1}). according to Theorem 72 . One can check that it is symmetric in terms of \sigma\to\sigma^{-1} and reaches its minimum at \sigma=1 , which is the maximal eigenvalue of the minimal sharpness solution. We also have \lambda_{\max}(\mathbb{E}[H])\approx D\max(\sigma,\sigma^{-1}) (22) for large D and \sigma\gg 1 or \sigma^{-1}\gg 1 . Therefore, the maximal eigenvalue scales similarly to the trace of the Hessian and grows as the noise becomes imbalanced. Figure 4 : Progressive sharpening with non-isotropic noise in the labels is unique for SGD. The red dotted line denotes the time when we switch to GD. Progressive sharpening or flattening. Theorem 1 suggests that SGD converges to a sharpness level that is independent of the initialization. Consequently, progressive sharpening (at least on average and throughout the trajectory) occurs if the initialization is sufficiently small, while progressive flattening occurs if it is large. In Proposition 68 , we characterize the initial sharpness and observe that it remains independent of the labels for standard initialization schemes. Comparing Proposition 68 with Theorem 1 shows many mechanisms of progressive sharpening. Noting that the initial sharpness is independent of the label or noise distribution, while the final sharpness is label and noise-dependent, one finds a simple and straightforward mechanism for introducing flatening and sharpening: for a fixed initialization, rescaling the labels y to be smaller causes a flattening, while rescaling y to be larger can cause sharpening. Again, this showcases the key message: SGD intrinsically may not have a preference for sharpness, and it is the data distribution that changes, if not determines, the sharpness. This is a new mechanism of progressive sharpening. The previous known mechanisms were about gradient flow and full batch gradient descent, as we mentioned in Section 2 . The progressive sharpening that our theory discovers is not. Indeed, it is about noise covariance, and can not be seen in a deterministic training pipeline, as in the case of ( 53 ; 29 ; 30 ; 14 ; 13 ; 4 ) . In fact, the second term of ( 12 ) is always zero at any local minimum for GD, and does not lead to progressive sharpening. This is demonstrated in Figure 4 . Moreover, our progressive sharpening mechanism is not linked to the edge of stochastic stability and instability in general; it can be observed even with vanishing learning rates. Experiments. To empirically validate our theory, we perform experiments on both deep linear networks and several nonlinear architectures (MLP, RNN, and Transformer) across regression and classification tasks. These results, shown in Figures 2 and 3 , demonstrate that the observed phenomena are not unique to linear models; specifically, imbalanced label noise induces sharper solutions as predicted by Corollary 1 . Furthermore, the left side of Figure 2 verifies Corollary 2 , where linear networks converge to the minimal sharpness solution under isotropic label noise. In Figure 3 , we observe that sharpness increases with the non-isotropy of label noise. The plateau at high condition numbers is a consequence of the finite-horizon effect, where highly imbalanced noise necessitates a very long convergence time to reach the sharpest minima. To isolate the role of stochasticity, Figure 4 presents a direct intervention: when switching to GD in the linear network setting, progressive sharpening disappears. This identifies the mechanism as unique to SGD, aligning with ( 12 ) where the second term vanishes for GD at local minima, exerting no implicit bias. Comprehensive details and additional experiments on eigenvalues, condition numbers, and Transformers can be found in Appendix B . 5 General Theory As discussed, while many studies suggest SGD promotes flatness, recent observations of EoS behaviors indicate that SGD can also converge to remarkably "sharp" solutions. Our theory is that this difference in tendency arises from a subtle distinction between sharpness and fluctuation. In this section, we look at this problem from a general perspective and emphasize how our result leads to improved understanding here. Consider the Mean Squared Error (MSE) loss 4 4 4 Similarly, one can extend the discussion to a generic loss function by looking at the imbalance in the spectrum of the label Hessian: \nabla_{y}^{2}\ell . \ell(x,\theta):=\frac{1}{2}||f(x,\theta)-y(x)||^{2}. (23) The Hessian is \displaystyle H(\theta):=\mathbb{E}_{x}\nabla^{2}\ell(x,\theta)=\mathbb{E}_{x}\left(\nabla f(\theta,x)\nabla f(\theta,x)^{T}+\sum_{j=1}^{d_{y}}[e(x,\theta)]_{j}\nabla^{2}_{\theta}[f(x,\theta)]_{j}\right), (24) where e(x,\theta):=y(x)-f(\theta,x) is the residual error. Near a global minimum, e(x,\theta) becomes negligible, allowing us to approximate the sharpness (quantified by the trace of the Hessian) as {\rm Tr}H(\theta)\approx\mathbb{E}_{x}||\nabla f(\theta,x)||^{2}, (25) Such an approximation is well known in existing literature 50 ; 38 . In contrast, rather than the sharpness, recent works showed that SGD minimizes the following gradient fluctuation: \mathbb{E}_{x}||\nabla\ell(x,\theta)||^{2}=\mathbb{E}_{x}||\nabla f(\theta,x)^{T}e(x,\theta)||^{2}, (26) Assuming that the model is well-trained, the residual e(x,\theta) primarily reflects noise in the labels. If we treat this noise as a random variable with covariance \Sigma_{\epsilon} independent of x , the fluctuation becomes: \mathbb{E}_{x}||\nabla\ell(x,\theta)||^{2}=\mathbb{E}_{x}[\nabla f(x,\theta)^{T}\Sigma_{\epsilon}\nabla f(x,\theta)], (27) which is linked but not identical to the sharpness. Theorem 2 . Assuming that the residue e(x,\theta):=y(x)-f(\theta,x) is a random variable with zero mean, variance \Sigma_{\epsilon} , and independent of the dataset. Then, \lambda_{\min}(\Sigma_{\epsilon})S\leq T\leq\lambda_{\max}(\Sigma_{\epsilon})S, (28) where T is the sharpness and S is the gradient fluctuation. Proof. By ( 24 ) we have T:={\rm Tr}H(\theta)=\mathbb{E}_{x}||\nabla f(\theta,x)||^{2}, (29) where we use the assumption that e(x,\theta) is independent of x and has zero mean. By ( 27 ) we have S:=\mathbb{E}_{x}||\nabla\ell(x,\theta)||^{2}={\rm Tr}[\Sigma_{\epsilon}\mathbb{E}_{x}[\nabla f(x,\theta)\nabla f(x,\theta)^{T}]]. (30) The proof is complete by combining them. ∎ This inequality makes it precise when minimizing fluctuation is not the same as minimizing the flatness. Crucially, the fluctuation is only a good approximation of the sharpness when the noise in the labels is isotropic. When \frac{\lambda_{\max}(\Sigma_{\epsilon})}{\lambda_{\min}(\Sigma_{\epsilon})} is large, the bound becomes very loose and minimizing fluctuation no longer leads to minimizing flatness. 6 Related Works Existing minimal models of sharpness and edge of stability. Following the initial empirical discovery of progressive sharpening and the Edge of Stability (EoS) effect in GD and SGD ( 53 ; 29 ; 30 ; 14 ; 4 ; 15 ) , considerable effort has been devoted to analyzing these phenomena through minimal models. 1 analytically demonstrated progressive sharpening in a quadratic model as a direct consequence of EoS. 36 showed progressive sharpening on a shallow network with tanh-activation. 17 attributed these effects to the influence of cubic terms in the Taylor expansion of the loss function. Our work departs from these studies by demonstrating that such phenomena can arise from the intrinsic stochasticity of SGD, rather than being solely a function of the step size. Importantly, 2 proved the existence of oscillations at EoS in two-layer ReLU networks and highlighted its benefits for generalization, while 10 ; 23 argued that oscillations in linear models implicitly bias GD toward flatter solutions. Recent work investigated the convergence and other implicit regularization aspects of EoS regime, e.g., see ( 6 ; 48 ; 3 ; 55 ; 37 ) . Deep linear networks. The key object of our study is the deep linear network model (also known as deep matrix factorization). Deep linear networks serve as essential proxies for understanding the training dynamics of complex architectures. However, most existing literature focuses on their behavior under gradient descent or gradient flow ( 8 ; 45 ; 25 ; 5 ; 20 ; 7 ; 19 ; 24 ; 47 ; 54 ; 41 ) , where some of them suggest that GD possesses an implicit bias toward flatter minima in parameter space in these models ( 39 ; 10 ; 23 ) . While some studies have also characterized their loss landscapes—showing that all local minima are global ( 32 ; 34 ) —the role of noise remains less explored. Although 57 analyzed the global minima of deep linear networks with weight decay, our analysis specifically addresses the implicit bias induced by the stochasticity of SGD. Other closely related works are 42 ; 21 ; 9 , which analyze diagonal neural networks (multiple dynamics with 1-dimensional label covariances), and show that the stochasticity is beneficial. Effect of noise in the labels. There are a lot of studies on the implicit bias of noise in the labels. It is shown that the noise in the labels has an implicit regularization effect that biases the model to minima of weaker noise 12 ; 35 ; 26 or flatter minimum 16 . They also show that such a regularization effect is beneficial for generalization 43 ; 28 . Importantly, all these work allow only 1-dimensional output. 1-dimensional matrices always have condition number 1. Our work, in contrast, shows that the noisy labeling can also bias the model to sharper solutions, dependent on the condition number. 7 Conclusion In this work, we studied in depth how and when SGD training can have a flatness or sharpness-seeking behavior. We provide an exact analytical solution for the dynamics of deep linear networks under the minimal gradient fluctuation constraint. In this model, we demonstrate that the converged sharpness is unique and independent of initialization. A key insight is that noise in the labels imbalance is a primary driver of the sharpening phenomenon – a conclusion further validated through experiments on realistic datasets across model types. This is a novel form of progressive sharpening, of a different nature from the ones previously discussed in the literature. There are various limitations, each of which is worthwhile to be addressed by future work. First of all, our theory is effective in nature and does not directly reveal the dynamical aspects of training. A valuable open problem is thus to study the dynamics of reaching sharper and flatter solutions. Secondly, our theory only studies the leading order effect in \eta during training, which tends to suppress fluctuation. This essentially corresponds to the small learning rate regime of training (but not as small as what is needed for gradient flow), and it is unclear what role higher-order terms in the entropic loss could play. Thus, understanding the flattening and sharpening effects with larger learning rates remains open. For example, it could well be the case that at a very large learning rate, the sharpness-seeking behavior becomes strongly suppressed again. For a similar reason, within our framework, only progressive sharpening can be explained, but not EoS. This is because to derive the entropic loss ( 12 ), one needs to essentially assume that the learning is somewhat stable 46 . Solving these problems will certainly advance our understanding of deep learning optimization. Appendix A Theory A.1 Proof of Lemma 1 Proof. By the definition of A -symmetry, the loss function satisfies the following invariance property for any \lambda : \ell(x,y,e^{\lambda A}\theta)=\ell(x,y,\theta). (31) To analyze the sharpness, we take the second-order derivative with respect to \theta on both sides. Applying the chain rule, we obtain: e^{\lambda A}\nabla_{e^{\lambda A}\theta}^{2}\ell(x,y,e^{\lambda A}\theta)e^{\lambda A}=\nabla^{2}\ell(x,y,\theta), (32) where we use the fact that A is symmetric. Using the linearity and cyclic property of the trace, we have: T(e^{\lambda A}\theta):={\rm Tr}[\mathbb{E}_{x,y}\nabla_{e^{\lambda A}\theta}^{2}\ell(x,y,e^{\lambda A}\theta)]={\rm Tr}[e^{-2\lambda A}\mathbb{E}_{x,y}\nabla^{2}\ell(x,y,\theta)]. (33) Let A:=\sum_{i}\mu_{i}n_{i}n_{i}^{T} . The sharpness can then be expressed as a weighted sum: T(e^{\lambda A}\theta)=\sum_{i}e^{-2\lambda\mu_{i}}(n_{i}^{T}\mathbb{E}_{x,y}\nabla^{2}\ell(x,y,\theta)n_{i}). (34) By the assumption A\mathbb{E}{x,y}\nabla^{2}\ell(x,y,\theta)\neq 0 , there must exist at least one index i such that \mu_{i}\neq 0 and n_{i}^{T}\mathbb{E}\nabla^{2}\ell(x,\theta)n_{i}\neq 0 . Consequently, the term e^{-2\lambda\mu_{i}} will dominate as \lambda tends to infinity with the appropriate sign. Specifically, we have \lim_{\lambda\to+\infty}|T(e^{\lambda A}\theta)|=+\infty if there exists \mu_{i}<0 , and \lim_{\lambda\to-\infty}|T(e^{\lambda A}\theta)|=+\infty if there exists \mu_{i}>0 . ∎ A.2 Proof of Lemma 2 We first provide a formal statement of Lemma 2 . Lemma 3 . Let f,S be continuous functions, and let f be coercive (i.e., \lim_{|\theta|\to\infty}f(\theta)=+\infty ). Define the perturbed objective function F_{\eta}(\theta)=f(\theta)+\eta S(\theta). (35) Let \{\eta_{k}\}_{k=1}^{\infty} be a sequence of positive numbers such that \eta_{k}\to 0 as k\to\infty . Suppose \theta_{k}\in\arg\min_{\theta}f(\theta)+\eta_{k}S(\theta). (36) and \theta_{k}\to\bar{\theta} 5 5 5 As f is coercive, M is compact and thus there exists a subsequence of \{\theta_{k}\} which converges. Here for simplicity we assume \{\theta_{k}\} converges. as k\to\infty . Then \bar{\theta} minimizes S over the set of minimizers of f , i.e., \bar{\theta}\in\arg\min_{\theta\in M}S(\theta),\quad\text{where }M:=\arg\min_{\theta}f(\theta). (37) Proof. Since f is coercive and continuous, the set of minimizers M is non-empty and compact. Let y\in M be any global minimizer of f . By the optimality of \theta_{k} for F_{\eta_{k}} , we have: f(\theta_{k})+\eta_{k}S(\theta_{k})\leq f(y)+\eta_{k}S(y). (38) Rearranging the terms, we obtain: f(\theta_{k})-f(y)\leq\eta_{k}(S(y)-S(\theta_{k})) (39) Since y\in M , we have f(\theta_{k})\geq f(y) , which implies f(\theta_{k})-f(y)\geq 0 . Consequently, from ( 39 ) and the fact that \eta_{k}>0 , it follows that: S(\theta_{k})\leq S(y). (40) Taking the limit as k\to\infty in ( 39 ), and noting that S is continuous (thus S(\theta_{k}) remains bounded as \theta_{k}\to\bar{\theta} ), the right-hand side vanishes: 0\leq\lim_{k\to\infty}(f(\theta_{k})-f(y))\leq\lim_{k\to\infty}\eta_{k}(S(y)-S(\theta_{k}))=0. (41) By the continuity of f , we have f(\bar{\theta})=f(y) , which implies \bar{\theta}\in M . Finally, we consider the inequality ( 40 ). Taking the limit k\to\infty on both sides and utilizing the continuity of S , we obtain: S(\bar{\theta})\leq S(y)\quad\text{for all }y\in M. (42) This inequality demonstrates that \bar{\theta} minimizes S over M , i.e., \bar{\theta}\in\arg\min_{\theta\in M}S(\theta) , which completes the proof. ∎ A.3 Proof of Theorem 1 Proof. Step 1: Computing the Hessian Trace. We begin by evaluating the trace of the Hessian with respect to the weight matrices \{W_{k}\}_{k=1}^{D} . For each layer k , let A_{k}:=W_{D}\cdots W_{k+1} and B_{k}:=W_{k-1}\cdots W_{1} , with the conventions A_{D}=I_{d_{y}} and B_{1}=I_{d_{x}} . The loss function for a single sample can be written as \ell(x,y,\theta)=\frac{1}{2}\|y-A_{k}W_{k}B_{k}x\|^{2} . The second-order partial derivative with respect to W_{k} yields the following identity: {\rm Tr}\frac{\partial^{2}\ell}{\partial W_{k}^{2}}={\rm Tr}A_{k}^{T}A_{k}{\rm Tr}B_{k}xx^{T}B_{k}^{T}. (43) By taking the expectation over the data distribution and summing over all layers k=1,\dots,D , the total sharpness T(\theta) is given by: T(\theta)=\sum_{k=1}^{D}\mathbb{E}_{x,y}{\rm Tr}\frac{\partial^{2}\ell}{\partial W_{k}^{2}}=\sum_{k=1}^{D}{\rm Tr}[A_{k}^{T}A_{k}]{\rm Tr}[B_{k}\Sigma_{x}B_{k}^{T}]. (44) This establishes the structural form of the sharpness. The minimization of T(\theta) as per ( 15 ) then follows directly from applying Lemma 4 to the summation in ( 44 ). Step 2: Analysis of Sharpness at the Minimum of ( 12 ). For the solutions from Lemma 5 , the product matrices are decomposed as A_{k}=\sqrt{\Sigma_{\epsilon}^{-1}}L\Sigma_{D}\cdots\Sigma_{k+1}U_{k}^{T},\ B_{k}=U_{k-1}\Sigma_{k-1}\cdots\Sigma_{1}R\Sigma_{x}^{-1/2}. (45) For the intermediate layers k\in\{2,\dots,D-1\} , the trace product simplifies to: \displaystyle{\rm Tr}[A_{k}^{T}A_{k}]{\rm Tr}[B_{k}\Sigma_{x}B_{k}^{T}] \displaystyle={\rm Tr}[\Sigma_{\epsilon}^{-1}L\Sigma^{2}_{D}\cdots\Sigma_{k+1}^{2}L^{T}]{\rm Tr}[\Sigma_{1}^{2}\cdots\Sigma_{k-1}^{2}] (46) \displaystyle=\left({\rm Tr}S\right)^{-2/D}({\rm Tr}[\Sigma_{\epsilon}]{\rm Tr}[\Sigma_{x}])^{1/D}{\rm Tr}[\Sigma_{\epsilon}^{-1}LSL^{T}]{\rm Tr}[S] The boundary layers require separate treatment. For k=1 and k=D , we have: {\rm Tr}[A_{1}^{T}A_{1}]{\rm Tr}[B_{1}\Sigma_{x}B_{1}^{T}]={\rm Tr}[\Sigma_{\epsilon}^{-1}L\Sigma^{2}_{D}\cdots\Sigma_{2}^{2}L^{T}]{\rm Tr}[\Sigma_{x}]=\left({\rm Tr}S\right)^{(D-2)/D}\frac{{\rm Tr}[\Sigma_{\epsilon}]^{1/D}}{{\rm Tr}[\Sigma_{x}]^{(D-1)/D}}{\rm Tr}[\Sigma_{\epsilon}^{-1}LSL^{T}]{\rm Tr}[\Sigma_{x}] (47) and {\rm Tr}[A_{D}^{T}A_{D}]{\rm Tr}[B_{D}\Sigma_{x}B_{D}^{T}]={\rm Tr}[I_{d_{y}}]{\rm Tr}[\Sigma_{1}^{2}\cdots\Sigma_{D-1}^{2}]=d_{y}\left({\rm Tr}S\right)^{(D-2)/D}\frac{{\rm Tr}[\Sigma_{x}]^{1/D}}{{\rm Tr}[\Sigma_{\epsilon}]^{(D-1)/D}}{\rm Tr}[S]. (48) Summing these contributions over k=1,\dots,D yields the final expression for T(\theta) : \displaystyle T(\theta) \displaystyle=\sum_{k=1}^{D}{\rm Tr}[A_{k}^{T}A_{k}]{\rm Tr}[B_{k}\Sigma_{x}B_{k}^{T}] (49) \displaystyle=(D-1)\left({\rm Tr}S\right)^{(D-2)/D}({\rm Tr}[\Sigma_{\epsilon}]{\rm Tr}[\Sigma_{x}])^{1/D}{\rm Tr}[\Sigma_{\epsilon}^{-1}LSL^{T}]+d_{y}\left({\rm Tr}S\right)^{2(D-1)/D}\frac{{\rm Tr}[\Sigma_{x}]^{1/D}}{{\rm Tr}[\Sigma_{\epsilon}]^{(D-1)/D}}, which proves ( 14 ). Step 3: Invariance of the Hessian Spectrum. Finally, we verify that the spectrum of the Hessian H(\theta) is constant across the manifold of solutions defined in Lemma 5 . Consider two sets of weight matrices \theta=\{W_{k}\} and \theta^{\prime}=\{W_{k}^{\prime}\} . According to the lemma, any two such solutions are related by a sequence of orthogonal transformations R_{k}\in O(d_{k}) , such that W_{k}^{\prime}=R_{k}W_{k}R_{k-1}^{T} . This transformation can be represented as \theta^{\prime}=\mathcal{M}\theta , where \mathcal{M} is a block-diagonal orthogonal matrix. Since the loss function is invariant under these internal rotations, the Hessian transforms as H(\theta^{\prime})=\mathcal{M}H(\theta)\mathcal{M}^{T} . Because \mathcal{M} is orthogonal, this is a similarity transformation, which implies that the eigenvalues (and thus the spectrum) of the Hessian remain invariant. ∎ The following are technical lemmas for Theorem 1 . Lemma 4 . Consider \displaystyle T(W_{1},\cdots,W_{D}) \displaystyle=d_{y}{\rm Tr}[W_{D-1}\cdots W_{1}\Sigma W_{1}^{T}\cdots W_{D-1}^{T}]+{\rm Tr}[W_{D}^{T}W_{D}]{\rm Tr}[W_{D-2}\cdots W_{1}\Sigma W_{1}^{T}\cdots W_{D-2}^{T}] (50) \displaystyle+\cdots+{\rm Tr}[W_{2}^{T}\cdots W_{D}^{T}W_{D}\cdots W_{2}]{\rm Tr}[\Sigma], where d_{y} is the output dimension. Then, under the constraint W_{D}\cdots W_{1}=V , we have \min T(W_{1},\cdots,W_{D})=D({\rm Tr}\hat{S})^{2(D-1)/D}(d_{y}{\rm Tr}\Sigma)^{1/D}, (51) where V\sqrt{\Sigma}=\hat{L}\hat{S}\hat{R} is the singular value decomposition. The minimum is achieved when \displaystyle W_{D}=({\rm Tr}S)^{-\frac{D-2}{2D}}d_{y}^{\frac{D-1}{2D}}({\rm Tr}\Sigma)^{-\frac{1}{2D}}\hat{L}\hat{S}^{\frac{1}{2}}U_{D-1}^{T}, (52) \displaystyle W_{k}=({\rm Tr}S)^{\frac{1}{D}}(d_{y}{\rm Tr}\Sigma)^{-\frac{1}{2D}}U_{k}^{T}U_{k-1}, \displaystyle W_{1}\sqrt{\Sigma}=({\rm Tr}S)^{-\frac{D-2}{2D}}d_{y}^{-\frac{1}{2D}}({\rm Tr}\Sigma)^{\frac{D-1}{2D}}U_{1}\hat{S}^{\frac{1}{2}}\hat{R}, for k=2,\cdots,D-1 , where \{U_{i}\}_{i=1}^{D-1} are arbitrary matrices satisfying U_{i}^{T}U_{i}=I_{d\times d} . Proof. Step 1: Lower Bound. We seek to minimize the sum of D terms in ( 50 ). For brevity, let W_{1}^{\prime}=W_{1}\Sigma^{1/2} . By the AM-GM inequality, the sharpness T is bounded below by the geometric mean of its components: \displaystyle T(W_{1},\cdots,W_{D})^{D}/D^{D} \displaystyle\geq d_{y}||W_{D-1}\cdots W_{1}^{\prime}||_{F}^{2}||W_{D}||_{F}^{2}||W_{D-2}\cdots W_{1}\textquoteright||_{F}^{2}\cdots||W_{D}\cdots W_{2}||_{F}^{2}{\rm Tr}[\Sigma] (53) \displaystyle=d_{y}{\rm Tr}[\Sigma]\Pi_{k=1}^{D-1}||W_{D}\cdots W_{k+1}||_{F}^{2}||W_{k}\cdots W_{1}^{\prime}||_{F}^{2} Applying the matrix inequality ||A||_{F}||B||_{F}\geq||AB||_{*} (54) where ||\cdot||_{*} is the nuclear norm (sum of singular values), to each pair in the product, we have: ||W_{D}\dots W_{k+1}||_{F}||W_{k}\dots W_{1}^{\prime}||_{F}\geq||W_{D}\dots W_{1}^{\prime}||=||V\Sigma^{1/2}||_{*}={\rm Tr}\hat{S}. (55) Substituting this into the AM-GM result, we obtain the minimal sharpness: T(W_{1},\cdots,W_{D})^{D}/D^{D}\geq d_{y}{\rm Tr}[\Sigma]({\rm Tr}\hat{S})^{2(D-1)}. (56) Step 2: Conditions for Equality. The minimum is achieved if and only if A_{k}^{T}A_{k}=cB_{k}B_{k}^{T} for some c\geq 0 and every term in ( 50 ) is the same, which requires the following relations: W_{k+1}^{\top}\frac{W_{k+2}^{\top}\cdots W_{D}^{\top}W_{D}\cdots W_{k+2}}{{\rm Tr}\left[W_{k+2}^{\top}\cdots W_{D}^{\top}W_{D}\cdots W_{k+2}\right]}W_{k+1}=W_{k}\frac{W_{k-1}\cdots W_{1}\Sigma W_{1}^{\top}\cdots W_{k-1}^{\top}}{{\rm Tr}\left[W_{k-1}\cdots W_{1}\Sigma W_{1}^{\top}\cdots W_{k-1}^{\top}\right]}W_{k}^{\top} (57) for k=2,3,\cdots,D-2 and \displaystyle W_{2}^{\top}\frac{W_{3}^{\top}\cdots W_{D}^{\top}W_{D}\cdots W_{3}}{{\rm Tr}\left[W_{3}^{\top}\cdots W_{D}^{\top}W_{D}\cdots W_{3}\right]}W_{2}=\frac{W_{1}\Sigma W_{1}^{\top}}{{\rm Tr}\left[\Sigma\right]}, (58) \displaystyle\frac{W_{D}^{\top}W_{D}}{d_{y}}=W_{D-1}\frac{W_{D-2}\cdots W_{1}\Sigma W_{1}^{\top}\cdots W_{D-2}^{\top}}{{\rm Tr}\left[W_{D-2}\cdots W_{1}\Sigma W_{1}^{\top}\cdots W_{D-2}^{\top}\right]}W_{D-1}^{\top}. Step 3: Determining the Weights. By Lemma 2 of [ 59 ] , we can decompose the matrices W_{1}^{\prime},W_{2},\cdots,W_{D-1},W_{D} as W_{D}=U_{D}\Sigma_{D}U_{D-1}^{T},\ W_{D-1}=U_{D-1}\Sigma_{D-1}U_{D-2}^{T},\cdots,\ W_{1}^{\prime}=U_{1}\Sigma_{1}U_{0}, (59) where W_{1}^{\prime}:=W_{1}\sqrt{\Sigma} . From W_{D}\cdots W_{1}=V we have U_{D}=\hat{L} , U_{0}=\hat{R} and \Sigma_{D}\Sigma_{D-1}\cdots\Sigma_{1}=\hat{S}. (60) We can assume \Sigma_{D},\cdots,\Sigma_{1}\in\mathbb{R}^{d\times d} without loss of generality because their ranks are the same by ( 57 ), ( 58 ). Then, ( 57 ) gives \frac{\Sigma_{i+1}^{2}\Sigma_{i+2}^{2}\cdots\Sigma_{D}^{2}}{{\rm Tr}[\Sigma_{i+2}^{2}\cdots\Sigma_{D}^{2}]}=\frac{\Sigma_{1}^{2}\cdots\Sigma_{i-1}^{2}\Sigma_{i}^{2}}{{\rm Tr}[\Sigma_{1}^{2}\cdots\Sigma_{i-1}^{2}]}, (61) and thus \Sigma_{i}=cI_{d} for i=2,3,\cdots,D-2 and \frac{\Sigma_{1}^{2}}{{\rm Tr}[\Sigma_{1}^{2}]}=\frac{\Sigma_{D}^{2}}{{\rm Tr}[\Sigma_{D}^{2}]}. (62) ( 58 ) gives \frac{\Sigma_{2}^{2}\Sigma_{3}^{2}\cdots\Sigma_{D}^{2}}{{\rm Tr}[\Sigma_{3}^{2}\cdots\Sigma_{D}^{2}]}=\frac{\Sigma_{1}^{2}}{{\rm Tr}[\Sigma]},\ \frac{\Sigma_{D}^{2}}{d_{y}}=\frac{\Sigma_{1}^{2}\cdots\Sigma_{D-1}^{2}}{{\rm Tr}[\Sigma_{1}^{2}\cdots\Sigma_{D-2}^{2}]}, (63) and thus c^{2}\frac{\Sigma_{D}^{2}}{{\rm Tr}[\Sigma_{D}^{2}]}=\frac{\Sigma_{1}^{2}}{{\rm Tr}[\Sigma]},\ \frac{\Sigma_{D}^{2}}{d_{y}}=c^{2}\frac{\Sigma_{1}^{2}}{{\rm Tr}[\Sigma_{1}^{2}]}. (64) Combining ( 60 ), ( 62 ) and ( 64 ), we finish the proof. ∎ The following lemma is a restatement of Theorem 12 of [ 59 ] . Lemma 5 . Consider a D- layer linear network which has more than d hidden units in every layer. Then at the global minimum of ( 12 ), we have \sqrt{\Sigma_{\epsilon}}W_{D}=L\Sigma_{D}U_{D-1}^{T},\ W_{i}=U_{i}\Sigma_{i}U_{i-1}^{T},\ W_{1}\sqrt{\Sigma_{x}}=U_{1}\Sigma_{1}R, (65) for i=2,\cdots,D-1 , where \{U_{i}\}_{i=1}^{D-1} are arbitrary matrices satisfying U_{i}^{T}U_{i}=I_{d\times d} , and \Sigma_{x}=\mathbb{E}[xx^{T}] , \Sigma_{\epsilon}=\mathbb{E}[\epsilon\epsilon^{T}] . Moreover, \displaystyle\Sigma_{1}=({\rm Tr}S^{\prime})^{-\frac{D-2}{2D}}\frac{{\rm Tr}[\Sigma_{x}]^{\frac{D-1}{2D}}}{{\rm Tr}[\Sigma_{\epsilon}]^{\frac{1}{2D}}}\sqrt{S^{\prime}}, (66) \displaystyle\Sigma_{D}=({\rm Tr}S^{\prime})^{-\frac{D-2}{2D}}\frac{{\rm Tr}[\Sigma_{\epsilon}]^{\frac{D-1}{2D}}}{{\rm Tr}[\Sigma_{x}]^{\frac{1}{2D}}}\sqrt{S^{\prime}}, \displaystyle\Sigma_{i}=({\rm Tr}S^{\prime})^{1/D}({\rm Tr}[\Sigma_{\epsilon}]{\rm Tr}[\Sigma_{x}])^{-\frac{1}{2D}}I_{d}. A.4 Initial sharpness To understand the optimization landscape at the start of training, we analyze the expected sharpness under standard random initialization schemes. Proposition 1 . Consider a D -layer linear network where each weight matrix W_{k}\in\mathbb{R}^{d_{k+1}\times d_{k}} is initialized with i.i.d. entries w_{i,j}^{(k)}\sim\mathcal{N}(0,\sigma_{k}^{2}) , where d_{1}=d_{x} and d_{D+1}=d_{y} . At initialization, the expected sharpness is given by: \mathbb{E}[T(\theta)]=d_{y}\Pi_{k=2}^{D}d_{k}{\rm Tr}[\Sigma_{x}]\sum_{k=1}^{D}\Pi_{j\neq k}\sigma_{j}^{2}. (67) If we further assume \sigma_{k}^{2}=\sigma^{2} and d_{k}=d for all k , this simplifies to: \mathbb{E}[T(\theta)]=D\cdot d_{y}{\rm Tr}[\Sigma_{x}](d\sigma^{2})^{D-1}. (68) Proof. Recall that T(\theta)=\sum_{k=1}^{D}{\rm Tr}[A_{k}^{T}A_{k}]{\rm Tr}[B_{k}\Sigma_{x}B_{k}^{T}] . Since the weight matrices \{W_{k}\} are initialized independently, the expectation of the trace product decomposes into the product of expectations. For A_{k} , we have \mathbb{E}[{\rm Tr}[A_{k}^{T}A_{k}]]=\mathbb{E}[{\rm Tr}[W_{k+1}^{T}\cdots W_{D}W_{D}^{T}\cdots W_{k+1}]]=d_{y}\Pi_{j=k+1}^{D}d_{j}\sigma_{j}^{2}. (69) Similarly, for B_{k} : \mathbb{E}[{\rm Tr}[B_{k}\Sigma_{x}B_{k}^{T}]]=\mathbb{E}[{\rm Tr}[W_{k-1}\cdots W_{1}\Sigma_{x}W_{1}^{T}\cdots W_{k-1}]]={\rm Tr}[\Sigma_{x}]\Pi_{j=1}^{k-1}d_{j+1}\sigma_{j}^{2}. (70) Substituting these into the total sharpness expression ( 44 ) yields ( 67 ). ∎ For the Xavier initialization, we have \sigma_{k}^{2}=\frac{2}{d_{k}+d_{k+1}} . Taking d_{k}=d for all k , we have \mathbb{E}[T(\theta)]=Dd_{y}{\rm Tr}[\Sigma_{x}] . For the Kaiming initialization with linear activation, we have \sigma^{2}=\frac{1}{d_{k}} . Then the sharpness is the same if d_{k}=d for any k . A.5 Largest eigenvalue The following theorem characterizes the spectral radius of the Hessian at the optimal solutions. Theorem 3 . Suppose that \Sigma_{x}=\hat{R}^{-1}\Lambda_{x}\hat{R} for some diagonal matrix \Lambda_{x} , where \hat{R} is the right singular matrix of V\sqrt{\Sigma_{x}} . Then, the largest eigenvalue of the Hessian matrix for the solution ( 52 ) is \displaystyle\lambda_{\max}(\mathbb{E}[H(\theta)]) \displaystyle=(D-2)({\rm Tr}\hat{S})^{-\frac{2}{D}}(d_{y}{\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(\hat{S})^{2}+({\rm Tr}\hat{S})^{\frac{D-2}{D}}d_{y}^{-\frac{D-1}{D}}({\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(\hat{S}) (71) \displaystyle+({\rm Tr}\hat{S})^{\frac{D-2}{D}}d_{y}^{\frac{1}{D}}({\rm Tr}\Sigma_{x})^{-\frac{D-1}{D}}\lambda_{\max}(\hat{S}\Lambda_{x}^{-1})\lambda_{\max}(\Sigma_{x}) Moreover, suppose that \Sigma_{x}=R^{-1}\Lambda_{x}R and \Sigma_{\epsilon}=L\Lambda_{\epsilon}L^{-1} for some diagonal matrices \Lambda_{x},\Lambda_{\epsilon} , where L,R are the left and right singular matrices of \sqrt{\Sigma_{\epsilon}}V\sqrt{\Sigma_{x}} . Then the largest eigenvalue of the Hessian matrix for the solution ( 66 ) is \displaystyle\lambda_{\max}(\mathbb{E}[H(\theta)]) \displaystyle=(D-2)({\rm Tr}S)^{-\frac{2}{D}}({\rm Tr}\Sigma_{\epsilon}{\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(\Lambda_{\epsilon}^{-1}S)\lambda_{\max}(S)+({\rm Tr}S)^{\frac{D-2}{D}}({\rm Tr}\Sigma_{\epsilon})^{-\frac{D-1}{D}}({\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(S) (72) \displaystyle+({\rm Tr}S)^{\frac{D-2}{D}}({\rm Tr}\Sigma_{\epsilon})^{\frac{1}{D}}({\rm Tr}\Sigma_{x})^{-\frac{D-1}{D}}\lambda_{\max}(\Lambda_{\epsilon}^{-1}S\Lambda_{x}^{-1})\lambda_{\max}(\Sigma_{x}) Proof. Step 1: Quadratic Form of the Hessian. By ( 24 ), for a deep linear network f(\theta,x)=W_{D}\cdots W_{1}x , the Hessian is H(\theta)=\mathbb{E}_{x}\nabla f(\theta,x)\nabla f(\theta,x)^{T}. (73) Let \Delta=\text{vec}(\Delta_{D},\cdots,\Delta_{1}) be the perturbation, where \Delta_{i} has the same dimension as W_{i} , and the quadratic form of the Hessian is \Delta^{T}H(\theta)\Delta=\mathbb{E}_{x}\left[\left\|\sum_{k=1}^{D}A_{k}\Delta_{k}B_{k}x\right\|_{2}^{2}\right]=\left\|\sum_{k=1}^{D}A_{k}\Delta_{k}B_{k}\sqrt{\Sigma_{x}}\right\|_{F}^{2}, (74) where A_{k}:=W_{D}\cdots W_{k+1} and B_{k}:=W_{k-1}\cdots W_{1} , and specially A_{D}=I_{d_{y}} , B_{1}=I . Step 2: Upper Bound. Using the triangle inequality and the sub-multiplicative property of the spectral norm, we bound the magnitude of the summed perturbations: \sqrt{\Delta^{T}H(\theta)\Delta}\leq\sum_{k=1}^{D}||A_{k}||_{2}||B_{k}\sqrt{\Sigma_{x}}||_{2}||\Delta_{k}||_{F}. (75) Applying the Cauchy-Schwarz inequality to the sum, we obtain: \frac{\Delta^{T}H(\theta)\Delta}{\Delta^{T}\Delta}\leq\frac{\left(\sum_{k=1}^{D}||A_{k}||_{2}||B_{k}\sqrt{\Sigma_{x}}||_{2}||\Delta_{k}||_{F}\right)^{2}}{\sum_{k=1}^{D}||\Delta_{k}||^{2}_{F}}\leq\sum_{k=1}^{D}||A_{k}||^{2}_{2}||B_{k}\sqrt{\Sigma_{x}}||^{2}_{2}. (76) This gives \displaystyle\lambda_{\max}(\mathbb{E}[H(\theta)]) \displaystyle\leq\sum_{k=1}^{D}\|W_{D}\dots W_{k+1}\|^{2}_{2}\cdot\|W_{k-1}\dots W_{1}\sqrt{\Sigma_{z}}\|_{2}^{2} (77) \displaystyle=\sum_{k=1}^{D}\|\hat{L}\Sigma_{D}\cdots\Sigma_{k+1}U_{k+1}\|_{2}^{2}\|U_{k-1}\Sigma_{k-1}\cdots\Sigma_{1}\hat{R}\|_{2}^{2} \displaystyle=(D-2)({\rm Tr}\hat{S})^{-2/D}(d_{y}{\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(\hat{S})^{2}+({\rm Tr}\hat{S})^{\frac{D-2}{D}}d_{y}^{-\frac{D-1}{D}}({\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(\hat{S}) \displaystyle+({\rm Tr}\hat{S})^{\frac{D-2}{D}}d_{y}^{\frac{1}{D}}({\rm Tr}\Sigma_{x})^{-\frac{D-1}{D}}\lambda_{\max}(\hat{S}\Lambda_{x}^{-1})\lambda_{\max}(\Sigma_{x}). Step 3: Tightness and Construction. To show that the upper bound is achievable, we construct a specific perturbation \Delta . Let u_{D} and u_{0} be the top left and right singular vectors of the full network product, respectively. For the solution ( 52 ), we choose rank-1 perturbations: \Delta_{k}=\alpha_{k}u_{i}u_{i-1}^{T} , where u_{k} is the column of U_{k} corresponding to the largest singular vector of \hat{S} (or \hat{S}\Lambda_{x}^{-1/2} for k=0 ), and specially U_{0}:=\hat{R},U_{D}:=\hat{L} . Thus, we have A_{k}\Delta_{k}B_{k}\sqrt{\Sigma_{x}}=\alpha_{k}||A_{k}||_{2}||B_{k}\sqrt{\Sigma_{x}}||_{2}u_{D}u_{0}^{T}, (78) where we use the fact that W_{1}=U_{1}\Sigma_{1}\hat{R}\Sigma_{x}^{-1/2}=U_{1}\Sigma_{1}\Lambda_{x}^{-1/2}\hat{R} (79) under the assumption \Sigma_{x}=\hat{R}^{-1}\Lambda_{x}\hat{R} . Then, we have \Delta^{T}H(\theta)\Delta=\left\|\sum_{k=1}^{D}\alpha_{k}||A_{k}||_{2}||B_{k}\sqrt{\Sigma_{x}}||_{2}u_{D}u_{0}^{T}\right\|_{F}^{2}=\left(\sum_{k=1}^{D}\alpha_{k}||A_{k}||_{2}||B_{k}\sqrt{\Sigma_{x}}||_{2}\right)^{2}, (80) which, by choosing \alpha_{k}=||A_{k}||_{2}||B_{k}\sqrt{\Sigma_{x}}||_{2} , gives \frac{\Delta^{T}H(\theta)\Delta}{\Delta^{T}\Delta}=\sum_{k=1}^{D}||A_{k}||^{2}_{2}||B_{k}\sqrt{\Sigma_{x}}||^{2}_{2}. (81) This proves ( 71 ). For the solution ( 66 ), we have W_{D}=\Sigma_{\epsilon}^{-1/2}L\Sigma_{D}U_{D-1}^{T}=L\Lambda_{\epsilon}^{-1/2}\Sigma_{D}U_{D-1}^{T} (82) under the assumption that \Sigma_{\epsilon}=L\Lambda_{\epsilon}L^{-1} . Then we can prove ( 72 ) in the same manner. ∎ Remark . From ( 74 ), one obtains \Delta^{T}H(\theta)\Delta=\left\|\left[\sum_{k=1}^{D}(B_{k}\sqrt{\Sigma_{x}})^{T}\otimes A_{k}\right]\text{vec}(\Delta_{k})\right\|_{2}^{2}, (83) which implies \lambda_{\max}(H(\theta))=\sigma_{\max}\left(\sum_{k=1}^{D}(B_{k}\sqrt{\Sigma_{x}})^{T}B_{k}\sqrt{\Sigma_{x}}\otimes A_{k}A_{k}^{T}\right), (84) which recovers Theorem 3.1 of a concurrent work [ 31 ] . However, unless B_{k}\sqrt{\Sigma_{x}} and \Sigma_{x} share the same singular spaces, ( 84 ) cannot be further simplified under the solution ( 52 ). Thus, we introduce the commutation assumption in Theorem 72 . A.6 Proof of Corollary 1 Proof. We begin by invoking Von Neumann’s trace inequality to derive three useful bounds: \lambda_{\min}(\sqrt{\Sigma_{\epsilon}})||V\sqrt{\Sigma_{x}}||_{*}\leq{\rm Tr}S\leq\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})||V\sqrt{\Sigma_{x}}||_{*}, (85) {\rm Tr}[\Sigma_{\epsilon}^{-1}LSL^{T}]\geq\frac{\sigma_{\min}(S)}{\lambda_{\min}(\Sigma_{\epsilon})}\geq\frac{\sigma_{\min}(V\sqrt{\Sigma_{x}})}{\sqrt{\lambda_{\min}(\Sigma_{\epsilon})}}, (86) and {\rm Tr}[\Sigma_{\epsilon}^{-1}LSL^{T}]\leq\frac{||S||_{*}}{\lambda_{\min}(\Sigma_{\epsilon})}\leq\frac{\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})||V\sqrt{\Sigma_{x}}||_{*}}{\lambda_{\min}(\Sigma_{\epsilon})}, (87) where ||\cdot||_{*} denotes the kernel norm. To find the lower bound for T(\theta) , we utilize the first term of the sharpness expression. Applying ( 85 ) and ( 86 ), we obtain: \displaystyle\left({\rm Tr}S\right)^{(D-2)/D}({\rm Tr}\Sigma_{\epsilon})^{1/D}{\rm Tr}[\Sigma_{\epsilon}^{-1}LSL^{T}] \displaystyle\geq\left(\lambda_{\min}(\sqrt{\Sigma_{\epsilon}})||V\sqrt{\Sigma_{x}}||_{*}\right)^{(D-2)/D}({\rm Tr}\Sigma_{\epsilon})^{1/D}\frac{\sigma_{\min}(V\sqrt{\Sigma_{x}})}{\sqrt{\lambda_{\min}(\Sigma_{\epsilon})}} (88) \displaystyle\geq\max(\sigma_{\min}(V\sqrt{\Sigma_{x}})^{2},1)\frac{\lambda_{\max}(\Sigma_{\epsilon})^{\frac{D-2}{2D}+\frac{1}{D}}}{\sqrt{\lambda_{\min}(\Sigma_{\epsilon})}} \displaystyle:=2c_{1}\sqrt{\kappa(\Sigma_{\epsilon})}, where c_{1} is a constant depending solely on the data and model through V\sqrt{\Sigma_{x}} . It follows from ( 14 ) that the total sharpness is lower-bounded by the square root of the condition number: T(\theta)\geq 2(D-1)c_{1}\sqrt{\kappa(\Sigma_{\epsilon})}\geq Dc_{1}\sqrt{\kappa(\Sigma_{\epsilon})}. (89) For the upper bound, we consider both terms of the sharpness. First, using ( 85 ) and ( 87 ), we bound the primary term: \displaystyle\left({\rm Tr}S\right)^{(D-2)/D}({\rm Tr}\Sigma_{\epsilon})^{1/D}{\rm Tr}[\Sigma_{\epsilon}^{-1}LSL^{T}] (90) \displaystyle\leq(\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})||V\sqrt{\Sigma_{x}}||_{*})^{(D-2)/D}(d_{y}\lambda_{\max}(\sqrt{\Sigma_{\epsilon}}))^{1/D}\frac{\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})||V\sqrt{\Sigma_{x}}||_{*}}{\lambda_{\min}(\Sigma_{\epsilon})} \displaystyle\leq\max(d_{y}^{1/D}||V\sqrt{\Sigma_{x}}||^{2}_{*},1)\frac{\lambda_{\max}(\Sigma_{\epsilon})}{\lambda_{\min}(\Sigma_{\epsilon})}. Next, we bound the second term in the expression of T(\theta) using ( 85 ): \displaystyle d_{y}\left({\rm Tr}S\right)^{2(D-1)/D}\frac{({\rm Tr}\Sigma_{x})^{1/D}}{({\rm Tr}\Sigma_{\epsilon})^{(D-1)/D}} \displaystyle\leq d_{y}({\rm Tr}\Sigma_{x})^{1/D}\frac{(\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})||V\sqrt{\Sigma_{x}}||_{*})^{2(D-1)/D}}{(d_{y}\lambda_{\min}(\Sigma_{\epsilon}))^{(D-1)/D}} (91) \displaystyle\leq d_{y}d_{x}\max(||V\sqrt{\Sigma_{x}}||_{*}^{2},1)\frac{\lambda_{\max}(\Sigma_{\epsilon})}{\lambda_{\min}(\Sigma_{\epsilon})}. By plugging these two upper bounds into ( 14 ), we arrive at the final linear relation: T(\theta)\leq(D-1)\max(d_{y}||V\sqrt{\Sigma_{x}}||^{2}_{*},1)\frac{\lambda_{\max}(\Sigma_{\epsilon})}{\lambda_{\min}(\Sigma_{\epsilon})}+d_{y}d_{x}\max(||V\sqrt{\Sigma_{x}}||_{*}^{2},1)\frac{\lambda_{\max}(\Sigma_{\epsilon})}{\lambda_{\min}(\Sigma_{\epsilon})}\leq Dc_{2}\kappa(\Sigma_{\epsilon}), (92) where c_{2} is a constant determined by V\sqrt{\Sigma_{x}} and the dimensions d_{x},d_{y} . This concludes the proof. ∎ A.7 Proof of Corollary 2 Proof. The proof follows from a direct comparison of the minimal sharpness conditions derived in Lemma 4 and the SGD solutions characterized in Lemma 5 . By setting \Sigma_{\epsilon}=I in ( 66 ), the matrix balancing and the singular value distributions across layers align perfectly with the requirements for minimal sharpness ( 52 ). ∎ As a sanity check, when we substitute \Sigma_{\epsilon}=I into ( 14 ), we have V^{\prime}=V\sqrt{\Sigma_{x}}=LSL^{T} , S=\hat{S} , and thus T(\theta)=D({\rm Tr}S)^{2(D-1)/D}(d_{y}{\rm Tr}\Sigma_{x})^{1/D}, (93) which is the same as the minimal sharpness. A.8 Proof of Corollary 3 Proof. nder the commutation assumption, the eigenvectors of the covariance matrices \Sigma_{\epsilon},\Sigma_{x} and the singular vectors of the model V align. This allows for a simplified spectral analysis: \Sigma_{\epsilon}=L\Lambda_{\epsilon}L^{-1},\ \Sigma_{x}=R^{-1}\Lambda_{x}R,\ \sqrt{\Sigma_{\epsilon}}V\sqrt{\Sigma_{x}}=LSR. (94) Consequently, we can bound the spectral radius of the product matrix S and its scaled variants: \lambda_{\max}(\sqrt{\Sigma_{\epsilon}})\sigma_{\min}(V\sqrt{\Sigma_{x}})\leq\lambda_{\max}(S)\leq\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})\sigma_{\max}(V\sqrt{\Sigma_{x}}) (95) and \frac{\lambda_{\min}(V\sqrt{\Sigma_{x}})}{\lambda_{\min}(\sqrt{\Sigma_{\epsilon}})}\leq\lambda_{\max}(\Lambda_{\epsilon}^{-1}S)\leq\frac{\lambda_{\max}(V\sqrt{\Sigma_{x}})}{\lambda_{\min}(\sqrt{\Sigma_{\epsilon}})}. (96) For the upper bound, we evaluate the three terms in ( 72 ) individually using the trace inequality ( 85 ). For the first term: \displaystyle({\rm Tr}S)^{-\frac{2}{D}}({\rm Tr}\Sigma_{\epsilon}{\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(\Lambda_{\epsilon}^{-1}S)\lambda_{\max}(S) (97) \displaystyle\leq(\lambda_{\min}(\sqrt{\Sigma_{\epsilon}})||V\sqrt{\Sigma_{x}}||_{*})^{-\frac{2}{D}}(d_{y}\lambda_{\max}(\Sigma_{\epsilon}){\rm Tr}\Sigma_{x})^{\frac{1}{D}}\frac{\sigma_{\min}(V\sqrt{\Sigma_{x}})}{\lambda_{\min}(\sqrt{\Sigma_{\epsilon}})}\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})\sigma_{\min}(V\sqrt{\Sigma_{x}}) \displaystyle\leq\max(\sigma_{\min}(V\sqrt{\Sigma_{x}})^{2},1)\max(d_{y}{\rm Tr}\Sigma_{x},1)\kappa(\Sigma_{\epsilon}). For the second term: \displaystyle({\rm Tr}S)^{\frac{D-2}{D}}({\rm Tr}\Sigma_{\epsilon})^{-\frac{D-1}{D}}({\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(S) (98) \displaystyle\leq(\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})||V\sqrt{\Sigma_{x}}||_{*})^{\frac{D-2}{D}}\lambda_{\min}(\Sigma_{\epsilon})^{-\frac{D-1}{D}}({\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})\sigma_{\max}(V\sqrt{\Sigma_{x}}) \displaystyle\leq\max(||V\sqrt{\Sigma_{x}}||_{*}^{2},1)\max({\rm Tr}\Sigma_{x},1)\kappa(\Sigma_{\epsilon}). And for the third term: \displaystyle({\rm Tr}S)^{\frac{D-2}{D}}({\rm Tr}\Sigma_{\epsilon})^{\frac{1}{D}}({\rm Tr}\Sigma_{x})^{-\frac{D-1}{D}}\lambda_{\max}(\Lambda_{\epsilon}^{-1}S\Lambda_{x}^{-1})\lambda_{\max}(\Sigma_{x}) (99) \displaystyle\leq(\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})||V\sqrt{\Sigma_{x}}||_{*})^{\frac{D-2}{D}}\lambda_{\max}(\Sigma_{\epsilon})^{\frac{1}{D}}({\rm Tr}\Sigma_{x})^{-\frac{D-1}{D}}\frac{\sigma_{\max}(V\sqrt{\Sigma_{x}})}{\lambda_{\min}(\sqrt{\Sigma_{\epsilon}})\lambda_{\min}(\Sigma_{x})}\lambda_{\max}(\Sigma_{x}) \displaystyle\leq\max(||V\Sigma_{x}||_{*}^{2},1)\max(\lambda_{\max}(\Sigma_{x})^{-1},1)\kappa(\Sigma_{x})\kappa(\Sigma_{\epsilon}). Summing these contributions, we conclude that the spectral radius of the Hessian is bounded by \lambda_{\max}(\mathbb{E}[H(\theta)])\leq c_{2}D\kappa(\Sigma_{\epsilon}) (100) for a constant c_{2} determined by the problem geometry ( V,\Sigma_{x},d_{y} ). We derive the lower bound in the same way. For the first term: \displaystyle({\rm Tr}S)^{-\frac{2}{D}}({\rm Tr}\Sigma_{\epsilon}{\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(\Lambda_{\epsilon}^{-1}S)\lambda_{\max}(S) (101) \displaystyle\geq(\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})||V\sqrt{\Sigma_{x}}||_{*})^{-\frac{2}{D}}(\lambda_{\max}(\Sigma_{\epsilon}){\rm Tr}\Sigma_{x})^{\frac{1}{D}}\frac{\sigma_{\min}(V\sqrt{\Sigma_{x}})}{\lambda_{\min}(\sqrt{\Sigma_{\epsilon}})}\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})\sigma_{\min}(V\sqrt{\Sigma_{x}}) \displaystyle\geq\min(\sigma_{\min}(V\sqrt{\Sigma_{x}})^{2},1)\min({\rm Tr}\Sigma_{x},1)\sqrt{\kappa(\Sigma_{\epsilon})}. For the second term: \displaystyle({\rm Tr}S)^{\frac{D-2}{D}}({\rm Tr}\Sigma_{\epsilon})^{-\frac{D-1}{D}}({\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(S) (102) \displaystyle\geq(\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})\sigma_{\min}(V\sqrt{\Sigma_{x}}))^{\frac{D-2}{D}}\lambda_{\max}(\Sigma_{\epsilon})^{-\frac{D-1}{D}}({\rm Tr}\Sigma_{x})^{\frac{1}{D}}\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})\sigma_{\min}(V\sqrt{\Sigma_{x}}) \displaystyle\geq\min(\sigma_{\min}(V\sqrt{\Sigma_{x}})^{2},1)\min({\rm Tr}\Sigma_{x},1)\sqrt{\kappa(\Sigma_{\epsilon})}. Finally, for the third term: \displaystyle({\rm Tr}S)^{\frac{D-2}{D}}({\rm Tr}\Sigma_{\epsilon})^{\frac{1}{D}}({\rm Tr}\Sigma_{x})^{-\frac{D-1}{D}}\lambda_{\max}(\Lambda_{\epsilon}^{-1}S\Lambda_{x}^{-1})\lambda_{\max}(\Sigma_{x}) (103) \displaystyle\geq(\lambda_{\max}(\sqrt{\Sigma_{\epsilon}})\sigma_{\min}(V\sqrt{\Sigma_{x}}))^{\frac{D-2}{D}}\lambda_{\max}(\Sigma_{\epsilon})^{\frac{1}{D}}({\rm Tr}\Sigma_{x})^{-\frac{D-1}{D}}\frac{\sigma_{\min}(V\sqrt{\Sigma_{x}})}{\lambda_{\min}(\sqrt{\Sigma_{\epsilon}})\lambda_{\min}(\Sigma_{x})}\lambda_{\max}(\Sigma_{x}) \displaystyle\geq\min(\sigma_{\min}(V\Sigma_{x})^{2},1)\min(d_{y}^{-1}\lambda_{\max}(\Sigma_{x})^{-1},1)\kappa(\Sigma_{x})\sqrt{\kappa(\Sigma_{\epsilon})}. Assembling these lower bounds in ( 72 ) yields: \lambda_{\max}(\mathbb{E}[H(\theta)])\geq c_{1}D\sqrt{\kappa(\Sigma_{\epsilon})} (104) for some constant c_{1} depending on V , \Sigma_{x} and d_{y} . This completes the proof. ∎ A.9 Proof of Corollary 107 Proof. It is proven by plugging {\rm Tr}\Sigma_{\epsilon},{\rm Tr}\Sigma_{x},{\rm Tr}S=\Theta(d^{\max(0,1-a)}),\Theta(d^{\max(0,1-b)}),\Theta(d^{\max(0,1-a/2-b/2-c)}) into ( 14 ). ∎ A.10 An Example: Power-law Spectrum Corollary 4 . Suppose that \Sigma_{\epsilon},\Sigma_{x},V have power-law spectrum \{i^{-a}\}_{i=1}^{d},\{i^{-b}\}_{i=1}^{d},\{i^{-c}\}_{i=1}^{d} for a,b,c>0 and they commute. Then for large d we have \displaystyle T(\theta) \displaystyle=\Theta\left((D-1)d^{\kappa_{1}}+d^{\kappa_{2}}\right) (105) at the global minimum of ( 12 ), where \displaystyle\kappa_{1}: \displaystyle=\frac{D-2}{D}\max\left(0,1-\frac{a}{2}-\frac{b}{2}-c\right)+\frac{1}{D}\max(0,1-a) (106) \displaystyle+\frac{1}{D}\max(0,1-b)+\max\left(0,1+\frac{a}{2}-\frac{b}{2}-c\right) and \displaystyle\kappa_{2}: \displaystyle=1+\frac{2(D-1)}{D}\max(0,1-\frac{a}{2}-\frac{b}{2}-c) (107) \displaystyle+\frac{1}{D}\max(0,1-b)-\frac{D-1}{D}\max(0,1-b). Corollary 107 further verifies that the noise imbalance is the main cause of sharpening, because the sharpness only increases when a increases, but not when b,c increase. Appendix B Experiments Figure 5 : The maximal eigenvalue of the Hessian evolves similarly to the trace. Left : linear networks, with the minimal sharpness predicted by ( 71 ). Middle : Two layer ReLU networks trained under a teacher-student setting. Right : Two layer ReLU networks trained on MNIST. Each line is averaged over five trials shown with the standard error. Figure 6 : The evolution of the condition numbers for three-layer linear networks. Left : Balanced noise in the labels. Right : Imbalanced noise in the labels. Figure 7 : The evolution of sharpness for linear networks of various depths. Left : Traces of Hessian. Right : Maximal eigenvalues of Hessian. Experiments on linear networks For the linear network settings, we set D=3 , d_{x}=d_{y}=2 , and V=I , with data drawn from a standard Gaussian distribution ( \Sigma_{x}=I ). We configure the noise scales as \sqrt{\Sigma_{\epsilon}}=\text{diag}(10,0.1) for the non-isotropic case and \sqrt{\Sigma_{\epsilon}}=\text{diag}(10,10) for the isotropic case. Training is performed using online SGD for 10^{5} epochs (learning rate 0.01 , batch size 10 ). The Hessian is estimated via 200 random samples, where the trace is calculated exactly and the maximal eigenvalue is computed using the power method. For experiments in Figure 1 and Figure 4 , we maintain these hyperparameters but adjust the initialization strength and learning rate (to 0.001 ), or transition to a single-batch gradient descent (GD) setting (batch size 1000 ). To complement the observations in Figure 2 , we evaluate the maximal eigenvalue of the Hessian matrix, as shown in Figure 5 . The resulting curves exhibit consistent trends, further corroborating Corollary 3 . Figure 6 illustrates the condition numbers across layers for a three-layer linear network. Aligning with our analytical solution ( 66 ), we observe that balanced noise in the labels leads to small condition numbers across all layers. Conversely, imbalanced noise primarily inflates the condition numbers of the first and last layers, while intermediate layers remain stable. Furthermore, Figure 7 demonstrates that depth acts only as a multiplicative factor for sharpness, as suggested by Corollary 1 . Figure 8 : Non-isotropic noise leads to progressive sharpening also for transformers. Left : The evolution of the Hessian trace. Right : The evolution of the maximal eigenvalue. Experiments on non-linear settings In the ReLU teacher-student setting (in Figures 2 and 5 ), we employ a single-hidden-layer teacher-student framework with d_{x}=d_{y}=2 . The teacher and student networks consist of 100 and 4 hidden units, respectively, both initialized using the Kaiming method. We use MSE loss and online SGD (learning rate 0.1 , batch size 100 ) for 10^{5} epochs. Noise scales are set to \text{diag}(3,0.03) and \text{diag}(3,3) . Hessian statistics are computed using 200 samples, with the trace approximated by the Hutchinson estimator. For the MNIST dataset (in Figures 2 and 5 ), a two-layer ReLU network ( 128 hidden units) is trained using cross-entropy loss. noise in the labels is introduced by adding Gaussian noise to one-hot labels, followed by a softmax operation. The non-isotropic noise covariance \sqrt{\Sigma_{\epsilon}} is set to \text{diag}(10,\dots,10,1,\dots,1) (five entries each), while the isotropic case uses \sqrt{\Sigma_{\epsilon}}=10I . We run SGD for 10^{4} epochs (learning rate 0.01 , batch size 128 ) and evaluate the Hessian on a test batch of size 1024 . To verify the universality of our findings beyond linear networks and MLPs, we extend our experiments to transformers and RNNs in a teacher-student framework. In Figures 3 and 8 , we employ a simplified Transformer comprising a two-head self-attention layer followed by an MLP with residual connections. The model takes inputs of dimension 8 and sequence length 4 , with the output dimension matching the input. The hidden dimension of the MLP is set to 100 for the teacher and 4 for the student, both initialized via the Kaiming method. Training is conducted using online SGD (learning rate 0.03 , batch size 300 ) for 10^{4} epochs on standard Gaussian data. The Hessian is evaluated on a test set of size 1000 . We configure the noise scales as \sqrt{\Sigma_{\epsilon}}=\text{diag}(4,0.04,\dots,0.04) for the non-isotropic case and \sqrt{\Sigma_{\epsilon}}=\text{diag}(4,\dots,4) for the isotropic case. The results in Figure 8 confirm that non-isotropic noise in the labels induces the same progressive sharpening phenomenon in transformers as observed in simpler models. Furthermore, we investigate the specific impact of the noise covariance condition number on sharpness in Figure 3 . Maintaining the same experimental setup, we systematically vary the smaller eigenvalues of \Sigma_{\epsilon} while keeping other hyperparameters fixed. This analysis is also extended to RNNs, where the teacher and student hidden units are set to 100 and 4 , respectively. For experiments involving cross-entropy loss, an additional softmax layer is applied to the network output. Across all these diverse settings—including different architectures and loss functions—the observed trends remain highly consistent, further validating the robustness of our theoretical predictions.