SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates
Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun, Rene Vidal, Department of CIS, AMCS, ESE, IDEASUniversity of Pennsylvania · 2026
Read SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates by Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun, Rene Vidal, Department of CIS, AMCS, ESE, IDEASUniversity of Pennsylvania (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30930.txt.
Introduction
Emmanouilidis and three colleagues analyze stochastic gradient descent, the workhorse training method of deep learning, in the regime where learning rates are far larger than classical theory allows. In practice models are routinely trained at this edge of stability, where curvature-driven oscillations appear, yet most existing analysis covers only deterministic gradient descent and leaves the random, mini-batch setting open.
The authors prove sharp convergence guarantees for the multiclass cross-entropy loss, covering both linear classifiers and two-layer neural networks. Their results show the dynamics alternate between an unstable, oscillating phase and a stable phase where the expected loss falls at a controlled rate. Crucially, the randomness itself self-stabilizes the iterates, returning them to stability within a fixed number of steps. Experiments confirm the picture.
Catalog record
| CREATOR | Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun, Rene Vidal, Department of CIS, AMCS, ESE, IDEASUniversity of Pennsylvania |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 162 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30930
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun, Rene Vidal, Department of CIS, AMCS, ESE, IDEASUniversity of Pennsylvania Abstract Modern deep learning has been shown to operate at the edge of stability, routinely using learning rates far larger than those justified by classical optimization theory. Most prior analyses of the edge of stability phenomenon focus on deterministic gradient descent, leaving the stochastic setting largely unexplored. In this work, we provide sharp convergence guarantees for Stochastic Gradient Descent (SGD) applied to the multiclass cross-entropy loss, for both linear classifiers and two-layer neural networks. We show that the stochasticity of SGD may cause the dynamics to alternate between an edge-of-stability regime that is dominated by curvature-driven oscillations, and a stable regime in which the expected loss decreases at a controlled rate. Despite that, we prove that SGD self-stabilizes the dynamics, ensuring that the iterates return to stability in a fixed number of iterations and allowing convergence in the best-iterate sense even with large learning rates. Experiments validate our theoretical findings and illustrate the benefits of SGD in the large-stepsize regime. 1 Introduction Modern machine learning models are trained at unprecedented scales and with stepsizes that lie far outside the regime covered by classical optimization theory. Yet, stochastic gradient descent (SGD) and its variants ( 1 ; 13 ; 9 ) remain the basic workhorses of deep learning. A striking empirical phenomenon, reported repeatedly across architectures, datasets, and training paradigms, is that state-of-the-art performance is typically achieved by using large learning rates, often far exceeding the stability thresholds predicted by smooth convex optimization. This phenomenon, commonly referred to as the Edge of Stability (EoS) ( 4 ) , has attracted considerable attention in recent years and has emerged as a unifying perspective on the behavior of gradient methods in modern deep networks ( 7 ; 5 ) . Figure 1: Experiment on CIFAR-10. An 8-layer NN with GELU activation function trained with SGD and large step sizes. The training loss decreases faster for large step sizes. In the deterministic setting, the EoS phenomenon is now relatively well understood in linear logistic regression ( 22 ) . For Gradient Descent (GD) applied to the logistic regression setup \displaystyle L(W)=\frac{1}{n}\sum_{i=1}^{n}\ell(Wx_{i},y_{i}), (1) the classical theory dictates that convergence with a monotonic decrease of the loss requires a step size \eta satisfying \displaystyle\lambda_{\max}(\nabla^{2}L(W_{t}))\leq\frac{2}{\eta}, (2) where \lambda_{\max}(\nabla^{2}L(W_{t})) denotes the maximum eigenvalue of the Hessian of the loss at the current iterate W_{t} . However, prior empirical results have demonstrated that GD often converges even when this condition is violated, entering a regime characterized by non-monotonic decrease of the loss and oscillatory dynamics. Moreover, in the case of the square objective \lambda_{\max}(\nabla^{2}L(W_{t})) hovers around the critical value \frac{2}{\eta} ( 4 ; 7 ) . The aforementioned behavior was analyzed in a sequence of works ( 21 ; 3 ; 24 ) culminating in the analysis of large-stepsize GD for logistic regression. A recurring conclusion is that the training dynamics can be divided in two regimes: 1. An early EoS phase, where the iterates experience curvature-driven oscillations, the loss is not monotonically decreasing, and the Hessian’s top eigenvalue exceeds the stability boundary. 2. A late stable phase, in which the algorithm self-stabilizes, the iterates enter a region of the landscape where \lambda_{\max}(\nabla^{2}L(W_{t}))\leq\frac{2}{\eta} , and the loss decreases at a predictable rate. A key insight is that, although the dynamics during the EoS phase appear erratic, their structure is surprisingly rigid: the oscillations persist only for a finite window, after which the system transitions to the stable regime where standard descent analysis becomes applicable. These advances have significantly sharpened our understanding of large-stepsize GD. Nevertheless, the deterministic viewpoint does not capture the behavior of the stochastic counterpart of the algorithm (SGD), that is commonly used in practice. Indeed, stochasticity introduces several new layers of complexity. First, the notion of a stability threshold becomes ill-defined. Second, the loss function need not decrease monotonically even in the stable phase. Third, oscillations may be caused either by curvature or by noise. As a consequence, the deterministic EoS theory, based on precise control of the Hessian’s largest eigenvalue does not extend in a straightforward or even meaningful way to the stochastic setting. This leaves a fundamental question open: What is the appropriate notion of stability for SGD with the cross-entropy loss and can we characterize the dynamics of SGD beyond the classical small learning rate regime? Empirically, practitioners routinely observe that training at large stepsizes remains effective even under stochastic noise, often with improved generalization. Prior work ( 22 ) only provides theoretical guarantees for the average loss of SGD in logistic regression, without describing what happens when the iterates become stable or what are the intrinsic differences with the GD dynamics. Thus, the theoretical picture remains incomplete: we lack a principled understanding of how stochasticity alters the two-phase structure identified in the deterministic GD, and what stability even means in the presence of noise. In this work, we aim to characterize the complex behaviour of the dynamics of SGD on the multi-class cross-entropy loss under arbitrarily large stepsizes and provide insights on the convergence behaviour of the algorithm in the large learning rate regime. Our contributions can be described as follows: \star We propose a notion of stability for SGD in the cross-entropy loss setting, by leveraging the stochastic Lyapunov stability theory for dynamical systems. An important characteristic is that the stochastic stability criterion reduces to the deterministic one in the full-batch limit and captures the precise balance between curvature and noise that determines whether SGD remains stable. \star We provide convergence guarantees for SGD on the multi-class cross-entropy loss with arbitrarily large stepsizes \eta>0 , establishing the first theoretical guarantees for the multi-class setting. \star We provide a refined analysis of the dynamics, decoupling the EoS from the stable regime. We show that SGD enters the stable regime in a fixed number of iterations with high probability. We prove that, despite the inherent stochasticity, SGD stabilizes the trajectory, returning the dynamics back to stability after a fixed number of iterations and remaining there with high probability. \star We provide a fine-grained analysis for the two-layer neural networks dynamics of SGD in the large learning rate setting, establishing distinct EoS and stable regimes and characterizing the behaviour of the dynamics in each regime. \star We provide extensive experimental validation of our theoretical results, showing the benefits of large step-sizes in practice. 2 SGD for Cross-entropy Loss We consider the multi-class classification problem ( 3 ) with K classes, where x_{i}\in\mathbb{R}^{d} is the input, y_{i}\in\{1,\ldots,K\} is the corresponding label, f denotes the prediction function parameterized by weights W=[w_{1},\ldots,w_{K}]\in\mathbb{R}^{d\times K} , and \ell\left(\cdot\right) is the cross-entropy loss function L(W)=\frac{1}{n}\sum_{i=1}^{n}\ell\big(f(x_{i};W),y_{i}\big),\qquad\ell(f,y)=\log\!\bigg(\sum_{j=1}^{K}e^{f_{j}}\bigg)-f_{y}.\vskip-7.11317pt (3) Our goal is to analyze the behavior of SGD in the large stepsize regime. For a mini-batch \mathcal{B}_{t}\subseteq[n] of size b\geq 1 , the SGD update at time t\geq 0 is given by W_{t+1}=W_{t}-\eta\,g_{t}, (SGD) where g_{t}=\frac{1}{b}\sum_{i\in\mathcal{B}_{t}}\nabla\ell(f(x_{i};W_{t}),y_{i}) is the stochastic gradient oracle and \eta>0 is the stepsize. 2.1 Blanket Assumptions We, next, introduce the assumptions needed for presenting our theoretical results. We assume without loss of generality that inputs satisfy \|x_{i}\|_{2}\leq 1,\forall i\in[n] . The next