A Minimalist Example of Edge-of-Stability and Progressive Sharpening

Liming Liu Zixuan Zhang Simon Du Tuo Zhao · 2025

Read A Minimalist Example of Edge-of-Stability and Progressive Sharpening by Liming Liu Zixuan Zhang Simon Du Tuo Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02809.txt.

Introduction

Liming Liu, Zixuan Zhang, Simon Du, and Tuo Zhao study odd training behavior in a 2025 paper on deep learning. When learning rates run large, gradient descent shows two quirks: the loss curve sharpens step by step, then settles at the edge of stability. Older theory cannot explain either quirk well. The team builds the smallest case that still shows both: a two layer network fed a two part input, where only one part matters.

Within that spare setup they prove the sharpening and self settling really occur. Their math tracks training and sharpness along the whole path, not just near the end. They also link the small case to broader frameworks and to gradient flow results. The findings may guide sharper tuning choices for real networks trained at bold learning rates.

Catalog record

CREATORLiming Liu Zixuan Zhang Simon Du Tuo Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE98 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-02809

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Minimalist Example of Edge-of-Stability and Progressive Sharpening Thanks: 1 Georgia Tech, 2 University of Washington. Emails: { \{ lliu606, zzhang3105, tourzhao } \} @gatech.edu, [email protected] Liming Liu Zixuan Zhang Simon Du Tuo Zhao Abstract Recent advances in deep learning optimization have unveiled two intriguing phenomena under large learning rates: Edge of Stability (EoS) and Progressive Sharpening (PS), challenging classical Gradient Descent (GD) analyses. Current research approaches, using either generalist frameworks or minimalist examples, face significant limitations in explaining these phenomena. This paper advances the minimalist approach by introducing a two-layer network with a two-dimensional input, where one dimension is relevant to the response and the other is irrelevant. Through this model, we rigorously prove the existence of progressive sharpening and self-stabilization under large learning rates, and establish non-asymptotic analysis of the training dynamics and sharpness along the entire GD trajectory. Besides, we connect our minimalist example to existing works by reconciling the existence of a well-behaved “stable set” between minimalist and generalist analyses, and extending the analysis of Gradient Flow Solution sharpness to our two-dimensional input scenario. These findings provide new insights into the EoS phenomenon from both parameter and input data distribution perspectives, potentially informing more effective optimization strategies in deep learning practice. 1 Introduction Deep learning has revolutionized many fields, from computer vision to natural language processing. However, this progress has also posed significant challenges to classical optimization theory. Most classical gradient descent (GD) analysis assumes small learning rates for easing convergence analysis. Consider minimizing a smooth loss function L(\theta) with respect to the parameter \theta , classical analyses show that when choosing a learning rate \eta such that S(\theta)\leq 2/\eta , where S(\theta) denotes the largest eigenvalue of the Hessian matrix \nabla^{2}L(\theta) , the optimization is “stable” and the loss function decreases monotonically to guarantee convergence Nesterov (2013) . Recent works such as Cohen et al. (2021) , however, have observed that such a stability assumption does not hold when training modern neural networks with GD. In particular, they summarize two specific phenomena: The first one is called “ P rogressive S harpening” ( PS ), that is, S(\theta) , which is also referred to as the “sharpness” in Cohen et al. (2021) , keeps increasing until it reaches the instability threshold 2/\eta during training; The second one is called “ E dge o f S tability” ( EoS ), that is, the sharpness hovers at the instability threshold 2/\eta after progressive sharpening, with the loss decreases continuously and nonmonotonically. These two phenomena undoubtedly challenge the classical analyses and have already attracted the attention of many researchers. Current research on understanding these phenomena has developed along two lines. The first line purses a so-called “generalist analysis” frameworks that, while being generic, rely on hard-to verify assumptions. For example, Li et al. (2022b) analyze GD for training two-layer wide neural networks. By characterizing the norms of the second-layer weights, they prove a four-stage behavior, covering the PS and EoS phenomena. However, their analysis requires assumptions that are hard to verify for two-layer neural networks, e.g., the sharpness being upper bounded, and only works for extremely wide settings, which diverges from practical scenarios. Damian et al. (2022) prove a similar four-stage behavior based on a general loss function, and their analysis also relies on hard-to-verify assumptions, such as the existence of progressive sharpening and the existence of a certain well-behaved “stable set”, which is doubted and showed badly-behaved for scalar networks in ( Kreisler et al., 2023 ) . Overall speaking, such analysis provide results similar to real experiments, but their hard-to-verify assumptions significantly restrict their applicability to practice. Relaxing these assumptions is also extremely challenging. The second line focuses on minimalist examples, offering more concrete and intuitive insights without requiring specific theoretical assumptions. For instance, Zhu et al. (2022) study 2d slices of a 4-layer linear scalar network and prove convergence to a minimum with sharpness slightly below to 2/\eta , and Wang et al. (2023) consider the behavior of sharpness with GD on a special class of 2-layer scalar networks (with nonlinear activation) in the form of F(xy) . However, these works suffer from two drawbacks: (1) They can only characterize the asymptotic sharpness of the converged minimum. Due to the lack of analyzing sharpness for the entire trajectory, they cannot provide more desirable nonasymptotic guarantees for the PS and EoS stages; (2) The setting of scalar networks is over-simplified compared with practice, and the obtained results cannot explain the role of the input data dimension in the EoS stage. Besides, another notable example of the minimalist analysis is Kreisler et al. (2023) , which analyze a variant of sharpness. Specifically, they introduce a concept of Gradient Flow Solution (GFS), and prove a monotonic decrease in the sharpness of GFS at the EoS stage for scalar networks. In addition to the aforementioned drawbacks, Kreisler et al. (2023) also suffer from another drawback: The sharpness of GFS does not directly transfer to that of GD trajectory, thus providing no immediate explanation for the EoS phenomenon. In this paper, we aim to address the limitations of the minimalist analysis by providing a more sophisticated example: A two-layer neural network of width one with a two-dimensional input – in particular, one input dimension is relevant to the response, and the other input dimension is irrelevant. We establish nonasymptotic analysis of the training dynamics along the entire trajectory: (1) We prove the existence of progressive sharpening and self-stabilization under large learning rates; (2) We provide sharpness guarantees for the entire trajectory, showing that GD trajectory will never exceed a sharpness upper bound; (3) We prove that the non-monotonically decreasing loss is essentially monotonically decreasing when projected to the only relevant dimension. Through such theory, we provides new insights of why EoS happens from the perspective of both parameters and input data distribution. Moreover, we highlight two connections of our theory to existing works: (1) We reduce the gap on the existence of a well-behaved “stable set” between the minimalist and generalist analyses. Specifically, Kreisler et al. (2023) prove that the stable set hypothesized in Damian et al. (2022) can be disjoint in the scalar networks studied by Zhu et al. (2022) ; Kreisler et al. (2023) , which essentially violates the assumption of Damian et al. (2022) . Therefore, the projected GD considered in Damian et al. (2022) cannot smoothly decrease the loss toward zero. In contrast, we prove that in our considered two-layer neural network with the two-dimensional input admits a nontrivial well-behaved set, which is the subset of the stable set defined in Damian et al. (2022) . This indicates a potential separation between the bivariante and scalar inputs for linear networks; (2) We extend the analysis of Kreisler et al. (2023) and provide the monotonic decrease of GFS sharpness for our considered two-layer neural network with the two-dimensional input. 2 Related Works The asymptotic property of GD sharpness was first mentioned in Wu et al. (2018) as an empirical observation, that is, the minimal sharpness that the GD trajectory with learning rate \eta ultimately converges to is always around \frac{2}{\eta} . Cohen et al. (2021) made a comprehensive empirical study on the sharpness of the entire GD

Read the complete text