Sharpness-Aware Minimization: General Analysis and Improved Rates

Dimitris Oikonomou, Nicolas Loizou · 2025

Read Sharpness-Aware Minimization: General Analysis and Improved Rates by Dimitris Oikonomou, Nicolas Loizou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02225.txt.

Introduction

Oikonomou and Loizou give a fresh math review of a training method called sharpness-aware minimization in this twenty twenty-five paper. The method trains machine learning models by also keeping the loss surface flat, which tends to help models work well on new data. Open points include how step rules, noise, and data sampling change the proof that it settles well.

Inside, the authors merge the plain and scaled forms of the method into one shared update rule. They prove settle rates under a weaker and more natural noise rule than past work used. Tests on image tasks with deep networks match the new theory and show the shared method works well in practice.

Catalog record

CREATORDimitris Oikonomou, Nicolas Loizou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-02225

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sharpness-Aware Minimization: General Analysis and Improved Rates Dimitris Oikonomou, Nicolas Loizou Abstract Sharpness-Aware Minimization (SAM) has emerged as a powerful method for improving generalization in machine learning models by minimizing the sharpness of the loss landscape. However, despite its success, several important questions regarding the convergence properties of SAM in non-convex settings are still open, including the benefits of using normalization in the update rule, the dependence of the analysis on the restrictive bounded variance assumption, and the convergence guarantees under different sampling strategies. To address these questions, in this paper, we provide a unified analysis of SAM and its unnormalized variant (USAM) under one single flexible update rule (Unified SAM), and we present convergence results of the new algorithm under a relaxed and more natural assumption on the stochastic noise. Our analysis provides convergence guarantees for SAM under different step size selections for non-convex problems and functions that satisfy the Polyak-Lojasiewicz (PL) condition (a non-convex generalization of strongly convex functions). The proposed theory holds under the arbitrary sampling paradigm, which includes importance sampling as special case, allowing us to analyze variants of SAM that were never explicitly considered in the literature. Experiments validate the theoretical findings and further demonstrate the practical effectiveness of Unified SAM in training deep neural networks for image classification tasks. 1 Introduction Consider the classical finite-sum optimization problem \displaystyle\min_{x\in\mathbb{R}^{d}}\left[f(x)=\frac{1}{n}\sum_{i=1}^{n}f_{i}(x)\right], (1) where each f_{i} is differentiable, L_{i} -smooth and lower bounded. Let X^{*} be the set of minimizers of f , which we assume is non-empty. In practical scenarios, the variable x represents the model parameters, n is the total number of training instances, and the functions f_{i} are loss functions that measure how close our model is to the i -th training data point. The goal is to minimize the average loss of all training instances. Understanding the generalization capabilities of Deep Neural Networks (DNNs) is a central concern in machine learning research, ( Zhang et al., 2016 ; Hardt et al., 2016 ; Neyshabur et al., 2017 ; Wilson et al., 2017 ; Neyshabur et al., 2018 ; Zhang et al., 2021 ) . The training objective function f has numerous stationary points that perfectly fit the training data ( Liu et al., 2020 ) , and each of these points can lead to dramatically varying generalization performances. This suggests that minimizing the training objective using a carefully selected optimization algorithm can lead to convergence at a point with improved generalization performance ( Foret et al., 2021 ) . Recent studies have observed that the sharpness of the training loss, that is, how rapidly it changes in a neighborhood around the model parameters, correlates strongly with the generalization error ( Keskar et al., 2016 ; Jiang et al., 2019 ; Singh et al., 2025 ; Dziugaite & Roy, 2018 ) . This observation has motivated recent works ( Foret et al., 2021 ; Zheng et al., 2021 ; Wu et al., 2020 ; Andriushchenko et al., 2023 ; Xie et al., 2024 ; Tahmasebi et al., 2024 ) aiming to minimize sharpness to improve generalization. More specifically, building on these ideas Foret et al. (2021) , proposed reformulating the optimization problem in ( 1 ) into a min-max problem of the following form: \displaystyle\min_{x\in\mathbb{R}^{d}}\max_{\|\varepsilon\|\leq\rho}f(x+\varepsilon) where \varepsilon represents the radius of the desired neighborhood. The merits of such a formulation reside in the fact that essentially we minimize the empirical sharpness measure \max_{\|\varepsilon\|\leq\rho}f(x+\varepsilon)-f(x) , which inevitably will lead to flatter minima. The objective now is to find x that minimizes f not just at a specific point but across the entire \varepsilon -neighborhood. By taking the first-order Taylor expansion of f around x and solving for the optimal \varepsilon^{*} , the (Normalized) Sharpness-Aware Minimization (SAM) update rule is obtained: \displaystyle x^{t+1}=x^{t}-\gamma_{t}\nabla f_{S_{t}}\left(x^{t}+\rho_{t}\frac{\nabla f_{S_{t}}(x^{t})}{\|\nabla f_{S_{t}}(x^{t})\|}\right), (SAM) where S_{t}\subseteq[n] is a random subset of data points (mini-batch) with cardinality \tau sampled independently at each iteration t . The normalization of the inner gradient ensures that the point \tilde{x}^{t}=x^{t}+\rho_{t}\frac{\nabla f_{S_{t}}(x^{t})}{\|\nabla f_{S_{t}}(x^{t})} is a good approximation of x^{t} , since \|\tilde{x}^{t}-x^{t}\|=\rho_{t} . This leads to a more stable optimization process, as explained in Dai et al. (2023) . In an orthogonal direction and building upon SAM , Unnormalized Sharpness-Aware Minimization (USAM) was introduced by Andriushchenko & Flammarion (2022) and further investigated in Shin et al. (2024) ; Dai et al. (2023) . The update rule for USAM is defined as follows: \displaystyle x^{t+1}=x^{t}-\gamma_{t}\nabla f_{S_{t}}\left(x^{t}+\rho_{t}\nabla f_{S_{t}}(x^{t})\right). (USAM) In contrast to SAM , USAM omits the normalization thus the point \tilde{x}^{t} can be potentially far from x^{t} making the \tilde{x}^{t} ’s of USAM updates much larger. This means that the removal of normalization can lead to much more aggressive steps making the USAM potentially more unstable. Although the two variants appear closely related, the proof techniques and upper bounds used in the convergence analysis of SAM are substantially different from those in USAM . Furthermore, the convergence guarantees of the two variants vary significantly. For example, in the deterministic setting (full-batch), SAM guarantees convergence only to a neighborhood of the solution, whereas USAM does not. Additionally, the step sizes \gamma_{t} and \rho_{t} used in the two update rules to guarantee convergence are very different. All of these differences motivate the importance and necessity of a novel general analysis of SAM-type algorithms, unifying the two main variants ( SAM and USAM ) and providing the ability to design and analyze new SAM-like methods filling existing gaps in the theoretical understanding of the update rules. In this work we develop such unified framework that allows the combination of the two approaches and, at the same time, obtains the best-known convergence guarantees under relaxed assumptions. Main Contributions. Our main contributions are summarized below. \diamond Unified Framework. We propose the Unified SAM, an update rule that is a convex combination of SAM and USAM , given by: \displaystyle x^{t+1}=x^{t}-\gamma_{t}\nabla f_{S_{t}}\left(x^{t}+\rho_{t}\left(1-\lambda_{t}+\frac{\lambda_{t}}{\|\nabla f_{S_{t}}(x^{t})\|}\right)\nabla f_{S_{t}}(x^{t})\right) (Unified SAM) where \lambda\in[0,1] . The new formulation captures both USAM and SAM as special cases ( \lambda=0 and \lambda=1 , respectively), but more importantly, it opens up a wide range of possible update rules beyond these traditional settings. The unified framework offers the flexibility to adjust the degree of normalization (using different values for \lambda ) based on specific model needs, offering a more versatile approach to SAM. \diamond Technical Assumptions on the Stochastic Noise. Existing convergence analyses of stochastic SAM rely heavily on the bounded variance assumption, that is, there exists a \sigma\geq 0 such that \E\|\nabla f_{S_{t}}(x)-\nabla f(x)\|^{2}\leq\sigma^{2} , ( Andriushchenko & Flammarion, 2022 ; Si & Yun, 2023 ; Li & Giannakis, 2023 ; Harada & Iiduka, 2024 ; Mi et al., 2022 ; Zhuang et al., 2022 ) or sometimes to the much stronger bounded gradient condition \E\|\nabla f_{S_{t}}(x)\|^{2}\leq q^{2} , where q\geq 0 ( Mi et al., 2022 ; Zhuang et al., 2022 ) . While these assumptions have been crucial in previous analyses, they can be overly restrictive. In the literature of convergence analysis for stochastic gradient descent (SGD), there have been a lot of efforts recently on relaxing such assumptions ( Gower et al., 2019 ; Khaled & Richtárik, 2020 ; Gower et al., 2021 ) , but, to date, no work has successfully used

Read the complete text