Asynchronous Sharpness-Aware Minimization for Fast and Accurate Deep Learning

Junhyuk Jo, Jihyun Lim, Vincent-Daniel Yun, Sunwoo Lee · 2025

Read Asynchronous Sharpness-Aware Minimization for Fast and Accurate Deep Learning by Junhyuk Jo, Jihyun Lim, Vincent-Daniel Yun, Sunwoo Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11147.txt.

Introduction

Junhyuk Jo and co-authors present in 2025 AsyncSAM, a fast form of a sharpness aware training method for deep nets. That method lifts how well nets cope with new data, yet its cost kept it from wide use. Past cuts still ran its two steps in strict turn.

Their form lets the shake step and the update step run at once on split gear. Batch size for the shake is tuned to the rig, so even plain chips can help at no time cost. Proof shows the same meet pace class as the base form. Tests gave top marks and the best train speed of the kin set.

Catalog record

CREATORJunhyuk Jo, Jihyun Lim, Vincent-Daniel Yun, Sunwoo Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11147

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Asynchronous Sharpness-Aware Minimization for Fast and Accurate Deep Learning Junhyuk Jo, Jihyun Lim, Vincent-Daniel Yun, Sunwoo Lee Abstract . Sharpness-Aware Minimization (SAM) is an optimization method that improves the generalization performance of machine learning models. Despite its effectiveness, it has not been widely adopted in real-world applications due to its high computational cost. Existing approaches reduce this overhead by approximating or reusing gradients in the perturbation step, but still assume sequential perturbation and update steps. In this study, we introduce AsyncSAM , a variant of SAM that enables concurrent model perturbation and parameter updates across multiple compute resources. By adjusting the perturbation batch size in a system-aware manner, our asynchronous scheme fully overlaps perturbation with updates, eliminating the overhead and allowing even CPUs to perform perturbations without extra time. We also provide theoretical analysis showing that AsyncSAM achieves the same order of convergence as conventional SAM. Extensive experiments demonstrate that AsyncSAM delivers superior generalization performance and the highest training throughput among state-of-the-art SAM variants. For example, in ViT fine-tuning on CIFAR-100, it achieves accuracy comparable to SAM while maintaining training time nearly identical to SGD. Conference: ; ; 1. Introduction Sharpness-Aware Minimization (SAM) is an optimization method that improves generalization performance of machine learning models ( Foret et al., 2021 ) . Although SAM improves generalization, its high computational cost has limited its adoption in real-world applications. Specifically, it calculates gradients twice for a single training step – one for model perturbation (gradient ascent) and another for model update (gradient descent). To make SAM a more compelling option for deep learning applications, thus, addressing this high computational cost is crucial. While several previous works have studied ways to mitigate the high computational cost of model perturbation in SAM, a recent trend is to partially discard perturbation gradients. For example, LookSAM ( Liu et al., 2022b ) periodically computes perturbation gradients and reuses them across multiple iterations. AE-SAM ( Jiang et al., 2023 ) alternates between SGD and SAM depending on the loss landscape geometry. ESAM ( Du et al., 2022a ) perturbs only a randomly selected subset of parameters. Although these methods effectively reduce the computational cost, they inevitably lead to a non-negligible drop in accuracy due to the poor quality of the perturbation directions. Figure 1. The cosine similarity between gradients from two consecutive SGD iterations, measured over 1,000 iterations. All curves consistently maintain a high similarity ( >0.8 ), indicating that the parameter space with respect to a given dataset does not change drastically during training. In this paper, we design a novel asynchronous model perturbation scheme which leverages a critical observation on gradient stability during neural network training. Figure 1 presents the cosine similarity between gradients from two consecutive SGD iterations, measured across several image classification benchmarks. The similarity remains consistently high (greater than 0.8), indicating that the gradient direction in parameter space tends to be stable over short training intervals. Similar observations have been reported in different contexts ( Azam et al., 2021 ; Liu et al., 2022b ) . This observation naturally motivates us to develop an asynchronous-parallel model perturbation scheme. Table 1. A feature-wise comparison between AsyncSAM and other SAM-based methods. Method Small Batch No Variance Model Unbiased Efficient Input Data- Overlap Perturbation System-aware Availability Tracking Agnostic Gradient Perturbation Independent with Update Batch Size SAM ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ Generalized SAM ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ SAM-ON ✓ ✓ ✗ ✓ ✓ ✓ ✗ ✗ ESAM ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ LookSAM ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✗ MESA ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ AE-SAM ✓ ✗ ✓ ✓ ✓ ✗ ✗ ✗ SAMPa ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ AsyncSAM ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ We propose AsyncSAM , an Asynchronous model perturbation-based Sharpness-Aware Minimization algorithm. AsyncSAM decouples the model perturbation and update steps and hides the perturbation time with gradient descent time in SAM. That is, we introduce a controlled degree of asynchrony so that the perturbation and update steps can run concurrently. Instead of computing the perturbation gradient using the most recent model parameters, we employ slightly stale parameters to break the data dependency. Our empirical results show that, with a system-aware choice of batch size, asynchronous perturbation can be performed even on relatively slow CPU cores, effectively eliminating the perturbation overhead. Our study provides two key insights as follows. First, gradients in large-scale deep learning tend to remain stable throughout training. As a result, a limited degree of staleness in model perturbation does not considerably harm the quality of the perturbation direction in parameter space, which in turn preserves the gradient norm penalization effect of SAM. Second, although reducing the batch size for gradient ascent introduces stronger noise into the model perturbation, the resulting penalty is offset by the shorter training time made possible by asynchronous–parallel perturbation. Based on these findings, we argue that SAM can be effectively applied to many real-world deep learning applications without significantly increasing training time. Table 1 compares AsyncSAM with SAM variants. We evaluate AsyncSAM on diverse deep learning benchmarks, including CIFAR-10, CIFAR-100, Google Speech Commands, Oxford_Flowers102, Tiny-ImageNet, ImageNet and Vision Transformer fine-tuning. We also compare it with state-of-the-art SAM variants designed to reduce perturbation cost. Our experiments show that AsyncSAM eliminates perturbation overhead while maintaining superior generalization. For example, it achieves 92.66\% accuracy on CIFAR-10, comparable to SAM’s 92.53\% , with training throughput similar to SGD. It also matches SAM in Vision Transformer fine-tuning without increasing training time. These results suggest that AsyncSAM is a practical and efficient optimizer for real-world deep learning tasks. 2. Related Work Problem Statement – We consider the following standard minimization problem: (1) \displaystyle L(w):=\mathbb{E}_{\xi\sim D}\left[l(w,\xi)\right], where \xi is a data sample randomly drawn from the distribution D , and l(\cdot) is a non-convex and smooth objective function. For convenience, we abbreviate \mathbb{E}_{\xi\sim D}[\cdot] as \mathbb{E}[\cdot] . Sharpness-Aware Minimization (SAM) – SAM is an optimization method that improves the model’s generalization performance ( Foret et al., 2021 ) . It first computes the gradient of the loss function and uses it to perturb the model. Next, the gradient is computed again with the perturbed model parameters. Finally, the model is reverted to its original parameters and updated using the gradient calculated from the perturbed model. The parameter update rule is as follows. (2) \displaystyle w_{t+1}=w_{t}-\eta\nabla L\left(w_{t}+r\frac{\nabla L(w_{t})}{\|\nabla L(w_{t})\|}\right), where w_{t} is the model parameters at iteration t , \eta is the learning rate, and the r is a hyper-parameter that determines how strongly the model is perturbed. Recently, it has been shown that SAM improves generalization performance by penalizing the gradient norm ( Zhao et al., 2022 ) . System-Efficient Variants of SAM – There are several previous works that tackle the expensive model perturbation cost in SAM. LookSAM reuses gradients for multiple perturbations ( Liu et al., 2022b ) , but its generalization deteriorates with smaller batch sizes. While AE-SAM reduces the overall training time by adaptively alternating between SGD and SAM based on the loss landscape geometry ( Jiang et al., 2023 ) , it does not fundamentally reduce the perturbation cost. ESAM perturbs a random subset

Read the complete text