Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning

Yifei Cheng, Xianglin Yang, Guoxia Wang, Chao Huang, Fei Ma, Dianhai Yu, Xiaochun Cao, Li Shen · 2026

Read Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning by Yifei Cheng, Xianglin Yang, Guoxia Wang, Chao Huang, Fei Ma, Dianhai Yu, Xiaochun Cao, Li Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09395.txt.

Introduction

Sharpness-aware minimization favors flat minima that generalize well, but its extra perturbation step doubles compute cost, a bottleneck for fine-tuning. Yifei Cheng and seven co-authors propose SL-SAM, a 2026 method that sparsifies the approach at the layer level. Each iteration samples a subset of layers by gradient norm to join the perturbation and update steps, framing layer choice as a multi-armed bandit problem.

The paper backs the method with a convergence analysis, then tests fine-tuning across several tasks. Performance matches leading baselines, including a top rank on language-model fine-tuning. Crucially, backpropagation activates only a fraction of the parameters that standard sharpness-aware minimization always uses. The savings grow with model size, making the method most valuable exactly where fine-tuning costs most.

Catalog record

CREATORYifei Cheng, Xianglin Yang, Guoxia Wang, Chao Huang, Fei Ma, Dianhai Yu, Xiaochun Cao, Li Shen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09395

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning Yifei Cheng, Xianglin Yang, Guoxia Wang, Chao Huang, Fei Ma, Dianhai Yu, Xiaochun Cao, Li Shen Abstract Sharpness-aware minimization (SAM) seeks the minima with a flat loss landscape to improve the generalization performance in machine learning tasks, including fine-tuning. However, its extra parameter perturbation step doubles the computation cost, which becomes the bottleneck of SAM in the practical implementation. In this work, we propose an approach SL-SAM to break this bottleneck by introducing the sparse technique to layers. Our key innovation is to frame the dynamic selection of layers for both the gradient ascent (perturbation) and descent (update) steps as a multi-armed bandit problem. At the beginning of each iteration, SL-SAM samples a part of the layers of the model according to the gradient norm to participate in the backpropagation of the following parameter perturbation and update steps, thereby reducing the computation complexity. We then provide the analysis to guarantee the convergence of SL-SAM . In the experiments of fine-tuning models in several tasks, SL-SAM achieves the performances comparable to the state-of-the-art baselines, including a #1 rank on LLM fine-tuning. Meanwhile, SL-SAM significantly reduces the ratio of active parameters in backpropagation compared to vanilla SAM ( SL-SAM activates 47%, 22% and 21% parameters on the vision, moderate and large language model respectively while vanilla SAM always activates 100%), verifying the efficiency of our proposed algorithm. I Introduction Generalization ability is an important metric in machine learning as the model is susceptible to being over-trained on the training set and may face distribution shifts between the training and test data. The studies in [ 19 , 33 ] reveal the connection between the generalization ability and sharpness of the loss landscape. Based on this, Sharpness-Aware Minimization (SAM) is proposed and renowned as a powerful optimization technique for its ability to enhance model generalization by seeking flat loss minima [ 10 ] . Its mechanism operates in a two-step process: it first computes a worst-case adversarial perturbation that maximizes loss within a local neighborhood, and then updates the model parameters by minimizing the loss at this perturbed point. This approach has been highly successful, further incurring widespread adoption in resource-intensive tasks such as fine-tuning large-scale Transformers models and Large Language Models (LLMs) with billion-level parameters [ 48 , 35 , 23 ] . Fig. 1: Average number of model parameters that participate in gradient calculation per iteration vs. task accuracy. We compare SL-SAM with AdaSAM [ 35 ] (vanilla SAM with AdamW) and a representative efficient variant ESAM [ 9 ] . The points connected by dotted lines indicate that they achieve comparable performances, while the parameters participate in backpropagation in SL-SAM is 47%, 22% and 21% of that in AdaSAM across three tasks. The results in the table show the GPU memory and wall-clock time savings compared to AdaSAM. Details are referred to the Experiments section. However, this performance gain comes at a steep price: by requiring two sequential forward and backward passes (perturbation and update), SAM effectively doubles the gradient calculation time of classical optimizers. This prohibitive computational overhead severely limits its practical adoption, creating a critical bottleneck in real-world applications. Naturally, we raise a question: Can we improve the computation efficiency of SAM? Fig. 2: The workflow of our algorithm SL-SAM . Step 1: all the parameter blocks (in blue) are sampled according to the distributions: the active layers participate in this training iteration while the others are frozen; Step 2 & 3: calculate the gradients for active layers to perform the parameter perturbation and update steps; Step 4: update the distributions by the gradient norms obtained in the perturbation. While prior works have sought to improve SAM’s efficiency, they often focus on reducing the frequency of the perturbation step [ 47 , 28 ] or introducing sparsity in a suboptimal manner. For instance, ESAM [ 9 ] applies perturbation to a randomly selected subset of parameters, failing to distinguish which are most critical to stabilize. SSAMF [ 31 ] considers the adoption of the sparse mask, but still needing to obtain the exact gradient first. This situation leaves a crucial gap: proposing a method that indeed reduces the gradient computation overhead in both of SAM’s two steps. To address this gap, we introduce Sparse-Layer SAM ( SL-SAM ), a novel framework that injects the adaptive layerwise sparsity into SAM. Our approach is predicated on the insight that the gradient norm of the model layer is a strong proxy for a layer’s importance to both the perturbation and update steps, motivating us to activate only partial layers’ parameters to participate in the gradient calculation. However, a naive greedy strategy for conducting the sparse gradient that simply selects the layers with the top gradient norms at each step is brittle. The set of important layers is highly dynamic, and a greedy approach is susceptible to the stochasticity of mini-batch gradients, failing to explore layers that are temporarily dormant but may become critical for convergence. Therefore, our key innovation is to reframe layer selection as a multi-armed bandit problem. This enables SL-SAM to robustly balance exploration and exploitation, dynamically identifying the most salient layers for optimization over time. In each iteration, the bandit samples an active set of layers for SAM’s process: a sparse gradient ascent for perturbation, followed by a sparse gradient descent for model update. All other layers are frozen to skip the gradient calculation to save computation in this iteration. The bandit is guided by using each layer’s gradient norm as a reward signal, ensuring computational resources are intelligently focused on the most critical layers of the model. This process is visualized in Figure 2 . We evaluate our method with both theoretical and empirical validation. We formally prove that SL-SAM achieves a convergence rate of \frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\|\nabla f(x_{t})\|_{1}=\mathcal{O}(T^{-1/4}) , guaranteeing its optimization property. Furthermore, extensive fine-tuning experiments on DeiT, RoBERTa, and large language models demonstrate that SL-SAM delivers a state-of-the-art trade-off between the efficiency and performance. It consistently achieves top-tier performances, including a #1 rank in downstream evaluation tasks for the LLM, while reducing training time by up to 25.8% and significantly cutting memory costs compared to vanilla SAM. Our main contributions are as follows: • A Novel Two-Sided Sparsity Framework: We propose SL-SAM , an optimization algorithm that applies the layerwise sparsity to both the perturbation and update steps of SAM to improve its efficiency. To our best knowledge, this is the first work to achieve this sparsification effect. • Formal Convergence Guarantee: We theoretically prove that SL-SAM achieves a convergence rate of \frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\|\nabla f(x_{t})\|_{1}=\mathcal{O}(T^{-1/4}) , providing the first theoretical evidence that a sparse SAM method can maintain the rate of the original SAM. • Superior Efficiency and Competitive Performance: We demonstrate that SL-SAM offers the best efficiency while maintaining the great performance. Across several fine-tuning tasks, it always achieves scores comparable to other baselines, especially the highest score on LLM fine-tuning. Benefiting from the sparse gradient, the average number of parameters involved in gradient calculation of SL-SAM is dramatically reduced, resulting in significant savings in GPU memory and training time. We show an overview in Figure 1 . • Great Compatibility: We extend to inject the sparse layerwise gradient

Read the complete text