GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation

Sungmin Kang, Jisoo Kim, Salman Avestimehr, Sunwoo Lee · 2025

Read GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation by Sungmin Kang, Jisoo Kim, Salman Avestimehr, Sunwoo Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16191.txt.

Introduction

Sungmin Kang, Jisoo Kim, Salman Avestimehr, and Sunwoo Lee propose a lean way to fine-tune large models in a 2025 paper. Standard efficient methods update a few parameters by absolute step size, ignoring how big each parameter was to start. A step that is huge for one weight may be trivial for another, so behavior barely changes.

Their framework scales each update to its parameter's own size and lets each layer's share of tuned weights follow the spread of its values. Tests on general language benchmarks and on math and code tasks beat full fine-tuning by up to a point and a half while touching only a tenth of a percent of parameters. The trick, they argue, is spending the update budget where it counts.

Catalog record

CREATORSungmin Kang, Jisoo Kim, Salman Avestimehr, Sunwoo Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-16191

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation Sungmin Kang, Jisoo Kim, Salman Avestimehr, Sunwoo Lee Abstract Parameter-efficient fine-tuning (PEFT) has become a popular way to adapt large pre-trained models to new tasks. Most PEFT methods update only a small subset of parameters while freezing the rest, avoiding redundant computation. As they maximize the absolute size of the updates without regard to the parameters’ original scale, the resulting changes in model behavior can be minimal. In contrast, we maximize updates relative to each parameter’s scale, yielding more meaningful downstream adaptation. We propose G radient-to-Weight Ratio and E ntropy-guided M asking (GEM), a parameter scale-aware, distribution-sensitive sparse fine-tuning framework. GEM prioritizes parameters whose updates are significant in proportion to their initial pre-trained values. It also adaptively determines how many parameters to tune at each layer based on the entropy of parameter values, thereby making the most effective use of the computational budget in PEFT. Our empirical study demonstrates the efficacy of GEM on both general-domain tasks (GLUE and SuperGLUE) and domain-specific tasks (GSM8k and MBPP), achieving up to a 1.6 % 1.6\% improvement in fine-tuning accuracy over full fine-tuning while updating only 0.1 % 0.1\% of model parameters. 1 Introduction With the success of large-scale pre-trained models Brown et al. 2020 ; Zhang et al. 2022 ; Touvron et al. 2023 ; Gunasekar et al. 2023 , fine-tuning has been an efficient yet effective way of adapting large models to downstream tasks using fewer training data. Despite the success, fine-tuning the entire large-scale model demands intensive resource and time, which poses significant challenges to adapt the model to multiple target tasks. To reduce the computational burden, parameter-efficient fine-tuning (PEFT) methods have been proposed with the aim to adapt models by updating only a small subset of parameters while keeping the majority of weights fixed. Several lines of research have been proposed to address the high cost of fine-tuning, such as Low-rank Adaptation (LoRA) Hu et al. 2021 , Adapter Houlsby et al. 2019 , and prompt tuning Li and Liang 2021 . Parameter masking-based sparse fine-tuning methods Zhao et al. 2020 have also been proposed, showing that most parameters can be frozen, with only a small portion being fine-tuned to achieve comparable accuracy to full fine-tuning. These methods construct masks by estimating the importance of individual parameters and selecting those with the highest scores, using criteria such as gradient magnitude Li et al. 2025 ; Zhang et al. 2024 , Fisher matrix Agarwal et al. 2025 ; Sung et al. 2021 ; Das et al. 2023 , or Hessian-based sensitivity measures Xu and Bu 2025 . However, existing approaches overlook the scale of pre-trained parameter weights during parameter selection. A common goal of PEFT is to maximize downstream adaptation within a constrained budget. Here, we define downstream adaptation not as the absolute magnitude of parameter updates, but as the extent to which parameters change relative to their pre-trained values . In other words, a parameter that undergoes a large proportional shift from its initial value is considered to contribute more significantly to task-specific adaptation, even if its absolute update is small. Therefore, maximizing downstream adaptation under a limited budget entails identifying and updating parameters that exhibit the greatest relative change. This motivates our focus on relative parameter change \lvert\Delta w\rvert/\lvert w\rvert as a principled criterion for identifying the most adaptable parameters under tight tuning budgets. In this work, we propose GEM, a general PEFT framework built upon a novel parameter prioritization metric and an entropy-guided parameter selection method. The main principle of our framework is to select a small portion of parameters that have the most critical impact on model behavior and fine-tune only those parameters efficiently. First, we investigate the impact of a parameter selection strategy based on the gradient magnitude relative to the parameter values. We empirically explore and demonstrate why the gradient-to-weight ratio identifies important parameters more effectively than the gradient magnitude alone. Second, we investigate how to exploit entropy information at each network layer to determine the number of parameters to be tuned per layer. Our study empirically proves that entropy is a useful metric for identifying layers with an irregular distribution of parameter importance. We can safely freeze more parameters in layers with irregular distributions, expecting a greater reduction in computational cost while maintaining fine-tuning accuracy. Method Model Agnostic No Extra Train Param No Extra Infer Param Param-wise Importance Layer-wise Importance Parameter Scale-aware Tuning Budget Controllable Static Mask LoRA ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✓ BitFit ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✓ Adapter ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ AdaLoRA ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✗ Random Mask ✓ ✓ ✓ ✗ ✗ ✗ ✓ ✓ Top-Grad Mask ✓ ✓ ✓ ✓ ✓ * ✗ ✓ ✓ GEM (ours) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ * Some implementations of Top-Grad Masking optionally consider per-layer allocation. Table 1: Comparison of GEM with representative PEFT baselines. GEM is a versatile method that combines scale-awareness, hierarchical selectivity, and model-agnostic design, while being static and parameter-efficient. To evaluate the performance of GEM, we conduct benchmark experiments on several popular datasets, including RTE, SST-2, WiC, BoolQ, MultiRC, CoPA, and SQuAD. We directly compare our proposed method to a variety of SOTA PEFT methods such as LoRA, BitFit, Adapter, AdaLoRA, random masking, and top gradient masking. Our extensive experiments consistently demonstrate that the gradient-to-weight ratio and the entropy-aware parameter selection strategy achieve the best fine-tuning accuracy while significantly reducing the computational cost. We also present a few ablation study results to further evaluate the efficacy of our method. Based on our observations and analyses, we conclude that GEM is a highly efficient PEFT framework that can be readily applied to a wide range of real-world LLM-based applications. Our main contributions are summarized as follows: • We propose a novel parameter prioritization method that maximizes weight changes relative to their original scale and analyze its impact on fine-tuning performance. • We propose an entropy-guided, layer-wise parameter-allocation method that dynamically selects the number of tunable parameters in each layer, and we analyze how exploiting both the magnitude and distribution of learning signals maximizes fine-tuning performance. • Building on these two methods, we introduce a general PEFT framework, GEM, and empirically validate its effectiveness through extensive experiments across diverse tasks, including both general-domain and domain-specific benchmarks. 2 Related Works We first group existing PEFT methods into three categories: (1) reparameterization-based methods, which introduce tunable modules or embeddings into a frozen model; (2) sparsification and masking-based methods, which selectively update a subset of important parameters; and (3) quantization- and pruning-based methods, which compress the model to reduce memory or computational cost during fine-tuning. Some recent approaches combine ideas from multiple categories to improve performance or efficiency. 2.1 Reparameterization-based Methods. A key family of PEFT techniques reparameterizes pre-trained models by modifying internal parameterizations, either by injecting lightweight modules, or prepending tunable inputs, while keeping the original model weights frozen. Low-Rank Adaptation (LoRA) Hu et al. 2021 is a popular PEFT method, which utilizes a low-rank decomposition to efficiently fine-tune models with minimal parameter overhead. There are adaptive rank allocation methods such as AdaLoRA

Read the complete text