AdaGradSelect: An adaptive gradient-guided layer selection method for efficient fine-tuning of SLMs
Anshul Kumar, Gagan Raj Gupta, Manisha Chawla · 2025
Read AdaGradSelect: An adaptive gradient-guided layer selection method for efficient fine-tuning of SLMs by Anshul Kumar, Gagan Raj Gupta, Manisha Chawla (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15764.txt.
Introduction
Anshul Kumar, Gagan Raj Gupta, and Manisha Chawla present AdaGradSelect, a gradient guided layer selection method for fine tuning small language models, in a 2025 paper, dated December 12. Full fine tuning is costly, and low rank adapter methods restrict training to a narrow subspace that can cost performance. Small models need efficiency most.
AdaGradSelect picks which transformer blocks to update based on gradient norms, adapting the choice as training proceeds. Dirichlet based sampling favors blocks updated often before, balanced by epsilon greedy exploration. In tests the method trains about 12 percent faster and uses 35 percent less GPU memory than full fine tuning, with near equal quality. On GSM8K it beats a high rank LoRA baseline by about 3 percent.
Catalog record
| CREATOR | Anshul Kumar, Gagan Raj Gupta, Manisha Chawla |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15764
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdaGradSelect: An adaptive gradient-guided layer selection method for efficient fine-tuning of SLMs Anshul Kumar, Gagan Raj Gupta, Manisha Chawla Abstract . While Large Language Models (LLMs) excel at diverse NLP tasks, their adaptation through full fine-tuning is computationally expensive and memory-intensive. Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) mitigate this by introducing low-rank updates to frozen weights, but this constrains optimization to a low-rank subspace and can limit performance. Focusing on Small Language Models (SLMs), where efficiency gains offer significant practical benefits, we introduce AdaGradSelect , an adaptive, gradient-guided block selection strategy for efficient fine-tuning. Motivated by preliminary findings that selectively updating transformer blocks with the highest gradient norms approaches full fine-tuning performance, AdaGradSelect dynamically prioritizes which blocks to train. The method combines Dirichlet-based sampling, informed by historical update frequencies, with an ϵ \epsilon -greedy exploration strategy. This approach initially balances the exploitation of important blocks with the exploration of new candidates before transitioning to full exploitation in later epochs, optimizing the training process. Experimental results demonstrate that AdaGradSelect trains approximately 12% faster and uses 35% less GPU memory while achieving performance nearly identical to full fine-tuning. On the GSM8K dataset, our method consistently outperforms LoRA (rank 256) by an average of 3% across Qwen2.5-0.5B, LLaMA3.2-1B, and Phi4-mini-3.8B models. It also shows comparable accuracy on the MATH dataset, establishing AdaGradSelect as a more effective and resource-efficient fine-tuning approach. 1. Introduction Large Language Models (LLMs) have demonstrated exceptional performance across a wide range of tasks, including documentation generation, complex code synthesis, question answering, and human-like conversation ( 24 ) . Their success on diverse NLP benchmarks ( 10 ) is largely attributed to strong generalization and capabilities such as in-context learning. However, LLMs often underperform on specialized or low-resource tasks, necessitating domain-specific fine-tuning to bridge the performance gap ( 26 ; 27 ) . Despite its effectiveness, full fine-tuning of LLMs is computationally expensive and memory-intensive. The memory footprint is dominated not only by billions of model parameters but also by additional optimizer states and gradients, such as momentum and variance terms in Adam, which can exceed the storage requirements of the parameters themselves ( 29 ; 5 ) . These constraints make full fine-tuning prohibitively costly for most practical applications. A practical alternative is to focus on Small Language Models (SLMs) ( 28 ) . They are much smaller than LLMs, but can still be fine-tuned effectively. They are also easier to deploy in environments with limited resources. In this work, we go a step further. Our aim is to make fine-tuning even faster and more effective, so that SLMs provide the maximum possible benefit in practice. Figure 1. Comparison of training time vs Avg GPU usage for training Qwen2.5 0.5B using different methods Parameter Efficient Fine Tuning(PEFT) methods have emerged as a promising alternative, allowing adaptation of pre-trained language models to downstream tasks without updating all parameters ( 8 ) . Among these, Low-Rank Adaptation (LoRA) ( 17 ) is widely adopted. LoRA reparameterizes a weight matrix W\in\mathbb{R}^{m\times n} as: W=W_{0}+BA, where W_{0} is a frozen pre-trained matrix, and B\in\mathbb{R}^{m\times r} , A\in\mathbb{R}^{r\times n} are learnable low-rank adapters with r\ll\min(m,n) . This significantly reduces trainable parameters and optimizer states, lowering memory usage. While LoRA achieves efficiency gains, its optimization is constrained to a low-rank subspace, which can limit representational power and prevent it from matching full fine-tuning performance in many cases ( 8 ; 30 ) .And the case of SLMs with small hidden dimensions, adapter-based methods offer limited benefits and can even result in longer training times compared to full fine-tuning, as shown in Figure 1 . This occurs because the additional overhead from the adapter forward pass, combined with the computation of the model’s original weights, becomes significant in smaller models. While such effects are largely mitigated in larger models, they remain noticeable in smaller ones. To address these limitations, we introduce AdaGradSelect — an adaptive gradient-guided block selection strategy for efficient LLM fine-tuning (custom AdamW). Instead of updating all transformer blocks, AdaGradSelect selectively updates a subset of high-impact blocks determined by cumulative gradient norms. We enhance this approach with: • Dirichlet-based sampling , which biases block selection toward historically significant blocks using update frequency counts. • An \epsilon -greedy exploration strategy during the first epoch, enabling controlled exploration of alternative blocks. The exploration probability decays exponentially within the first epoch, after which the process transitions to pure exploitation. This formulation draws parallels with the Multi-Armed Bandit problem, where blocks act as arms, and the goal is to maximize adaptation efficiency under resource constraints. Our contributions are as follows: (1) We empirically analyze the effect of fine-tuning only a subset of transformer blocks based on gradient norms, showing that updating as few as 10% of blocks can be comparable to full fine-tuning performance. (2) We propose AdaGradSelect , a novel block selection algorithm that combines Dirichlet-based exploitation with \epsilon -greedy exploration to reduce computational overhead and memory usage without compromising accuracy. (3) Through extensive experiments, we show that AdaGradSelect consistently outperforms LoRA with rank 256 by an average of 3% on the GSM8K dataset across Qwen2.5-0.5B, LLaMA3.2-1B, and Phi4-mini-3.8B models, while achieving comparable accuracy on the MetaMathQA-40K dataset, establishing it as both more effective and more resource-efficient than existing fine-tuning methods. (4) Furthermore, our method demonstrates clear efficiency gains, achieving approximately 12% faster training and 35% lower GPU memory usage, while maintaining nearly the same accuracy as full fine-tuning as shown in Figure 1 . 2. Related Work LLMs such as GPT-3 ( 4 ) , PaLM ( 6 ) , LLaMA ( 29 ) , and T5 ( 26 ) have achieved state-of-the-art performance across a broad spectrum of natural language processing tasks. Traditionally, adapting these models to downstream tasks has relied on full fine-tuning, wherein all model parameters are updated. Although effective, full fine-tuning is computationally intensive and memory-expensive, making it infeasible for most resource-constrained environments. To overcome these limitations, recent research has shifted attention toward SLMs, compact but capable variants of LLMs. SLMs offer significant practical benefits, such as they are faster to train, cheaper to deploy, and more suitable for edge devices, low-resource environments, and specialized domains ( 13 ; 19 ; 31 ; 18 ) . Despite their smaller size, they remain competitive across a range of NLP tasks ( 1 ; 11 ) . This makes them an attractive alternative to LLMs, especially when efficient fine-tuning strategies can further unlock their potential. Our work is motivated by this direction, focusing on improving the speed and effectiveness of SLM fine-tuning to maximize their impact in practical settings. Beyond this, researchers have also explored Parameter-Efficient Fine-Tuning (PEFT) methods to reduce the cost of adapting large pre-trained models. Representative approaches include Low-Rank Adaptation (LoRA) ( 16 ) , which inserts trainable low-rank matrices into attention modules; Adapter Tuning ( 15 ) , which introduces lightweight modules between transformer blocks; Prefix-Tuning ( 20 ) , which prepends continuous task-specific vectors to the input; and BitFit ( 2 ) , which updates only bias terms. These methods significantly reduce the number of trainable parameters, but they often constrain optimization to a limited subspace, which can result in suboptimal performance compared to full fine-tuning. Recent methods