ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuning

Huandong Chang, Zicheng Ma, Mingyuan Ma, Zhenting Qi · 2025

Read ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuning by Huandong Chang, Zicheng Ma, Mingyuan Ma, Zhenting Qi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00254.txt.

Introduction

Fine-tuning a large model usually fixes how much each layer may change, yet layers differ in importance as training proceeds. The authors of a 2025 study propose ElaLoRA, a method that adjusts those limits during training. Ranks of the small update matrices grow where gradient-based scores show a layer matters. They shrink where a layer matters less. The team describes it as the first approach that both prunes and expands ranks in one run.

Across several benchmarks, the method beats other parameter-efficient techniques at every budget tested. Layers that received more room also proved to matter more for final quality, which supports the adaptive rule. The authors offer it as a practical option where computing resources are tight.

Catalog record

CREATORHuandong Chang, Zicheng Ma, Mingyuan Ma, Zhenting Qi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00254

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuning Huandong Chang, Zicheng Ma, Mingyuan Ma, Zhenting Qi Abstract Low-Rank Adaptation (LoRA) has become a widely adopted technique for fine-tuning large-scale pre-trained models with minimal parameter updates ( Hu et al., 2022 ) . However, existing methods rely on fixed ranks or focus solely on either rank pruning or expansion, failing to adapt ranks dynamically to match the importance of different layers during training. In this work, we propose ElaLoRA, an adaptive low-rank adaptation framework that dynamically prunes and expands ranks based on gradient-derived importance scores. To the best of our knowledge, ElaLoRA is the first method that enables both rank pruning and expansion during fine-tuning. Experiments across multiple benchmarks demonstrate that ElaLoRA consistently outperforms existing PEFT methods across different parameter budgets. Furthermore, our studies validate that layers receiving higher rank allocations contribute more significantly to model performance, providing theoretical justification for our adaptive strategy. By introducing a principled and adaptive rank allocation mechanism, ElaLoRA offers a scalable and efficient fine-tuning solution, particularly suited for resource-constrained environments. 1 Introduction Scaling laws of transformer-based Pre-trained Language Models (PLMs) ( Vaswani et al., 2017 ; He et al., 2020 ; Liu et al., 2019 ) suggest that increasing model size leads to improved generalization and task performance, which has driven the rapid expansion of model architectures ( Kaplan et al., 2020 ) , from 330M parameters in BERT ( Devlin et al., 2019 ) to 1.5B in GPT-2 ( Radford et al., 2019 ) , 175B in GPT-3 ( Brown et al., 2020 ) , and 671B in DeepSeek ( Bi et al., 2024 ) , highlighting the trend toward ever-larger pretrained models. Despite these advances, Large Language Models (LLMs) remain constrained by their knowledge boundaries, requiring fine-tuning to specialize in domain-specific applications and adapt to evolving datasets ( Brown et al., 2020 ; Achiam et al., 2023 ; Gunter et al., 2024 ) . Traditionally, full fine-tuning has been the standard approach ( Devlin et al., 2019 ; Radford et al., 2019 ) , which is nevertheless prohibitively expensive in terms of memory and computation. To address the computational burden of full fine-tuning, Parameter-efficient fine-tuning (PEFT) methods have been developed ( Ding et al., 2023b ) , with Low-Rank Adaptation (LoRA) ( Hu et al., 2022 ) being a widely used approach that reduces trainable parameters without increasing inference latency. However, LoRA’s fixed rank allocation leads to suboptimal performance by failing to account for layer-specific importance ( Zhang et al., 2023b ) . Dynamic rank allocation methods like AdaLoRA ( Zhang et al., 2023b ) and SaLoRA ( Hu et al., 2023 ) decompose a matrix using singular value decomposition (SVD) and selectively prune its singular values to control the rank of the matrix, but these methods are computationally inefficient as they begin with a high rank. IncreLoRA ( Zhang et al., 2023a ) mitigates this by starting with a minimal rank and increasing it heuristically. However, early training samples may not be effectively learned or utilized when the rank is small. Figure 1: Comparison of ElaLoRA, LoRA, and AdaLoRA. LoRA has fixed ranks, AdaLoRA prunes ranks, while ElaLoRA both prunes and expands ranks during training. R_{i} denotes the importance score of the i^{th} rank. To overcome these limitations, we propose ElaLoRA, a novel adaptive and dynamic LoRA framework that simultaneously prunes and expands ranks (as shown in Figure 1 ). By dynamically reallocating computational resources to the most critical layers, ElaLoRA ensures that essential layers receive more capacity while redundant ranks are removed. ElaLoRA operates through three key components: 1) SVD-based adaptation strategy; 2) importance score calculation to quantify the significance of each rank based on loss gradients; and 3) a dynamic rank learning algorithm that reallocates ranks at scheduled intervals. Experimental results across multiple Natural Language Understanding (NLU) ( Wang et al., 2018 ) , Natural Language Generation (NLG) ( Narayan et al., 2018 ) , and Visual Task ( Zhai et al., 2019 ) benchmarks demonstrate that ElaLoRA consistently outperforms existing PEFT methods under various parameter budgets. Notably, ElaLoRA achieves better average GLUE results with r=2 than other PEFT methods at r=4 , making it particularly well-suited for resource-constrained environments. Our key contributions include: • We introduce ElaLoRA, the first method to the best of our knowledge that enables both rank pruning and expansion simultaneously during fine-tuning. Comparisons are shown in Table 1 . • We conduct extensive experiments across multiple benchmarks under different parameter budgets. Our results consistently demonstrate the effectiveness of ElaLoRA, outperforming existing PEFT methods in performance. • We conduct analysis to verify that the layers and matrices identified as highly important for a specific task are indeed significant for that task, providing a principled validation of our adaptive rank allocation method. Method Pruning Expansion Dynamic Rank Allocation LoRA ( Hu et al., 2022 ) ✗ ✗ Fixed rank for all layers AdaLoRA ( Zhang et al., 2023b ) ✓ ✗ Adaptive pruning via SVD SaLoRA ( Hu et al., 2023 ) ✓ ✗ L_{0} -norm-based adaptive pruning IncreLoRA ( Zhang et al., 2023a ) ✗ ✓ Heuristic-based rank expansion DoRA ( Mao et al., 2024 ) ✓ ✗ Decomposes into rank-one components for pruning AutoLoRA ( Zhang et al., 2024 ) ✓ ✗ Meta-learning-based pruning SoRA ( Ding et al., 2023a ) ✓ ✗ Gated component-wise filtering DyLoRA ( Valipour et al., 2022 ) ✗ ✗ Rank sampling from a predefined distribution ElaLoRA (Ours) ✓ ✓ Fully dynamic rank reallocation Table 1: Comparison of ElaLoRA with existing PEFT methods. ElaLoRA is the only approach that supports both rank pruning and expansion. 2 Background and Related Work Fine-tuning large-scale pre-trained language models (LLMs) is an important technique for adapting them to domain-specific applications. In full model fine-tuning, all model parameters are updated during training, which is computationally expensive and memory-intensive. To address this, Adapter Tuning ( Houlsby et al., 2019 ; Pfeiffer et al., 2020 ; He et al., 2021a ; Rebuffi et al., 2017 ) introduces small trainable modules—adapter layers—inserted between transformer layers. BitFit ( Zaken et al., 2021 ) adopts an even more parameter-efficient strategy by fine-tuning only the bias terms in the model. Low-Rank Adaptation (LoRA) ( Hu et al., 2022 ) has gained popularity due to its strong performance-efficiency tradeoff. QLoRA ( Dettmers et al., 2023 ) builds on this by quantizing the frozen base model to 4-bit precision, significantly reducing memory usage while maintaining performance. However, despite its efficiency, LoRA may limit the model’s ability to memorize domain-specific knowledge and generalize to downstream tasks ( Biderman et al., 2024 ; Han et al., 2024 ; Sui et al., 2024 ; Jiang et al., 2024 ; Zhao et al., 2024 ) . A key limitation lies in its use of a fixed rank across all layers, which overlooks the fact that different layers contribute unequally to model adaptation. This uniform allocation can lead to inefficient use of trainable parameters—underfitting in layers that require more capacity and overfitting or wasted capacity in others. To address this issue, several adaptive rank allocation methods have been developed to dynamically adjust ranks based on importance ( Mao et al., 2025 ) . One approach is

Read the complete text