Less is More: Resource-Efficient Low-Rank Adaptation

Chunlin Tian, Xuyang Wei, Huanrong Liu, Zhijiang Guo, Li Li · 2025

Read Less is More: Resource-Efficient Low-Rank Adaptation by Chunlin Tian, Xuyang Wei, Huanrong Liu, Zhijiang Guo, Li Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00878.txt.

Introduction

Chunlin Tian, Xuyang Wei, Huanrong Liu, Zhijiang Guo, and Li Li present a leaner fine-tuning method in a 2025 paper. Low-rank adaptation, or LoRA, is a popular way to fine-tune large language models cheaply. Instead of changing every weight, it trains small add-on matrices, yet it still costs a fair amount and can suffer interference on complex data.

Newer variants split LoRA's update matrices to exploit asymmetry, but training stays pricey. The team's resource-efficient version pushes the savings further. Cheaper fine-tuning matters for groups that must adapt big models on limited hardware, a common real-world constraint.

Catalog record

CREATORChunlin Tian, Xuyang Wei, Huanrong Liu, Zhijiang Guo, Li Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00878

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Less is More: Resource-Efficient Low-Rank Adaptation Chunlin Tian, Xuyang Wei, Huanrong Liu, Zhijiang Guo, Li Li Abstract Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), but it still incurs notable overhead and suffers from parameter interference in complex datasets. While recent works decouple LoRA update matrices to exploit matrix-wise asymmetry, training costs remain high. We revisit LoRA from the perspective of inter-matrix and intra-layer parameter redundancy and propose Resource-Efficient Low-Rank Adaptation, EffiLoRA , a lightweight and generalizable approach for language, multimodal, and diffusion models. EffiLoRA employs a unified A matrix across all transformer layers and introduces a runtime selective B matrices update to dynamically trade-off the system resource budget and model performance. EffiLoRA consistently outperforms LoRA across diverse modalities, including commonsense reasoning, visual instruction tuning, and image generation, demonstrating improved efficiency and robustness. 1 Introduction Large Language Models (LLMs; 5 ; 10 ; 1 ; 36 ) offer impressive generalization capabilities but are exceedingly costly to train from scratch. Consequently, fine-tuning pretrained LLMs for multiple downstream tasks has emerged as a prevalent technique to meet domain-specific requirements, effectively balancing performance and resource efficiency. However, full fine-tuning (FFT)—which updates every parameter in models consisting of billions of parameters—remains computationally and memory-intensive. To overcome these limitations, Parameter-Efficient Fine-Tuning (PEFT) methods have been proposed, including LoRA ( 62 ; 17 ; 32 ) , adapters ( 41 ; 16 ; 21 ) , and various derivatives ( 26 ; 24 ; 9 ; 15 ) . PEFT selectively tunes only a subset of model parameters or incorporates specialized modules tailored to specific tasks. By maintaining most of the base model parameters frozen and fine-tuning only a limited number of task-specific parameters, PEFT, like LoRA, substantially decreases computational and memory overhead during both adaptation and deployment phases, thus extending the practical applicability of LLMs. Current research efforts largely aim at enhancing the efficiency of LoRA further, particularly by minimizing the number of trainable parameters ( 61 ; 50 ) . Nevertheless, excessively aggressive parameter reduction may hinder convergence ( 57 ) , while overly cautious approaches risk overfitting. Moreover, PEFT methods ( 20 ; 32 ) inherently underperform compared to FFT due to the limited parameter updates, highlighting an essential trade-off between efficiency and performance. This performance gap becomes particularly evident in complex domains characterized by diverse sub-domains and intricate task distributions ( 11 ; 25 ) . This situation presents a compelling research question: How to achieve high performance and efficient fine‑tuning across heterogeneous domains within tight resource constraints? Recent studies reveal significant parameter redundancy in low-rank adaptation. This redundancy manifests at both the matrix-wise ( 61 ; 48 ; 23 ) and layer-wise ( 56 ; 27 ; 42 ; 40 ) levels, as similar adaptation patterns often recur across different modules, leading to inflated parameter counts. Initial approaches to mitigate this issue involve sharing ( 48 ) or freezing ( 61 ) low-rank matrices across layers. While these techniques reduce parameter overhead, they often do so at the cost of model expressiveness and generality. The tension between efficiency and performance is particularly acute in complex task learning, where one must balance task interference against cross-task synergy. To address this, recent work has explored Mixture-of-Experts (MoE) frameworks ( 13 ; 49 ; 25 ) or has decoupled adapters into shared and task-specific components ( 50 ; 14 ) . However, such modular designs typically increase the total number of tunable parameters, highlighting an ongoing need for a more efficient trade-off between adaptation capacity and parameter efficiency. To address these challenges, we introduce Resource-Efficient Low-Rank Adaptation, EffiLoRA , a lightweight and generalizable framework designed to mitigate both parameter redundancy and interference. In particular, EffiLoRA tackles redundancy by employing a single, unified low-rank matrix A across all transformer layers. This design enforces a common adaptation subspace, thereby eliminating repetitive per-layer parameters. Meanwhile, EffiLoRA introduces a selective B matrices update to enhance both efficiency and robustness, further reducing parameter overhead. For complex settings, EffiLoRA deploys parallel, task-specific “B-heads” that learn distinct transformations while leveraging the shared subspace defined by A. This modular architecture effectively reduces potentially conflicting task objectives, mitigating interference and promoting shared knowledge transfer. The resulting design strikes a principled balance between parameter efficiency and model expressiveness, enabling scalable fine-tuning under stringent resource constraints. Across diverse natural language, multimodal, and diffusion tasks, EffiLoRA consistently outperforms strong baselines, establishing it as a robust and efficient framework for fine-tuning in complex, heterogeneous scenarios. 2 Background and Motivation 2.1 Low-Rank Adaptation Low-Rank Adaptation (LoRA) ( 17 ) is an efficient fine-tuning technique for large pre-trained models, introducing small low-rank matrices ( A and B ) that can be applied to arbitrary linear layers. Formally, for a linear transformation h=Wx with input x\in\mathbb{R}^{d_{i}} and weight W\in\mathbb{R}^{d_{o}\times d_{i}} , LoRA learns a low-rank decomposed update: y^{\prime}=y+\Delta y=Wx+BAx, (1) where y\in\mathbb{R}^{d_{o}} is the output, and A\in\mathbb{R}^{r\times d_{i}} , B\in\mathbb{R}^{d_{o}\times r} are low-rank matrices with r\ll\min(d_{o},d_{i}) as the chosen rank. Typically, B is initialized to zeros, while A follows a Gaussian matrix. During fine-tuning, only A and B are updated, keeping the original model parameters frozen, thus significantly reducing computational overhead. 2.2 Observations In this subsection, we revisit LoRA to analyze the trade-off between expressiveness and parameter efficiency and conduct systematic experiments that shed light on its underlying mechanisms. Observation I: LoRA exhibits significant parameter redundancy at both the inter-matrix and intra-layer level. Inter-matrix : for a single LoRA adapter with matrices A and B , recent studies ( 50 ; 14 ) observe that that the down-projection matrix A converges to a strikingly similar subspace across different layers. Consequently, strategies like freezing ( 61 ) and sharing ( 48 ) the A matrix after initialization can effectively capture this common basis while eliminating redundant parameters. As shown in Figure 2 , both approaches perform comparably to—and sometimes slightly better than—vanilla LoRA, confirming the high degree of inter-matrix redundancy. Intra-layer : Prior work has established that different layers in an LLM contribute unequally to fine-tuning, with adaptation often concentrated in a small subset of layers ( 56 ; 27 ; 42 ; 40 ) . Building on this insight, we find that the LoRA matrices themselves exhibit a similar pattern of varying importance. To demonstrate this, we adopt a shared- A design and randomly prune N layer-specific B matrices. For a 32-layer model, this reduces the parameter budget from (A+B)\times 32 to A+B\times(32-N) . As illustrated in Figure 2 , experiments on Llama-3-8B show that discarding 50% of the B matrices degrades performance by a mere 2.4%. This result indicates a long-tailed utility distribution, where a large fraction of layer-specific adapters are expendable and can be pruned with minimal impact on performance. Figure 1: Matrix-wise optimization of LoRA. Figure 2: Impact of dropping different numbers of B modules. Figure 3: Performance comparison on heterogeneous data on LLaVA-7B ( 30 ) , evaluated on the VizWiz dataset ( 3 ) . Observation II: Fine-tuning on heterogeneous data reveals a tension between task-specific conflicts and latent cross-domain commonalities. We illustrate this tension by fine-tuning LLaVA-v1.5-7B ( 30 ) on a mixed dataset from two distinct domains: Visual Question Answering (VQA) ( 2 ) and

Read the complete text