RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation

Jun Liu, Zhenglun Kong, Peiyan Dong, Changdi Yang, Xuan Shen, Pu Zhao, Hao Tang, Geng Yuan, Wei Niu, Wenbin Zhang, Xue Lin, Dong Huang, Yanzhi Wang · 2025

Read RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation by Jun Liu, Zhenglun Kong, Peiyan Dong, Changdi Yang, Xuan Shen, Pu Zhao, Hao Tang, Geng Yuan, Wei Niu, Wenbin Zhang, Xue Lin, Dong Huang, Yanzhi Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-04315.txt.

Introduction

Low-rank add-ons are the cheap way to tune big language models, but a snag hides in their scale knob. Jun Liu and twelve co-authors found in 2025 that as the add-on's rank grows, the old scale rule can choke gains or even hurt them. Their fix, RoRA, swaps the scale's plain rank for its square root. That one change lets skill keep rising as rank grows.

Tests ran on three runs of a known model family, in both full and pruned forms. On the base model the new rule beat the classic add-on and a close rival by clear marks. On a model pruned to a fifth of its size, it won back the most lost skill of the three. The team pitch the swap as a free gain for a tool folk already use.

Catalog record

CREATORJun Liu, Zhenglun Kong, Peiyan Dong, Changdi Yang, Xuan Shen, Pu Zhao, Hao Tang, Geng Yuan, Wei Niu, Wenbin Zhang, Xue Lin, Dong Huang, Yanzhi Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-04315

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation Jun Liu, Zhenglun Kong, Peiyan Dong, Changdi Yang, Xuan Shen, Pu Zhao, Hao Tang, Geng Yuan, Wei Niu, Wenbin Zhang, Xue Lin, Dong Huang, Yanzhi Wang Abstract Fine-tuning helps large language models (LLM) recover degraded information and enhance task performance. Although Low-Rank Adaptation (LoRA) is widely used and effective for fine-tuning, we have observed that its scaling factor can limit or even reduce performance as the rank size increases. To address this issue, we propose RoRA (Rank-adaptive Reliability Optimization), a simple yet effective method for optimizing LoRA’s scaling factor. By replacing α / r \alpha/r with α / r \alpha/\sqrt{r} , RoRA ensures improved performance as rank size increases. Moreover, RoRA enhances low-rank adaptation in fine-tuning uncompressed models and excels in the more challenging task of accuracy recovery when fine-tuning pruned models. Extensive experiments demonstrate the effectiveness of RoRA in fine-tuning both uncompressed and pruned models. RoRA surpasses the state-of-the-art (SOTA) in average accuracy and robustness on LLaMA-7B/13B, LLaMA2-7B, and LLaMA3-8B, specifically outperforming LoRA and DoRA by 6.5% and 2.9% on LLaMA-7B, respectively. In pruned model fine-tuning, RoRA shows significant advantages; for SHEARED-LLAMA-1.3, a LLaMA-7B with 81.4% pruning, RoRA achieves 5.7% higher average accuracy than LoRA and 3.9% higher than DoRA. I Introduction Large language models (LLMs) are typically trained on broad datasets during pretraining, which enables the model to develop general language understanding capabilities. Fine-tuning allows the model to perform better on specific tasks or domains. For example, fine-tuning can help the model better handle text in specialized areas such as healthcare or law. Fine-tuning helps reduce biases and generate more relevant and natural text. Moreover, Large-scale deep learning models [ 1 ] , [ 2 ] , [ 3 ] , [ 4 ] , [ 5 ] , [ 6 ] , [ 7 ] , [ 8 ] which often comprise billions or even hundreds of billions of parameters, face limitations in deployment on resource-constrained devices [ 9 ] , [ 10 ] , [ 11 ] , [ 12 ] , [ 13 ] , [ 14 ] , such as mobile phones. Pruning techniques are commonly applied to address these challenges by reducing model size and computational overhead while maintaining performance [ 15 ] , [ 16 ] , [ 17 ] . Parameter-Efficient Fine-Tuning (PEFT) [ 18 ] has also gained prominence as a strategy to reduce the high computational cost of full model fine-tuning. This method allows for efficient task-specific [ 19 ] , [ 20 ] fine-tuning of large language models (LLMs) without the need to retrain all parameters. Fig. 1: Average accuracy of LoRA, DoRA, and ours RoRA for varying ranks for LLaMA-7B on the commonsense reasoning tasks. Our goal is to maximize fine-tuning performance under resource constraints. Fully fine-tuning is costly, and Low-Rank Adaptation (LoRA) [ 21 ] offers an efficient Parameter-Efficient Fine-Tuning (PEFT) approach for language and vision [ 22 ] , [ 23 ] , [ 24 ] , [ 25 ] , [ 26 ] models. The rank r determines the dimensionality of low-rank weight updates, balancing resource efficiency and performance, with LoRA suggests that increasing the rank does not necessarily enhance subspace. We observed that the performance of both LoRA and its improved SOTA version, Weight-Decomposed Low-Rank Adaptation (DoRA) [ 27 ] , declines beyond r=32 , consuming more GPU without gains, as shown by the blue and green line [ 27 ] in Fig. 1 . To address this problem, we propose a method that uses an optimized scaling factor (OpS) \alpha/\sqrt{r} for fine-tuning both uncompressed and pruned [ 28 ] , [ 29 ] , [ 30 ] , [ 31 ] [ 32 ] , [ 33 ] LLMs. While rsLoRA [ 34 ] employs a similar scaling factor to study the impact of the scaling factor on the learning process, our motivation, theoretical derivation, and experimental design are independent. This scaling factor mitigates the impact of rank, ensuring that gradient updates remain independent of rank. Our proposed method, Reliability Optimization for Rank Adaptation (RoRA), outperforms both LoRA and DoRA in fine-tuning uncompressed and pruned models. The main contributions of our work are summarized as follows: • LoRA’s performance improvement was limited and declined with increasing rank. A mathematical analysis identified the scaling factor as a key factor in this decline. • We propose RoRA to address gradient instability caused by varying ranks using the optimization scaling factor (OpS) \alpha/\sqrt{r} . This approach ensures that gradient changes are independent of rank, enhancing stability and performance during optimization. • Extensive experiments on the commonsense reasoning dataset (see Fig. 1 ) show that RoRA outperforms LoRA and DoRA in average accuracy by 6.5% and 2.9% on LLaMA-7B, respectively. • RoRA method consistently improved performance with increasing rank, marking its first use in fine-tuning both pruned and uncompressed large models. II Background and Problem Formulation Pre-trained models like LLaMA [ 35 ] possess broad linguistic knowledge but may lack domain [ 36 ] , [ 37 ] , [ 38 ] specialization. Users often fine-tune them with task-specific data to enhance performance while maintaining overall language. LoRA [ 21 ] is an efficient and widely used fine-tuning method. In LoRA, the update of m_{0} is constrained by a low-rank decomposition: m_{0}+\Delta m=m_{0}+\gamma BA , where B\in\mathbb{R}^{p_{\text{out}}\times r} , A\in\mathbb{R}^{r\times p_{\text{in}}} , and \gamma is the scaling factor. During training, m_{0} remains fixed and does not receive gradient updates, while B and A are trainable parameters. The forward pass is given by: \mathcal{H}(x)=m_{0}x+\Delta mx=(m_{0}+\gamma BA)x, (1) where x\in\mathbb{R}^{p_{\text{in}}} is the input, \mathcal{H}(x)\in\mathbb{R}^{p_{\text{out}}} is the output, m_{0}\in\mathbb{R}^{p_{\text{out}}\times p_{\text{in}}} . The scaling factor \gamma is set to \alpha/r in Equation 1 . III The Proposed Method To address the accuracy drop with increasing LoRA rank r , we analyzed the relationship between gradient variance and rank r . We optimized the scaling factor from \alpha/r to \alpha/\sqrt{r} to ensure that the gradient variance remains unaffected by rank r . This optimized scaling factor is defined as OpS (Optimization Scaling) and the method is referred to as Reliability Optimization for Rank Adaptation (RoRA). III-A Mathematics Analysis for the Weight Variance The relationship between the gradient and its variance is critical: the gradient indicates the loss function’s rate of change, while the variance reflects stability. We analyze how rank r affects both using mathematical techniques. Mathematics Analysis. We represent the increment part of the output \mathcal{H} in eq. 1 by \mathbf{w}\in\mathbb{R}^{p_{\text{out}}} , \mathbf{x}\in\mathbb{R}^{p_{\text{in}}} is the input, and replace the scaling factor \alpha/r with \gamma . Each term w_{i} can be expressed as: \mathbf{w}_{i}=\gamma\sum_{j=1}^{p_{\text{in}}}\sum_{k=1}^{r}\mathbf{B}_{ik}\mathbf{A}_{kj}x_{j}. (2) Using the chain rule to compute the partial derivatives of the loss function L with respect to \mathbf{B}_{ik} and \mathbf{A}_{kj} , we have \frac{\partial L}{\partial\mathbf{A}_{kj}}=\gamma\sum_{i=1}^{p_{\text{out}}}\frac{\partial L}{\partial\mathbf{w}_{i}}\mathbf{B}_{ik}x_{j} , and \frac{\partial L}{\partial\mathbf{B}_{ik}}=\gamma\frac{\partial L}{\partial\mathbf{w}_{i}}\sum_{j=1}^{p_{\text{in}}}\mathbf{A}_{kj}x_{j} . LoRA [ 21 ] sets the learning rate \eta and initializes \mathbf{B}_{ik}=0 [ 39 , 40 ] , a common optimization assumption to analyze early training and parameter scaling effects. After the first step update, \mathbf{A}_{kj} remains unchanged, and \mathbf{B}_{ik} is updated as: \mathbf{B}_{ik}^{(t+1)}=\mathbf{B}_{ik}^{(t)}-\eta\frac{\partial L}{\partial\mathbf{B}_{ik}}=-\eta\frac{\partial L}{\partial\mathbf{w}_{i}}\gamma\sum_{j=1}^{p_{\text{in}}}\mathbf{A}_{kj}x_{j}^{(t)}, (3) where \mathbf{B}_{ik}^{(t)} represents the before updated value of \mathbf{B}_{ik} , and \mathbf{B}_{ik}^{(t+1)} represents its updated value

Read the complete text