GoRA: Gradient-driven Adaptive Low Rank Adaptation

Haonan He, Peng Ye, Yuchen Ren, Yuan Yuan, Luyang Zhou, Shucun Ju, Lei Chen · 2025

Read GoRA: Gradient-driven Adaptive Low Rank Adaptation by Haonan He, Peng Ye, Yuchen Ren, Yuan Yuan, Luyang Zhou, Shucun Ju, Lei Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12171.txt.

Introduction

He and colleagues propose in 2025 GoRA, a fine-tuning method that decides both the size and the starting values of low-rank adapters from gradient information. Low-rank adaptation is the standard cheap way to tune large language models. Earlier variants fixed either rank choice or initialization, not both, and often cost extra compute. GoRA adapts both inside one framework during training.

Across architectures and data types, it beat other adapter methods while keeping the original's efficiency. On a math reasoning task it topped standard adaptation by several points. It even beat full fine-tuning in a high-rank setting.

Catalog record

CREATORHaonan He, Peng Ye, Yuchen Ren, Yuan Yuan, Luyang Zhou, Shucun Ju, Lei Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12171

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GoRA: Gradient-driven Adaptive Low Rank Adaptation Haonan He, Peng Ye, Yuchen Ren, Yuan Yuan, Luyang Zhou, Shucun Ju, Lei Chen Abstract Low-Rank Adaptation (LoRA) is a crucial method for efficiently fine-tuning large language models (LLMs), with its effectiveness influenced by two key factors: rank selection and weight initialization. While numerous LoRA variants have been proposed to improve performance by addressing one of these aspects, they often compromise usability or computational efficiency. In this paper, we analyze and identify the core limitations of existing approaches and propose a novel framework— GoRA ( G radient-driven Adaptive L o w R ank A daptation)—that simultaneously adapts both the rank and initialization strategy within a unified framework. GoRA leverages gradient information during training to dynamically assign optimal ranks and initialize low-rank adapter weights in an adaptive manner. To our knowledge, GoRA is the first method that not only addresses the limitations of prior approaches—which often focus on either rank selection or initialization in isolation—but also unifies both aspects within a single framework, enabling more effective and efficient adaptation. Extensive experiments across various architectures and modalities show that GoRA consistently outperforms existing LoRA-based methods while preserving the efficiency of vanilla LoRA. For example, when fine-tuning Llama3.1-8B-Base for mathematical reasoning, GoRA achieves a 5.13-point improvement over standard LoRA and even outperforms full fine-tuning by 2.05 points under high-rank settings. Code is available at: https://github.com/hhnqqq/MyTransformers . 1 Introduction Open-source pre-trained large language models (LLMs) such as the Llama series [ 43 , 11 ] have demonstrated exceptional capabilities. Through supervised fine-tuning, these models can be adapted to various downstream tasks such as code generation [ 41 ] and mathematical problem solving [ 50 ] . However, when the model has a parameter size \phi and uses FP16/BF16 mixed-precision training strategy [ 33 , 24 ] with the Adam optimizer [ 25 ] , the parameters and gradients require 4\phi bytes of memory, while the optimizer states require 12\phi bytes. Thus, the minimum memory usage, excluding activations, reaches 16\phi bytes. Such high memory demands limit the training of large language models under constrained resources. To reduce memory usage, Low-Rank Adaptation (LoRA) [ 21 ] decomposes the weight matrix \mathbf{W}\in\mathbb{R}^{m\times n} into \mathbf{W}=\mathbf{W}_{0}+\Delta\mathbf{W}=\mathbf{W}_{0}+s\mathbf{AB} , where s is a scaling factor, and \mathbf{A}\in\mathbb{R}^{m\times r},\mathbf{B}\in\mathbb{R}^{r\times n},r\ll\text{min}(m,n) , as shown in Figure 1 (a). LoRA only updates the low-rank weights \mathbf{A} and \mathbf{B} , keeping the pre-trained weight \mathbf{W}_{0} unchanged, thereby significantly reducing the memory footprint of optimizer states. Although LoRA performs well on simple tasks, when applied to pre-trained large language models, its performance on more challenging tasks, such as mathematical reasoning and code generation, still lags behind full fine-tuning [ 2 , 13 ] . Figure 1: Illustration of (a) LoRA; (b) LoRA variants utilizing adaptive rank masking strategies; (c) LoRA variants employing nonzero initialization strategies; and (d) GoRA, which introduces adaptively leveraging the weight \mathbf{W} ’s gradient to allocate the rank of the low-rank adapter and initialize \mathbf{B} . \mathbf{A}_{0} and \mathbf{B}_{0} denote the initialized value of \mathbf{A} and \mathbf{B} . One of the critical factors in LoRA is its rank. Kalajdzievski et al. [ 23 ] demonstrate that increasing the rank of LoRA can significantly improve performance when paired with an appropriate scaling factor. However, a direct increase in rank leads to a substantial rise in memory requirement overhead, thus imposing constraints on rank selection. To address this, several studies [ 27 , 40 ] propose to ensemble multiple low-rank subspaces, allowing for rank increases without proportionally increasing the number of trainable parameters. Nevertheless, these approaches often come at the expense of usability due to their intrusion into architecture or training processes. Another promising line of research explores adaptively assigning ranks to pre-trained weights based on importance. For example, AdaLoRA [ 54 ] adaptively adjusts ranks by quantifying the importance of each rank during training and masking less significant ones, as illustrated in Figure 1 (b). However, this masking mechanism necessitates a larger parameter space ( e.g. , 1.5 times), increasing the number of trainable parameters and limiting the upper bound of rank. Consequently, as demonstrated in Section 5.1 , adaptively allocating ranks without significantly increasing the training cost remains an open challenge. Another vital factor in LoRA is its initialization strategy. In vanilla LoRA, \mathbf{A}_{0} is initialized with a normal distribution (In the PEFT library, \mathbf{A}_{0} is initialized with a Kaiming distribution [ 17 ] ) and \mathbf{B}_{0} is initialized with zeros. This initialization method ensures that the weights \mathbf{W}_{0}+\mathbf{A}_{0}\mathbf{B}_{0} remain unchanged at the beginning of training. Besides, zero initialization is not the only option: When \mathbf{A}_{0}\mathbf{B}_{0} is nonzero, manipulating the pre-trained weight by subtracting \mathbf{A}_{0}\mathbf{B}_{0} from \mathbf{W}_{0} also ensures stability. Existing nonzero initialization methods can be categorized into experience-driven and data-driven methods. In experience-driven methods, PiSSA [ 31 ] and MiLoRA [ 45 ] employ decomposition techniques such as Singular Value Decomposition (SVD) to capture specific features of pre-trained weights. However, these methods are inherently task-agnostic, which limits their generalizability across diverse tasks. In contrast, data-driven methods incorporate task information. For example, LoRA-GA [ 46 ] uses the singular features of gradients to initialize LoRA weights, minimizing the difference between LoRA and full fine-tuning. However, as illustrated in Figure 1 (c) and Section 2.2 , existing nonzero methods require manipulating the pre-trained weights, resulting in a training-inference gap. Thus, designing a nonzero initialization method without manipulating pre-trained weights remains an open problem. Given the challenges of adaptive rank allocation and nonzero initialization, we turn to gradients of pre-trained weights, which are crucial for assessing the importance of pre-trained weights and deeply related to LoRA adapters’ optimization processes [ 15 , 56 ] . As shown in Figure 1 (d), we propose GoRA. Specifically, before training, we compute gradients of pre-trained weights on a subset of training samples, using these gradients to assess the importance of each pre-trained weight. Given a reference rank, we calculate a trainable parameter budget. Based on the normalized importance and the trainable parameter budget, we allocate a new trainable parameter count and corresponding rank for each low-rank adapter, achieving adaptive rank allocation without significantly increasing the trainable parameter count compared to LoRA and allowing for higher rank allocation upper bounds, as shown in Table 9 . In GoRA’s initialization, we maintain LoRA’s initialization for \mathbf{A} , while \mathbf{B} is initialized using -(\mathbf{A}_{0}^{\top}\mathbf{A}_{0})^{-1}\mathbf{A}_{0}^{\top}\mathbf{G} , where \mathbf{G} is the gradient of the weight W . This initialization ensures that the computation result of the low-rank adapter -\mathbf{A}_{0}(\mathbf{A}_{0}^{\top}\mathbf{A}_{0})^{-1}\mathbf{A}_{0}^{\top}\mathbf{G}\approx-\mathbf{G} compresses the gradient optimally, setting a solid foundation for further optimization without manipulating the pre-trained weight. Our key contributions are summarized as follows: 1. We conduct an in-depth investigation into LoRA’s rank allocation and initialization strategy, uncovering the limitations of existing works. We propose GoRA, which achieves adaptive rank allocation and initialization without compromising usability and efficiency. 2. We use the gradients of weights to assess their importance and allocate ranks. We then initialize the low-rank weights using the pseudo-inverse of the compressed gradients, which enhances performance while ensuring training stability. 3. We conduct extensive experiments, demonstrating that GoRA consistently outperforms low-rank baselines and even rivals full fine-tuning in certain settings. For example, on the Llama3.1-8B-Base model fine-tuned for mathematical reasoning, GoRA achieves a 5.13-point improvement over

Read the complete text