Norm-Bounded Low-Rank Adaptation

Ruigang Wang, Krishnamurthy (Dj) Dvijotham, Ian R. Manchester · 2025

Read Norm-Bounded Low-Rank Adaptation by Ruigang Wang, Krishnamurthy (Dj) Dvijotham, Ian R. Manchester (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19050.txt.

Introduction

Ruigang Wang, Krishnamurthy Dvijotham, and Ian Manchester propose norm bounded low rank adaptation for efficient fine tuning of large models. Low rank adaptation updates only small matrices beside frozen weights, yet past versions give no direct control over the size of the update in norm terms. Their parameterization bounds each singular value of the adaptation matrix explicitly.

Any prescribed unitarily invariant norm bound can be enforced, including nuclear, Frobenius, and spectral norms. The form is unconstrained, smooth, and complete, covering every matrix within the rank and value bounds. Language generation tests match or beat rival low rank methods with stronger robustness to hyperparameters. Vision tests show less catastrophic forgetting at minor cost to adaptation gains.

Catalog record

CREATORRuigang Wang, Krishnamurthy (Dj) Dvijotham, Ian R. Manchester
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-19050

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Norm-Bounded Low-Rank Adaptation Ruigang Wang, Krishnamurthy (Dj) Dvijotham, Ian R. Manchester Abstract In this work, we propose norm-bounded low-rank adaptation (NB-LoRA) for parameter-efficient fine tuning. NB-LoRA is a novel parameterization of low-rank weight adaptations that admits explicit bounds on each singular value of the adaptation matrix, which can thereby satisfy any prescribed unitarily invariant norm bound, including the Schatten norms (e.g., nuclear, Frobenius, spectral norm). The proposed parameterization is unconstrained, smooth, and complete, i.e. it covers all matrices satisfying the prescribed rank and singular-value bounds. Natural language generation experiments show that NB-LoRA matches or surpasses performance of competing LoRA methods, while exhibiting stronger hyper-parameter robustness. Vision fine-tuning experiments show that NB-LoRA can avoid model catastrophic forgetting without minor cost on adaptation performance, and compared to existing approaches it is substantially more robust to a hyper-parameters such as including adaptation rank, learning rate and number of training epochs. A Preprint Keywords LoRA \cdot Robustness \cdot Catastrophic Forgetting 1 Introduction Large pretrained vision and language models have demonstrated impressive generalization capability across a wide variety of tasks; see, e.g. Achiam et al. (2023) ; Touvron et al. (2023) . When a more specific target task is identified, however, it has been observed that parameter-efficient fine-tuning (PEFT) techniques, e.g. Houlsby et al. (2019) ; Hu et al. (2022) , can improve performance via quick model adaption with low computation and data requirements. The primary goal for an effective PEFT method is to achieve good adaptation performance with high training efficiency, i.e., dramatically fewer trainable parameters and training epochs. Since training efficiency is the target, ideally such a method will be quite robust to hyper-parameters. Alongside this primary goal, it is often also desirable to maintain the generalization performance of the original pretrained model as much as possible, i.e. avoid “catastrophic forgetting” Qiu et al. (2023) ; Biderman et al. (2024) . Low-rank adaption (LoRA) ( Hu et al., 2022 ) is a widely applied PEFT method, which parameterizes the update of pretrained weights W_{p}\in\mathbb{R}^{m\times n} during finetuning as y=(W_{p}+W)x=\left(W_{p}+\frac{\alpha}{r}B^{\top}A\right)x (1) where A\in\mathbb{R}^{r\times n},B\in\mathbb{R}^{r\times m} are the learnable matrices, \alpha is a scaling factor, and r\ll\min(m,n) is the rank budget of weight adaptation W . Matrix rank is one way to quantify the “size” of a weight, corresponding the underlying dimensionality of its operation. But matrix norms – such as nuclear, Frobenius, or spectral norms – provide another notion of size, quantifying the magnitude of a matrix’s elements and of its operation on vectors. Recent works show that it is beneficial to control the rank and norm of the weight adaption. Jang et al. (2024) ; Kim et al. (2025) show that the global minimum of fine-tuning has low rank and small magnitude while spurious local minima (if they exist) have high rank and large magnitude. Moreover, bounding the magnitude of W can enhance training robustness ( Bini et al., 2025 ) . In Hu et al. (2025) , LoRA training can achieve sub-quadratic time complexity under certain norm-bound conditions. Motivated by those findings, we propose norm-bounded low-rank adaptation (NB-LoRA), a novel finetuning method that admits explicit bounds on both the rank and norm of weight update through matrix reparameterization (see Fig. 1 ). Our approach can control a family of matrix norms, called Schatten p -norms (i.e. p -norms of the singular value sequence), which include the nuclear norm, Frobenius norm, and spectral norm as special cases. We summarize our contributions as follows. Figure 1: Visualization (Left) of the original LoRA Hu et al. (2022) and (Right) of our proposed method NB-LoRA, where bounded rank and norm are enforced by reparameterization \mathcal{W}_{S} . • Our parameterization is a smooth map W=\mathcal{W}_{S}(\tilde{A},\tilde{B}) which takes as argument two free matrix variables of the same size as A,B , but the resulting W automatically satisfies user-prescribed bounds on both rank and all individual singular values of W , which further allows any Schatten p -norm bound on W to be specified. • Our parameterization is complete , i.e., for any W\in\mathbb{R}^{m\times n} satisfying the prescribed bounds on singular values, there exists a (not necessarily unique) \tilde{A},\tilde{B} such that W=\mathcal{W}_{S}(\tilde{A},\tilde{B}) . • LLM fine-tuning experiments show that NB-LoRA can substantially improve training stability, overall performance and robustness to learning rates. • Vision transformer fine-tuning experiments illustrate that NB-LoRA can achieve similar adaptation performance to LoRA and other existing methods while exhibiting less “forgetting” of the source model. Also, norm bounds appear to significantly reduce sensitivity to hyperparameter variation. 2 Related Work LoRA can be highly sensitive to learning rate ( Bini et al., 2024 ; Biderman et al., 2024 ) , model initialization ( Hayou et al., 2024 ) , and it is susceptible to over-training ( Qiu et al., 2023 ) . To mitigate these effects, several recent works have proposed regularization techniques for LoRA. For example, Gouk et al. (2021) ; Chen et al. (2023) propose an approach that preserves the Euclidean weight distances between pre-trained and fine-tuned models. In Liu et al. (2024) , DoRA was proposed based on investigation of the vector-wise norm of the adaption matrix, and introduces an adaptive scaling of W . Bini et al. (2025) proposed DeLoRA - a PEFT method that decouples the angular learning from adaptation strength. VeRA is another method which learns a scaling vector for LoRA weights ( Kopiczko et al., 2024a ) . Our method also contains a learnable scaling vector, which can be used to explicitly control bounds on each singular value of the weight adaptation. Another line of LoRA methods are closely related to singular value decomposition (SVD). Meng et al. (2024) proposed a novel SVD-based LoRA initialization, called PiSSA, which can significantly speed up the training of LoRA. Zhang et al. (2023) proposed a dynamical rank allocation scheme, called AdaLoRA, which adaptively update the rank bound in each LoRA layer. In Lingam et al. (2024) ; Bałazy et al. (2024) , the singular vectors of pretrained weights are re-used and a small square matrices are learned during fine-tuning. No explicit control of norm bounds or constraint on singular values were considered in these methods. 3 Motivating Analysis of LoRA In this section we provide some brief analyses of LoRA that motivate consideration of alternative parameterizations of A and B . Analysis of Gradients and Initialization. For LoRA the standard approach is to initialize with one of A or B equal to zero, so that W=0 initially, and the other as a small random matrix to enable learning but avoid training instability (see Hayou et al. (2024) for a discussion of approaches). Based on Eq. 1 , we can obtain the gradients of the loss \ell(\cdot) with respect to A and B as follows: \frac{\partial\ell}{\partial A}=\frac{\alpha}{r}B\left(\frac{\partial\ell}{\partial y}\right)x^{\top},\quad\frac{\partial\ell}{\partial B}=\frac{\alpha}{r}Ax\left(\frac{\partial\ell}{\partial y}\right)^{\top}. The component \partial\ell/\partial y is typically not large, since the pre-trained base model has reasonable generalization capability over a wide range of tasks. Thus, at the beginning of fine-tuning, if B=0 and A is a small random matrix, then the gradient \partial\ell/\partial A=0 and \partial\ell/\partial B is small and can be noisy. Therefore, both gradients could be small and uninformative for a large number of training steps, leading to slow convergence and poor performance since fine-tuning is often carried out for a few

Read the complete text