Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Valérie Castin, Kimia Nadjahi, Pierre Ablin, Gabriel Peyré · 2026

Read Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence by Valérie Castin, Kimia Nadjahi, Pierre Ablin, Gabriel Peyré (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31484.txt.

Introduction

Castin and colleagues present Balanced LoRA, a fix for low-rank adaptation. This is the most widely used way to fine-tune large language models. LoRA is overparameterised: many pairs of low-rank factors give the same adapted weight matrix. Yet those pairs can have very different condition numbers, which changes how fast training converges. Balanced LoRA projects each step onto a balanced manifold. That improves conditioning while keeping the adapted matrix intact, and it converges faster than standard LoRA with better results across fine-tuning tasks.

Catalog record

CREATORValérie Castin, Kimia Nadjahi, Pierre Ablin, Gabriel Peyré
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70975
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.31484

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence Valérie Castin, Kimia Nadjahi, Pierre Ablin, Gabriel Peyré Abstract Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show—both theoretically and empirically—that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks. 1 Introduction Pretrained foundation models are now ubiquitous in natural language processing ( 9 ; 48 ; 53 ) , computer vision ( 4 ) , and multimodal learning ( 30 ; 35 ) , thanks to their ability to generalize from large-scale, diverse training data. Their massive number of parameters allows them to capture a wide range of patterns, making them ideal bases for building specialized fine-tuned models on task-specific data. However, as model sizes expand, full fine-tuning (updating all parameters) becomes impractical due to its prohibitive computational and memory costs. To address this issue, parameter-efficient fine-tuning (PEFT) methods have become increasingly popular ( 22 ) . They leverage the observation that overparameterized models often exhibit low intrinsic dimensionality ( 29 ; 2 ) . These methods adapt large pretrained models by updating only a small subset of parameters, reducing computational costs while preserving performance. Among them, Low-Rank Adaptation (LoRA) ( 23 ) stands out as one of the most effective approaches. Rather than updating dense weight matrices, LoRA uses trainable low-rank matrices added to the frozen pretrained weights. Specifically, a pretrained weight matrix W\in\mathbb{R}^{a\times b} is updated as W+AB , where A\in\mathbb{R}^{a\times r} , B\in\mathbb{R}^{r\times b} , and r\ll\min(a,b) is the LoRA rank. By freezing W , this approach reduces the number of trainable parameters from a\times b (full fine-tuning) to r\times(a+b) , achieving substantial memory savings while preserving adaptability. Given its empirical success across diverse applications, LoRA has recently sparked theoretical interest, though still in its early stages. Recent studies have explored its expressivity in feedforward neural networks and transformers ( 64 ) , analyzed its fine-tuning dynamics in the Neural Tangent Kernel (NTK) regime ( 39 ; 24 ) , examined the distinct roles of the A and B matrices ( 69 ; 20 ) , and investigated the effects of various initialization strategies ( 21 ; 31 ) . In this paper, we analyze the asymptotic behavior of training dynamics of LoRA. Specifically, we bound the asymptotic convergence rate of LoRA when fine-tuning one layer of a deep, possibly non-linear, neural network, by establishing tight bounds on the loss condition number at convergence. Our study hinges on a key property of LoRA: its overparameterization. More precisely, for any invertible matrix R\in\mathbb{R}^{r\times r} , the low-rank factors (AR,R^{-1}B) yield the same adapter AB . In particular, if (A,B) is a minimizer of the loss, then every (AR,R^{-1}B) is also a minimizer, which yields a continuous manifold of minimizers. With that in mind, our novel theoretical analysis highlights that the conditioning of the loss can vary along this manifold: some minimizers are flatter than others, which, from an optimization perspective, makes them better candidates to converge to. Indeed, the loss around a flatter minimizer is better conditioned, so that the asymptotic convergence rate to the minimizer is faster. Our analysis identifies that balanced minimizers —minimizers (A,B) satisfying A^{\top}A=BB^{\top} —achieve optimal conditioning of the loss. Although the balance condition has been studied in other contexts, such as linear networks ( 44 ) , matrix factorization ( 60 ; 18 ) , and conservation laws in neural networks ( 40 ) , no previous work has connected it the conditioning of the loss when performing low-rank adaptation. Similarly, while ( 18 ; 67 ) have introduced balanced parameterizations of the low-rank adapters, they do not show that such factorizations are optimal in terms of conditioning—which is one of the main contributions of this paper, as it provides a principled way to select the best-conditioned parameterization of a fixed adapter matrix. Leveraging these insights, we introduce BaLoRA, an extension of LoRA that enforces balance during training to improve conditioning and accelerate convergence. BaLoRA is computationally lightweight, theoretically grounded, and compatible with standard optimization pipelines. The code for our experiments is available at https://github.com/vcastin/balora . Our contributions are the following: Figure 1 : BaLoRA in a nutshell. BaLoRA projects the low-rank adapters (A,B) on the balanced manifold after each optimizer step. This projection improves the conditioning of the loss while preserving the product \Delta W=AB=A^{\prime}B^{\prime} . • We introduce BaLoRA , a novel PEFT method that enforces balanced low-rank adapters throughout optimization with negligible computational overhead. (Section 3 ) • We theoretically analyze the conditioning of LoRA’s limiting points when fine-tuning one layer of a deep, possibly non-linear network, proving that balanced minimizers exhibit optimal conditioning. Consequently, BaLoRA converges to a better-conditioned minimizer than LoRA, improving its asymptotic convergence rate. (Section 2 ) • When optimizing with gradient descent, we demonstrate that BaLoRA iterations can be reformulated as an intrinsic optimization scheme on the product AB , which provides an elegant and more interpretable geometric perspective on this algorithm. (Section 3 ) • In our experiments on a range of large language models and datasets, BaLoRA consistently outperforms LoRA and matches or surpasses several state-of-the-art LoRA variants from the literature with negligible computational overhead. (Section 4 ) 1.1 Related Works Parameter-efficient fine-tuning. To enable efficient adaptation without full retraining, residual adapters were first introduced for computer vision tasks ( 50 ) and later extended to NLP through adapter-based transfer learning ( 22 ) . Other prominent PEFT strategies include pruning-based approaches, such as Diff-Pruning ( 19 ) , and low-rank adapters ( 23 ) . LoRA and its variants have been widely adopted, ranging from bridging language models with non-language tasks via LIFT ( 14 ) to fine-tuning image generation models ( 17 ) . Theoretical analyses of LoRA in the Neural Tangent Kernel (NTK) regime have been conducted ( 39 ; 24 ) , while its expressive power has been examined ( 64 ) . LoRA optimization. LoRA is typically optimized using Adam ( 26 ) or AdamW ( 38 ) . Recent works have sought to adapt optimization strategies to low-rank structures. Riemannian approaches ( 7 ; 43 ) tackle overparameterization through manifold-based optimization, but often require specialized solvers. Alternative algorithms for matrix factorization have been explored to strengthen convergence guarantees: 66 analyze projected gradient descent and demonstrate convergence rates independent of condition numbers under specific assumptions; 57 study alternating gradient descent, deriving bounds tied to spectral gaps; 65 enhance gradient updates with Riemannian preconditioners; and 45 introduce Gauss–Southwell descent methods, emphasizing step-size and balancing interactions. Although matrix factorization shares similarities with LoRA, these studies do not address the fine-tuning of pretrained machine learning models. Initialization and convergence dynamics.

Read the complete text