LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won’t Fail)

Junsu Kim, Jaeyeon Kim, Ernest K. Ryu · 2025

Read LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won’t Fail) by Junsu Kim, Jaeyeon Kim, Ernest K. Ryu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09376.txt.

Introduction

Junsu Kim, Jaeyeon Kim, and Ernest Ryu study why low-rank adaptation training works, in a research paper from 2025. The method, known as LoRA, fine-tunes large models by learning small low-rank updates. It is now standard, yet past theory relied on simple linear views or narrow setups that miss real training.

The authors sort training into a special regime and a generic, more realistic regime. In the generic case, training ends at a low-rank global best fit, or at a clearly distinct high-rank result. Starting at zero and shrinking weights bias the search toward the good region. The work explains why the method usually succeeds.

Catalog record

CREATORJunsu Kim, Jaeyeon Kim, Ernest K. Ryu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09376

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won’t Fail) Junsu Kim, Jaeyeon Kim, Ernest K. Ryu Abstract Low-rank adaptation (LoRA) has become a standard approach for fine-tuning large foundation models. However, our theoretical understanding of LoRA remains limited as prior analyses of LoRA’s training dynamics either rely on linearization arguments or consider highly simplified setups. In this work, we analyze the LoRA loss landscape without such restrictive assumptions. We define two regimes: a “special regime”, which includes idealized setups where linearization arguments hold, and a “generic regime” representing more realistic setups where linearization arguments do not hold. In the generic regime, we show that LoRA training converges to a global minimizer with low rank and small magnitude, or a qualitatively distinct solution with high rank and large magnitude. Finally, we argue that the zero-initialization and weight decay in LoRA training induce an implicit bias toward the low-rank, small-magnitude region of the parameter space—where global minima lie—thus shedding light on why LoRA training usually succeeds in finding global minima. 1 Introduction With the recent explosive trend of scale, fine-tuning a pre-trained foundational model to target downstream tasks has become a dominant approach to deep learning. Low-rank adaptation (LoRA) ( Hu et al., 2022 ) is a parameter-efficient fine-tuning method freezing the pre-trained weight matrix W_{0}\in\mathbb{R}^{m\times n} , and training a low-rank update X=AB^{\intercal} to it using W=W_{0}+X=W_{0}+AB^{\intercal}, where r\ll\min(m,n) , A\in\mathbb{R}^{m\times r} and B\in\mathbb{R}^{n\times r} . The low-rank factor matrices A and B are respectively initialized as a random Gaussian matrix and a zero matrix, leading to X=0 at initialization. By training fewer parameters, LoRA fine-tuning significantly reduces memory usage, making fine-tuning feasible on GPUs with limited GPU memory. 0 \text{rank}\leq r_{\star} r_{\star}<\text{rank}<r \text{rank}=r X_{\star} X_{\text{spurious}} X_{\text{spurious}} X_{\text{spurious}} Figure 1: In LoRA fine-tuning, under the assumption that the global minimum X_{\star} has low rank and small magnitude, we show that spurious local minima X_{\text{spurious}} may exist, but they have high rank and large magnitude. The broad use of LoRA has spurred theoretical works aimed at understanding its effectiveness. One line of work focuses on analyzing LoRA’s training dynamics, exploring why optimizers like SGD or Adam successfully find effective low-rank updates despite the significant non-convexity introduced by the factorization X=AB^{\intercal} , as well as the inherent non-convexity of neural networks, by utilizing some degree of linearization. Specifically, Malladi et al. (2023) studies LoRA under a complete linearization, effectively holding A fixed during fine-tuning and viewing the training as a convex optimization problem. A subsequent work ( Jang et al., 2024 ) presents a more refined analysis linearizing with respect to the product X=AB^{\intercal} , retaining the non-convexity arising from the interaction between A and B . Beyond linearization, Dayi & Chen (2024) analyzes a two-layer teacher-student setup for rank- 1 LoRA. In this work, we carry out a theoretical analysis without any linearizations and any restriction on layers or LoRA rank. Contribution. We analyze the loss landscape of LoRA fine-tuning and show that in the “generic regime”, a more practical setup where linearization arguments do not hold, a local minimizer is either (i) a global minimizer with small rank and small magnitude or (ii) a spurious local minimizer with high rank and large magnitude. We further argue that the zero-initialization and weight decay in LoRA training induce an implicit bias toward the low-rank, small-magnitude region of the parameter space, where global minima lie. Altogether, we shed light on why practical LoRA training effectively converges to global minima. Our key assumptions, formally defined and justified in Section 2 , are the existence of a low-rank global minimizer for full fine-tuning, restricted strong convexity, and restricted smoothness. Notably, our analysis does not rely on any linearization arguments , making it more applicable to practical fine-tuning setups compared to prior work. 1.1 Prior works PEFT methods and LoRA Parameter-Efficient Fine-tuning (PEFT) methods have emerged as effective approaches for fine-tuning large language models on downstream tasks while reducing computational and storage requirements. Among numerous proposed methods ( Ben Zaken et al., 2022 ; Li & Liang, 2021 ; Lester et al., 2021 ) , Low-Rank Adaptation (LoRA) ( Hu et al., 2022 ) has become a predominant approach by decomposing weight updates into low-rank matrices. Several variants such as LoRA+ ( Hayou et al., 2024 ) , rsLoRA ( Kalajdzievski, 2023 ) , PiSSA ( Meng et al., 2024 ) , and MiLoRA ( Wang et al., 2025 ) have been built upon the LoRA framework, addressing the discrepancy with full fine-tuning in optimization and performance. Theoretical foundation of LoRA. Existing theoretical works on LoRA focus on the expressive power and the training dynamics of LoRA. Zeng & Lee (2024) demonstrates that a certain LoRA rank suffices to express a given fine-tuning function. Jang et al. (2024) proves that under the NTK regime, LoRA with rank \Omega(\sqrt{N}) can express the global minimizer of the original model. Malladi et al. (2023) argues that the LoRA fine-tuning dynamics are nearly equivalent to the kernel regression. Under this framework, Jang et al. (2024) proves LoRA fine tuning loss has no spurious local minima when the rank is O(\sqrt{N}) . Beyond the kernel regime, Dayi & Chen (2024) analyzes a two-layer teacher-student setup for LoRA and explains why SGD leads to convergence to a global minimum in this context. Zhang et al. (2025) also identifies the training dynamics in a 2-layer setup, proving LoRA will align to a singular subspace of one-step gradient of full fine-tuning. Low-rank optimization. The low-rank optimization problem \displaystyle\min_{X\in\mathbb{R}^{m\times n},\,\mathrm{rank}(X)\leq r}f(X) has been extensively studied in the optimization literature, including matrix sensing ( Recht et al., 2010 ) and matrix completion ( Candès & Recht, 2012 ) . Rather than directly optimizing over the space of low-rank matrices, it is often preferred to employ the Burer-Monteiro factorization ( Burer & Monteiro, 2003 ) , which formulates the problem by parameterizing X as X=UV^{\intercal} , U\in\mathbb{R}^{m\times r},V\in\mathbb{R}^{n\times r} . As the Burer-Monteiro factorization introduces nonconvexity, a large body of work has identified conditions under which this approach avoids spurious local minima ( Bhojanapalli et al., 2016 ; Ge et al., 2017 ; Park et al., 2017 ; Zhang, 2021 ) . Further studies extend these results to general settings ( Ha et al., 2020 ; Zhang, 2024 ) . In our work, we utilize the framework established in these studies with novel techniques to extend its boundary to optimization guarantees in LoRA training. 1.2 Notation and preliminaries Matrix notation. For X\in\mathbb{R}^{m\times n} , denote its singular values as \sigma_{1}(X)\geq\sigma_{2}(X)\geq\dots\geq\sigma_{r}(X)\geq 0 . For matrices A and B , let \left\|A\right\|_{2}=\sigma_{1}(A) denote the spectral norm, \|A\|_{*}=\sum\sigma_{i}(A) the nuclear norm, \left\|A\right\|_{F}=\sqrt{\sum\sigma_{i}(A)^{2}} the Frobenius norm, and \langle A,B\rangle=\operatorname{tr}(A^{\intercal}B) the matrix inner product. For a tuple of matrices \mathbf{A}=(A^{(1)},\dots,A^{(L)}) , denote \|\mathbf{A}\|=\sum_{l=1}^{L}\|A^{(l)}\| for any matrix norm \|\cdot\| and \textrm{rank}(\mathbf{A})=\max_{1\leq l\leq L}\text{rank}(A^{(l)}) . Neural network. Let f(\cdot\ ;\cdot):\mathcal{P}\times\mathcal{X}\rightarrow\mathbb{R}^{K} be a neural network where \mathcal{P} is the parameter space, \mathcal{X} is the data space, and \mathbb{R}^{K} is the output space. Assume the model is pre-trained to \Theta_{0}\in\mathcal{P} , i.e., the pre-trained model is f(\Theta_{0};\cdot) . Fine-tuning loss. Let \mathbf{W}_{0}=(W_{0}^{(1)},\dots,W_{0}^{(L)})\subset\Theta_{0} be the pre-trained value of the weights \mathbf{W} that we choose to fine-tune. We wish to

Read the complete text