Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation
Yize Wu, Ke Gao, Ling Li, Yanjun Wu · 2026
Read Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation by Yize Wu, Ke Gao, Ling Li, Yanjun Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05204.txt.
Introduction
Yize Wu and team read feature-learning balance in Low-Rank Change, a setting-lean fine-tuning method updating a frozen weight matrix through low-rank factors. They establish that LoRA can sustain feature learning on its own under suitable hyperparameters, yet the required nonzero initialization of one factor undermines that balance and hurts scores. Stable-LoRA progressively shrinks that factor in the earliest steps, restoring balance while keeping the nonzero start's benefits. Tests show it beats older models across models and tasks with no extra memory and negligible compute.
Catalog record
| CREATOR | Yize Wu, Ke Gao, Ling Li, Yanjun Wu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40955 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05204
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation Yize Wu, Ke Gao, Ling Li, Yanjun Wu Abstract Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient method for fine-tuning Large Langauge Models. It updates the weight matrix as W = W 0 + s B A W=W_{0}+sBA , where W 0 W_{0} is the original frozen weight, s s is a scaling factor and A A , B B are trainable low-rank matrices. Despite its robust empirical effectiveness, the theoretical foundations of LoRA remain insufficiently understood, particularly with respect to feature learning stability. In this paper, we first establish that, LoRA can, in principle, naturally achieve and sustain stable feature learning (i.e., be self-stabilized) under appropriate hyper-parameters and initializations of A A and B B . However, we also uncover a fundamental limitation that the necessary non-zero initialization of A A compromises self-stability, leading to suboptimal performances. To address this challenge, we propose Stable-LoRA, a weight-shrinkage optimization strategy that dynamically enhances stability of LoRA feature learning. By progressively shrinking A A during the earliest training steps, Stable-LoRA is both theoretically and empirically validated to effectively eliminate instability of LoRA feature learning while preserving the benefits of the non-zero start. Experiments show that Stable-LoRA consistently outperforms other baselines across diverse models and tasks, with no additional memory usage and only negligible computation overheads. The code is available at https://github.com/Yize-Wu/Stable-LoRA. 1 Introduction Low-Rank Adaptation (LoRA) ( 12 ) is an effective and widely adopted parameter-efficient method for fine-tuning Large Language Models (LLMs). Unlike full fine-tuning, which updates all model parameters, LoRA freezes the original weight matrix W_{0} and introduces two low-rank trainable matrices, A and B , with the weight matrix updated by the multiplication of A and B . Formally, W=W_{0}+sBA,W_{0}\in\mathbb{R}^{m\times n},A\in\mathbb{R}^{r\times n},B\in\mathbb{R}^{m\times r} where s is a scaling factor. By choosing r<<\min(m,n) , the number of trainable parameters is reduced from mn to (m+n)r , substantially lowering computation and memory overhead while retaining strong learning capacity. The effectiveness of LoRA has been demonstrated through massive experiments across various models and tasks, and recent studies ( 8 ; 28 ; 15 ) have also begun to theoretically explore the fine-tuning dynamics of LoRA. However, no prior work has established a theoretical explanation for such robust effectiveness. In this paper, we first provide a theoretical analysis showing that, with appropriate hyper-parameters and initializations of A and B , LoRA can naturally achieves stable feature learning with respect to model width n (informally, the learned features scale as \Theta(n^{0}) ). Furthermore, once this stability is achieved, it will be sustained throughout the entire training process. Such self-stabilizing property provides a theoretical foundation for the observed effectiveness and robustness of LoRA. Figure 1: Illustration of Stable-LoRA. The weight-shrinkage operation is emphasized as a patch to the gradient-descent procedure. According to the analysis, the ideal initialization for ensuring self-stabilization is to set both A and B to zero. However, this leads to practical issues of saddle-point halting ( 28 ) , information loss and gradient vanishing/explosion ( 11 ) . The mostly-adopted and theoretically proven-effective ( 9 ) solution is to initialize only B to zero and A non-zero. Nevertheless, we both theoretically and empirically demonstrate that such a non-zero initialization A_{0} compromises stable feature learning and hence causes suboptimal performances, which motivates the design of novel LoRA optimization strategies. To address this problem, we propose Stable-LoRA, a weight-shrinkage strategy for LoRA optimization that dynamically enhances the stability of feature learning. We conclude from theoretical perspectives that the initialization-induced instability is a long-term problem whereas others are short-termed. Therefore, Stable-LoRA adopts the non-zero A_{0} for its benefits and progressively shrinks A as training proceeds. Specifically, a shrinkage ratio \lambda ( 0<\lambda<1 ) is applied to A in the earliest steps of training, updating A according to A_{t+1}=(1-\lambda)A_{t}-\eta g_{A}^{t} (as shown in Figure 1 ).This exponential decay diminishes the instability introduced by A_{0} while still preserving its advantages for early training. Shrinkage stops once the stability condition is satisfied—specifically, when the average norm of A becomes no larger than that of B (see Section 4 ). We theoretically proved that sufficient shrinkage of A guarantees the prevention of potential instability, thereby ensuring stable feature learning throughout training. We evaluated Stable-LoRA across different model architectures and tasks, where it uniformly outperforms AdamW and other baselines. Importantly, Stable-LoRA incurs no additional memory usage and introduces only negligible computational overhead—properties that are particularly important in the resource-constrained scenarios where LoRA is most commonly applied. 2 Preliminary 2.1 Feature learning of LoRA Consider training a weight matrix W with input Z , such that the output is Y=WZ . In LoRA, the original weight is frozen as W_{0} and two low-rank trainable matrices A and B are introduced, so that the updated weight becomes W=W_{0}+sBA , where s is a scaling factor. Given a learning rate \eta , the parameter updates at training step t are: A_{t+1}=A_{t}-\eta g_{A}^{t},B_{t+1}=B_{t}-\eta g_{B}^{t} , where g_{A} and g_{B} are the optimizer-processed gradients. The change in output after these updates is given by: \displaystyle\Delta Y_{t} \displaystyle=s(A_{t}-\eta g_{A}^{t})(B_{t}-\eta g_{B}^{t})Z-sA_{t}B_{t}Z (1) \displaystyle=-s\eta g_{B}^{t}A_{t}Z-s\eta B_{t}g_{A}^{t}Z+s\eta^{2}g_{B}^{t}g_{A}^{t}Z . We are particularly interested in the properties of \Delta Y_{t} , as it serves as the “learned feature” at step t . Specifically, \Delta Y_{t} directly influences the inputs to downstream layers and ultimately the model’s output, representing the contribution of LoRA updates to the model. 2.2 Stable feature learning As neural networks continue to scale, understanding their training dynamics with respect to parameter growth becomes increasingly important ( 8 ; 28 ; 7 ) . Much of this analysis has focused on the regime of model width, since in most architectures the parameter count is dominated by width, while depth (i.e., the number of layers) plays a comparatively smaller role. In this regime, a desirable property is that learned features remain “stable” as width increases—they neither explode nor vanish numerically. Such stability is crucial for ensuring meaningful representations can be learned, thereby allowing the model to achieve its full performance potential upon trained tasks. In the context of LoRA, stable feature learning requires that the output update \Delta Y_{t} does not scale positively or negatively with model width n , otherwise it would explode or vanish as n increases. Formally, this requirement can be expressed as \Delta Y_{t}=\Theta(n^{0})=\Theta(1) . Definition 1 . (LoRA stable feature learning) LoRA feature learning is stable, if \Delta Y_{t}=\Theta(1) for all training steps t . Note that Definition 1 is slightly different from similar concepts in prior works ( 8 ; 28 ) . It does not require intermediate representations (e.g. A_{t}Z ) to individually scale as \Theta(1) , but only constrains the final output update. This relaxation is motivated by practical considerations: for an actual (finite) n , the scale of intermediate representations can compensate for each other through multiplicative interactions and yield an overall stable output. For example, it is acceptable for components of \Delta Y_{t} to scale as U=\Theta(n) and V=\Theta(n^{-1}) , as long as \Delta Y_{t}=UV=\Theta(1) is ensured. 2.3 \gamma -function For convenience of notation, we introduce \gamma -function to characterize the scaling behavior of variables with respect to the model width n . It is defined as follows: for a real-valued