KeepLoRA: Continual Learning with Residual Gradient Adaptation

Mao-Lin Luo, Zi-Hao Zhou, Yi-Lin Zhang, Yuanyu Wan, Min-Ling Zhang, Tong Wei · 2026

Read KeepLoRA: Continual Learning with Residual Gradient Adaptation by Mao-Lin Luo, Zi-Hao Zhou, Yi-Lin Zhang, Yuanyu Wan, Min-Ling Zhang, Tong Wei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19659.txt.

Introduction

Mao-Lin Luo and five colleagues teach AI models new tasks without erasing old ones. Continual learning is hard because training on new data tends to overwrite what came before. For pretrained vision-language models, three goals compete. The model must keep its original knowledge, retain the tasks it already learned, and stay flexible enough to learn more.

Their method, KeepLoRA, adapts the model with small low-rank updates. A residual gradient adaptation steers those updates to protect earlier knowledge. The authors present it as a simple approach that balances the three goals well. Models that learn continually could stay useful for years instead of being retrained from scratch. The work targets vision-language systems used in changing real-world settings.

Catalog record

CREATORMao-Lin Luo, Zi-Hao Zhou, Yi-Lin Zhang, Yuanyu Wan, Min-Ling Zhang, Tong Wei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19659

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

KeepLoRA: Continual Learning with Residual Gradient Adaptation Mao-Lin Luo, Zi-Hao Zhou, Yi-Lin Zhang, Yuanyu Wan, Min-Ling Zhang, Tong Wei Abstract Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances these three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA . 1 Introduction (a) General-domain tasks (b) Specific-domain tasks (c) Performance comparison Figure 1: Analysis of model parameter subspaces and overall CL performance. In Fig. 1(a) and 1(b) , we measure zero-shot performance after reconstructing attention weights using only the top principal singular components. While performance on general-domain datasets remains highly robust, performance on most specific-domain datasets degrades sharply as more low-energy components are removed. In Fig. 1(c) , the Last metric measures the accuracy gain on the final learned task relative to a zero-shot baseline, while Transfer measures the accuracy degradation on unseen tasks. Vision-language models (VLMs) have demonstrated remarkable zero-shot transfer capabilities, making them cornerstones of many downstream applications ( 8 ; 1 ; 49 ) . Despite this success, their performance on certain datasets can be insufficient, motivating the need for continual learning (CL). An effective CL method requires balance of three competing objectives: maintaining the ability to learn new knowledge ( plasticity ), preventing the forgetting of previously learned tasks ( backward stability ), and crucially, preserving the general pre-trained knowledge that guarantees general transferability ( forward stability ) ( 43 ; 72 ) . Degradation of this pre-trained knowledge is particularly detrimental, as it erodes the core value of VLMs. Therefore, the central challenge is how to learn new knowledge effectively without undermining these critical stability constraints. Table 1: Comparison of LoRA-based approaches to continual learning. When learning the t -th task, O-LoRA regularizes the down-projection matrix to be orthogonal to that of previous tasks to improve stability ; InfLoRA constrains the optimization of task features \bm{H}_{t} to be orthogonal to previous dominant directions \bm{M}_{t-1} to jointly improve plasticity and stability; SD-LoRA optimizes a decoupled LoRA to improve stability and re-scales the magnitudes \left\{\alpha_{i}\right\}_{i=1}^{t-1} of parameters from previous tasks to improve plasticity. By contrast, our method constrains optimization to a subspace orthogonal to both the principal weight subspace \bm{W}_{p} and previous task directions \bm{M}_{t-1} to preserve stability, and initializes the update in an optimal gradient space derived from \bm{G}_{t} to boost plasticity. Method Initialization Training Objective LoRA \bm{A} LoRA \bm{B} O-LoRA ( 60 ) \bm{A}\leftarrow\mathcal{N}\left(0,\sigma^{2}\right) \bm{B}\leftarrow\mathbf{0} \mathcal{L}_{\text{cls}}(\bm{A}_{t},\bm{B}_{t})+{\color[rgb]{0,0,1}\sum_{i=0}^{t-1}||\bm{A}_{t}^{\top}\bm{A}_{i}||_{F}^{2}} InfLoRA ( 35 ) \bm{U}\bm{S}\bm{V}^{\top}=\operatorname{SVD}\left({\color[rgb]{1,0,0}\bm{H}_{t}}-{\color[rgb]{0,0,1}\bm{M}_{t-1}\bm{M}_{t-1}^{\top}\bm{H}_{t}}\right) \mathcal{L}_{\text{cls}}(\bm{B}_{t}) \bm{A}\leftarrow\bm{U}_{r} \bm{B}\leftarrow\mathbf{0} SD-LoRA ( 67 ) \bm{A}\leftarrow\mathcal{N}\left(0,\sigma^{2}\right) \bm{B}\leftarrow\mathbf{0} \mathcal{L}_{\text{cls}}({\color[rgb]{1,0,0}\left\{\alpha_{i}\right\}_{i=1}^{t-1}},{\color[rgb]{0,0,1}\alpha_{t}\overline{\bm{A}_{t}\bm{B}_{t}}}) KeepLoRA (This paper) \operatorname{SVD}({\color[rgb]{1,0,0}\bm{G}_{t}}-{\color[rgb]{0,0,1}\bm{W}_{p}\bm{W}_{p}^{\top}\bm{G}_{t}}-{\color[rgb]{0,0,1}\bm{M}_{t-1}\bm{M}_{t-1}^{\top}\bm{G}_{t}}) \mathcal{L}_{\text{cls}}(\bm{B}_{t}) \bm{A}\leftarrow\bm{U}_{r} \bm{B}\leftarrow\bm{S}_{r}\bm{V}^{\top}_{r} A straightforward solution, replaying pre-training data, is rarely viable due to prohibitive computational costs and the frequent unavailability of proprietary training corpora ( 59 ; 73 ; 51 ) . The current alternatives largely follow two paths. The first is (i) reference-data regularization , which uses reference data to anchor the model parameters and retain stability ( 66 ; 72 ) . However, the success of these approaches is highly sensitive to the choice of reference data with additional training costs ( 41 ; 72 ) . The second path involves (ii) architecture extension , such as prompt-pool ( 18 ; 63 ) or MoE-adapters ( 69 ; 14 ) while freezing the model backbone. Although effective in preventing forgetting, these modules increase inference costs ( 44 ) and complicate the deployment ( 3 ; 70 ) . Since trained weights are compact representations of data ( 9 ; 17 ) , an ideal CL method should infuse new knowledge directly into the existing parameter space, leveraging its inherent redundancy rather than accumulating external modules ( 55 ) . To identify where new knowledge can be infused without disrupting existing abilities, we analyze the parameter space of the backbone attention weights via singular value decomposition (SVD). As shown in Fig. 1(a) and 1(b) , our analysis reveals that the principal subspace, spanned by components with large singular values, predominantly encodes general knowledge, while the residual subspace, associated with small singular values, encodes domain-specific knowledge. This observation indicates that performance on specialized datasets is highly sensitive to alterations in the residual subspace, whereas general datasets remain robust to such changes. This insight forms the basis of our approach: to coherently achieve forward stability , backward stability , and plasticity , CL updates should be constrained to the residual subspace, enabling the acquisition of new domain-specific knowledge without affecting the principal subspace that encodes general knowledge. We implement our subspace-constrained updates using low-rank adaptation, a parameter-efficient method whose updates can be merged into the original weights post-training, thus incurring no inference overhead. Existing low-rank CL methods, such as O-LoRA ( 60 ) , InfLoRA ( 35 ) , and SD-LoRA ( 67 ) , lack considering the transfer ability of pre-trained model and update directions within suboptimal subspaces, which limit their plasticity and stability as shown in Tab. 1 . To overcome this, we initialize the low-rank update using the gradient from the first training step. This ensures that the update direction closely approximates the full-parameter tuning gradient to boost plasticity. To keep stability, we explicitly project this update into the residual subspace of the pre-trained weights. This constrains learning to directions that do not interfere with the model’s core transferable knowledge. Building on this, we construct a unified principal subspace that stores both the principal subspace of the model parameters and the dominant feature directions of each learned task. The unified principal subspace, with a size not exceeding the square of the feature dimension, effectively preserves both pre-trained and newly acquired knowledge. Our contributions are summarized as follows: • We empirically analyze the parameter space of pre-trained models and find that general knowledge is primarily encoded in the parameter principal subspace, while domain-specific adaptations are better captured by the residual subspace. • We propose KeepLoRA , a novel method that leverages residual subspace constraints for parameter updates, validated by theoretical analysis showing how it optimally balances plasticity and stability through orthogonal projections. • Experiments on dual-encoder (CLIP) and encoder-decoder (LLaVA) models validate that KeepLoRA effectively balances the three core challenges of plasticity, backward stability, and forward stability, establishing new state-of-the-art results on benchmark datasets. 2 Related Works In continual learning, forward stability is typically preserved using reference-data regularization and architecture extension techniques. In addition, gradient projection methods are commonly employed to address backward stability and plasticity. In this section, we review these lines of work. Reference-Data Regularization. Continual learning on

Read the complete text