Merge before Forget: A Single LoRA Continual Learning via Continual Merging

Fuli Qiao Mehrdad Mahdavi The Pennsylvania State University {fvq5015,mzm616}@psu.edu · 2025

Read Merge before Forget: A Single LoRA Continual Learning via Continual Merging by Fuli Qiao Mehrdad Mahdavi The Pennsylvania State University {fvq5015,mzm616}@psu.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23017.txt.

Introduction

Fuli Qiao and Mehrdad Mahdavi tackle continual learning for large language models, where a model must pick up new tasks without forgetting old ones and without the memory bill growing forever. Popular low-rank adapters, called LoRA, are usually frozen and stockpiled per task in current methods, so storage grows with every task and old adapters can interfere with new learning.

Their alternative keeps exactly one LoRA module. Each new task starts from an orthogonal basis extracted from what was learned before, which limits interference, and its update is then merged into the single shared adapter. A time-aware scaling balances old and new knowledge during merging, exploiting an asymmetry in how LoRA components behave. Memory stays constant as tasks accumulate. Theory backs the design, and experiments across continual learning benchmarks with several Llama models show strong performance with better efficiency than stockpiling approaches.

Catalog record

CREATORFuli Qiao Mehrdad Mahdavi The Pennsylvania State University {fvq5015,mzm616}@psu.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23017

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Merge before Forget: A Single LoRA Continual Learning via Continual Merging Fuli Qiao Mehrdad Mahdavi The Pennsylvania State University {fvq5015,mzm616}@psu.edu Abstract Parameter-efficient continual learning has emerged as a promising approach for large language models (LLMs) to mitigate catastrophic forgetting while enabling adaptation to new tasks. Current Low-Rank Adaptation (LoRA) continual learning techniques often retain and freeze previously learned LoRAs or generate data representations to overcome forgetting, typically utilizing these to support new LoRAs learn new tasks. However, these methods not only ignore growing computational memory with tasks and limited storage space but also suffer from potential task interference due to the lack of effective LoRA merging mechanisms. In this paper, we propose a novel continual learning method that orthogonally initializes and sequentially merges LoRAs updates into a single unified LoRA. Our method leverages orthogonal basis extraction from previously learned LoRA to initialize the learning of new tasks, further exploits the intrinsic asymmetry property of LoRA components by using a time-aware scaling mechanism to balance new and old knowledge during continual merging. Our approach maintains constant memory complexity with respect to the number of tasks, minimizes interference between past and new tasks via orthogonal basis initialization, and improves performance over asymmetric LoRA merging via adaptive scaling. We provide theoretical analysis to justify our design and conduct extensive experiments across diverse continual learning benchmarks using various Llama models, demonstrating the effectiveness and efficiency of our method. 1 Introduction Large Language Models (LLMs) ( 25 ; 1 ; 35 ) have been growing as the cornerstone of modern machine learning, achieving remarkable performance across a wide range of downstream tasks. However, despite their impressive capabilities, LLMs still suffer from catastrophic forgetting ( 23 ; 51 ; 44 ) when fine-tuning sequential tasks, and their huge model capacity makes full fine-tuning computationally expensive and memory-intensive ( 48 ) . These challenges have led to increasing attention in parameter-efficient continual learning , particularly via techniques such as LoRA (Low-Rank Adaptation) ( 9 ) , that injects trainable low-rank matrices \bm{A} and \bm{B} into pre-trained models, enabling task adaptation with minimal additional parameters. Recent studies have made progress in addressing catastrophic forgetting in LLMs through LoRA-based continual learning. For example, O-LoRA ( 36 ) freezes previously learned LoRAs and incrementally learns new tasks in their orthogonal subspace; InfLoRA ( 16 ) preserves prior LoRAs and uses task-dependent input matrices to define orthogonal subspaces for initializing new ones; SAPT-LoRA ( 49 ) retains earlier LoRAs and leverages generated previous tasks’ data to align new LoRA learning with shared modules; SD-LoRA ( 41 ) incrementally decouples the learning of magnitude and direction in LoRA components while preserving directions learned from previous tasks. However, these methods either keep and freeze previously learned LoRAs, resulting in parameter growth of the form [\bm{B}_{1}\bm{A}_{1},\dots,\bm{B}_{t}\bm{A}_{t}] , or generate and maintain task-specific data representations, leading to (i) linear growth in memory usage with the number of tasks, (ii) limited scalability due to constrained storage space, and (iii) potential task interference in the absence of principled LoRA merging mechanisms. These limitations motivate the question: Can we enable continual learning only using a single shared LoRA, without learning or storing task-specific LoRAs or data representations? To address this question, we take inspiration from model merging ( 7 ; 3 ; 40 ) , an emerging paradigm that aims to combine multiple task-specific models into a single unified model without retraining ( 30 ; 11 ; 42 ; 24 ) . By extending this idea, we frame continual learning as a sequential model merging problem, where the objective shifts from keeping all task-specific LoRAs or data to continually integrating their updates into a single shared LoRA as new tasks arrive. Recently, model merging has successfully been extended to the LoRA regime: KnOTS ( 31 ) leverages singular value decomposition to project LoRA updates into a shared latent space, where existing merging methods can be applied; LoRA-LEGO ( 50 ) decomposes LoRAs into minimal semantic units via grouping and clustering, enabling a reconstruction of multiple LoRAs into one. However, these LoRA merging methods generally assume concurrent access to all task-specific LoRAs fine-tuned from the same pre-trained model, which limits their applicability to the continual merging scenarios ( 4 ) , where tasks arrive sequentially. In such settings, the order of merging becomes critical and may degrade the performance of the final model. Moreover, continual LoRA merging remains underexplored in existing literature. While in the full-model setting, continual merging has received more attention, e.g., OPCM ( 34 ) sequentially projects new model updates onto subspaces orthogonal to the previously merged model and uses adaptive scaling to mitigate interference. However, these methods are not designed for LoRA, and the objective of merging differs from that of continual learning ( 24 ) . These challenges lead to the following question we aim to answer: How can we enable continual learning through LoRA-based continual merging? (a) LoRA \bm{A} (q) (b) LoRA \bm{B} (q) (c) LoRA \bm{A} (v) (d) LoRA \bm{B} (v) Figure 1 : Cosine similarity between 15 tasks from the large number of tasks benchmark for fine-tuned q and v attention LoRA \bm{A} and \bm{B} in the last layer (32nd) of Llama-2-7B-chat. We answer this question by maintaining a single pair of low-rank matrices \{\bm{A},\bm{B}\} , shared across tasks. Achieving this necessitates addressing key challenges, including how to initialize and continually update the shared LoRA to effectively balance the trade-off between forgetting and generalization. Moreover, in contrast to full-model continual merging, \bm{A} and \bm{B} play different roles in continual merging with LoRA. For instance, prior works ( 52 ; 32 ; 47 ; 14 ) have shown that in LoRA fine-tuning, training \bm{B} (initialized to zero) is critical for the performance, even randomly initialized \bm{A} often suffices, but reversing the roles of \bm{A} and \bm{B} substantially decreases performance. To further investigate the asymmetry of LoRA components, we separately fine-tune 15 tasks from a standard large number of tasks benchmark ( 36 ) in continual learning using 15 independent LoRAs on Llama-2-7B-chat ( 35 ) , and compute cosine similarity of \bm{A} and \bm{B} across 15 tasks using their last layer LoRA. Figure 1 shows that \bm{A} exhibits significantly higher similarity across tasks compared to \bm{B} , suggesting that LoRA components follow inherently different learning dynamics. This motivates us to treat \bm{A} and \bm{B} differently in continual merging. Figure 2 : Comparison of SLAO and O-LoRA memory usage of large number of tasks benchmark via Llama-2-7B-chat. To address the above questions, we propose a novel parameter-efficient continual learning method via continual merging into a single LoRA, which initializes new task learning in an orthogonal subspace and sequentially merges LoRA updates. We name this method SLAO ( S ingle L oR A continual learning with O rthogonal initialization via continual merging). Specifically, SLAO initializes each new task learning LoRA using orthogonal basis extracted from previously learned LoRA components, and exploits the asymmetric roles of \bm{A} and \bm{B} by applying a time-aware scaling mechanism that balances knowledge retention and plasticity during continual merging. As shown in Figure 2 , our approach ensures constant memory overhead regardless of the number of tasks. Additionally, it reduces interference between past and new tasks via orthogonal basis initialization

Read the complete text