Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models
Yan-Shuo Liang, Jia-Rui Chen, Wu-Jun Li · 2025
Read Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models by Yan-Shuo Liang, Jia-Rui Chen, Wu-Jun Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15424.txt.
Introduction
Yan-Shuo Liang, Jia-Rui Chen, and Wu-Jun Li help large language models learn new tasks in sequence without erasing old ones, in a paper published in two thousand twenty-five. Their method builds on LoRA, a popular trick that fine-tunes a model by adding small low-rank patches instead of retraining the whole network.
The usual recipe adds a fresh patch for each new task, but then old and new patches sway old tasks equally, which invites forgetting. The team's method, GainLoRA, also adds a patch per task, yet it routes them through small gating modules. The gates blend the patches while damping the new patch's effect on old tasks. That restraint protects what the model already knows. The authors report better overall performance across a run of tasks. The work targets assistants that must keep absorbing new skills over their working life.
Catalog record
| CREATOR | Yan-Shuo Liang, Jia-Rui Chen, Wu-Jun Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15424
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models Yan-Shuo Liang, Jia-Rui Chen, Wu-Jun Li Abstract Continual learning (CL), which requires the model to learn multiple tasks sequentially, is crucial for large language models (LLMs). Recently, low-rank adaptation (LoRA), one of the most representative parameter-efficient fine-tuning (PEFT) methods, has gained increasing attention in CL of LLMs. However, most existing CL methods based on LoRA typically expand a new LoRA branch to learn each new task and force the new and old LoRA branches to influence old tasks equally, potentially leading to forgetting. In this work, we propose a new method, called ga ted in tegration of lo w- r ank a daptation (GainLoRA), for CL of LLMs. GainLoRA expands a new LoRA branch for each new task and introduces gating modules to integrate the new and old LoRA branches. Furthermore, GainLoRA leverages the new gating module to minimize the influence from the new LoRA branch to old tasks, effectively mitigating forgetting and improving the model’s overall performance. Experimental results on CL benchmarks demonstrate that GainLoRA outperforms existing state-of-the-art methods. Code is available at https://github.com/liangyanshuo/gainlora . 1 Introduction Continual learning (CL), which requires the model to learn multiple tasks sequentially, is crucial for large language models (LLMs) [ 48 ] . Specifically, although existing LLMs have demonstrated strong performance for a wide range of tasks [ 4 , 8 , 53 , 54 , 73 ] with extensive pre-trained knowledge and further fine-tuning strategies, they may lose knowledge acquired from old tasks when learning multiple tasks sequentially. This phenomenon, known as catastrophic forgetting [ 36 , 41 , 59 , 61 ] , highlights the need for developing effective CL methods for LLMs. Existing CL methods can be categorized into two main categories. The first category [ 45 ] assumes that task identities are available during inference, while the second category [ 32 , 75 ] tackles a more difficult and practical setting where task identities are unavailable during inference. Recently, low-rank adaptation (LoRA) [ 21 ] , one of the most representative parameter-efficient fine-tuning (PEFT) methods, has gained increasing attention in the CL of LLMs [ 3 , 61 ] . Specifically, by reparameterizing pre-trained weights in a low-rank form, LoRA updates only a limited number of parameters to adapt LLMs to a downstream task, making the fine-tuning process much more efficient than updating all parameters of LLMs [ 17 ] . This efficiency also benefits CL, making LoRA increasingly popular in CL of LLMs. Most existing CL methods based on LoRA [ 32 , 75 ] typically expand a new LoRA branch for learning each new task while freezing all old LoRA branches. In this way, they avoid forgetting caused by directly updating the LoRA parameters of old tasks. However, to handle the practical CL scenario where task identities are unavailable at inference time, existing methods [ 32 , 50 , 61 ] based on LoRA integrate new and old LoRA branches through a simple addition. Consequently, they force the new and old LoRA branches to influence old tasks equally, which means that the new LoRA branch may cause a relatively large change in the model’s output on old tasks. This leads to forgetting and degrades the model’s overall performance in CL. In this work, we propose a new method, called ga ted in tegration of lo w- r ank a daptation (GainLoRA), for CL of LLMs. The contributions of GainLoRA are listed as follows: • GainLoRA expands a new LoRA branch to learn each new task and introduces gating modules to integrate the new and old LoRA branches. • GainLoRA leverages the new gating module to minimize the influence from the new LoRA branch to old tasks, effectively mitigating forgetting and improving the model’s overall performance. • Experimental results on CL benchmarks show that GainLoRA outperforms existing state-of-the-art CL methods. 2 Related Work and Preliminaries 2.1 Related Work Parameter-Efficient Fine-Tuning Parameter-efficient fine-tuning (PEFT) methods tune a limited number of parameters to adapt a pre-trained model for downstream tasks, showing much higher efficiency than tuning all the parameters of the pre-trained model, especially for LLMs [ 72 ] . For example, Adapter [ 20 ] modifies the model architecture by introducing trainable modules into Transformer layers and tunes these modules for downstream tasks. Prompt-tuning [ 26 ] and Prefix-tuning [ 27 ] insert learnable tokens into the input and tune them for downstream tasks. Low-rank adaptation (LoRA) [ 21 ] reparameterizes the original model parameters with low-rank matrices and tunes these matrices for downstream tasks. Although tuning significantly fewer parameters than full fine-tuning, PEFT can achieve comparable performance to full fine-tuning across a wide range of natural language processing (NLP) tasks [ 16 , 21 , 37 , 70 ] . Continual Learning There are three main types of CL methods, categorized as regularization-based methods, memory-based methods, and expansion-based methods. Regularization-based methods [ 23 , 24 , 29 ] incorporate a regularization term to mitigate catastrophic forgetting. Memory-based methods [ 10 , 31 , 34 , 52 , 75 ] utilize memory mechanisms to preserve knowledge from old tasks. Expansion-based methods [ 22 , 28 , 30 , 46 ] mitigate catastrophic forgetting by introducing new parameters for learning new tasks while typically freezing old parameters. Many CL methods [ 1 , 29 , 30 ] are designed to train models from scratch. Recent studies [ 32 , 45 , 56 , 61 , 67 ] have shown that leveraging PEFT strategies to fine-tune pre-trained models enables CL methods to achieve superior performance across tasks. For example, some methods [ 42 , 45 , 67 , 75 ] use prompt-tuning for continual learning. They either maintain independent prompts for each task or maintain a prompt pool and perform query-key matching to learn new tasks. There are also many methods [ 32 , 49 , 61 , 75 ] adopting LoRA for continual learning. Most of these methods expand a new LoRA branch to handle each new task while freezing old LoRA branches to mitigate catastrophic forgetting. However, they force the new and old LoRA branches to influence old tasks equally, potentially leading to forgetting. 2.2 Preliminaries Problem Definition We follow existing CL works [ 61 , 75 ] to formalize the problem definition for CL of LLMs. Specifically, in CL, a sequence of tasks \{\mathcal{T}_{1},\mathcal{T}_{2},...,\mathcal{T}_{T}\} is presented to the model sequentially, where T denotes the total number of tasks. The t -th task \mathcal{T}_{t} consists of a training dataset \mathcal{D}_{t} . For any given sample (\bm{x}_{t},\bm{y}_{t})\in\mathcal{D}_{t} , \bm{x}_{t} denotes an input sentence and \bm{y}_{t} denotes the corresponding output. When learning the t -th new task, the model is required to mitigate catastrophic forgetting of the t-1 previously learned tasks. Similar to existing CL works for LLMs [ 3 , 75 ] , we consider a more challenging CL setting with three key challenges: (1) the model is presented with a sequence of tasks spanning various types, such as dialogue generation, information extraction and so on; (2) the model is not provided with task identities at inference time; (3) the model must learn without access to real or synthetic samples from previously learned tasks. Low-Rank Adaptation LoRA