Sparse Adapter Fusion for Continual Learning in NLP
Min Zeng, Xi Chen, Haiqin Yang, Yike Guo · 2026
Read Sparse Adapter Fusion for Continual Learning in NLP by Min Zeng, Xi Chen, Haiqin Yang, Yike Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02502.txt.
Introduction
Min Zeng and 3 co-authors fuse small adapter modules so a language model can keep learning new tasks without forgetting old ones, in a 2026 arXiv paper. Continual learning in natural language processing fights catastrophic forgetting, but existing methods reuse parameters inefficiently, add fresh parameters for every task, and share too little between similar tasks.
Their sparse adapter fusion method works in two stages. A decision stage chooses whether to reuse an existing adapter, add an empty one, or incorporate a new one, with a search procedure biased toward reuse. A tuning stage then applies a layer-wise loss that encourages adapters to differentiate, capturing task-specific knowledge cleanly. Experiments show the method beating state-of-the-art rivals while using under sixty percent of their parameters. The design treats parameter budget as something to spend only when reuse truly fails.
Catalog record
| CREATOR | Min Zeng, Xi Chen, Haiqin Yang, Yike Guo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02502
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sparse Adapter Fusion for Continual Learning in NLP Min Zeng, Xi Chen, Haiqin Yang, Yike Guo Abstract Continual learning in natural language processing plays a crucial role in adapting to evolving data and preventing catastrophic forgetting. Despite significant progress, existing methods still face challenges, such as inefficient parameter reuse across tasks, risking catastrophic forgetting when tasks are dissimilar, and the unnecessary introduction of new parameters for each task, which hampers knowledge sharing among similar tasks. To tackle these issues, we propose a Sparse Adapter Fusion Method (SAFM) , which dynamically fuses old and new adapters to address these challenges. SAFM operates in two stages: the decision stage and the tuning stage. In the decision stage, SAFM determines whether to incorporate a new adapter, reuse an existing one, or add an empty adapter. The architecture search procedure, designed to prioritize reusing or adding empty adapters, minimizes parameter consumption and maximizes reuse. In the tuning stage, SAFM especially facilitates a layer-wise loss to encourage differentiation between adapters, effectively capturing knowledge within the same task. Experimental results consistently show that SAFM outperforms state-of-the-art (SOTA) methods, achieving comparable performance while utilizing less than 60% of the parameters 1 1 1 Code is available at https://github.com/OzymandiasChen/SAFM . . 1 Introduction Continual learning (CL) is a paradigm that emulates the human ability to learn and acquire knowledge ( 14 ; 25 ) continuously. It focuses on retaining previously learned information and transferring it effectively to master new tasks. Learning continually is essential for models to adapt rapidly to evolving tasks. As a result, continual learning methodologies have emerged, enabling models to seamlessly assimilate new information over time. However, CL faces a significant challenge: catastrophic forgetting (CF), where a model’s performance on earlier tasks deteriorates due to shifts in data distribution introduced by new tasks, potentially erasing previously acquired knowledge. Recent approaches to mitigating CF can be categorized into three main types: regularization-based, rehearsal-based, and architectural-based methods ( 10 ; 2 ) . Regularization-based methods 33 ; 10 ; 21 ; 1 maintain performance on previous tasks by constraining updates to critical parameters. However, excessive reliance on regularizers can overly restrict network parameters, limiting the model’s ability to learn new knowledge. Rehearsal-based methods ( 11 ; 20 ; 23 ; 15 ) rely on storing past samples or generating pseudo-samples from earlier tasks, facing limitations due to memory constraints or the lack of authenticity in the generated pseudo-samples. Architectural-based methods ( 13 ; 34 ) mitigate CF by adding task-specific adapters to approximate each task ( 8 ) . Notable publications, such as AdapterCL ( 13 ) , CPT4DST ( 37 ) and ACM ( 34 ) , have addressed CF through architectural modifications. However, as the number of tasks grows, the model parameters increase linearly, highlighting the importance of learning shared information to reduce parameter redundancy. While ACM aims to reduce parameters by reusing adapters from previous tasks, it still incurs high training costs as it does not reduce the adapters across tasks. To tackle these challenges, we propose a Sparse Adapter Fusion Method (SAFM) to reduce redundancy by eliminating unnecessary task-specific adapters while reusing those learned from earlier tasks. This strategy enables the model to learn both local and global information simultaneously, improving performance and parameter efficiency. In this context, local information refers to the unique data captured by adapters within the same task, while global information refers to task similarities that guide the reuse of adapters from previous tasks. SAFM operates in two stages: the decision stage and the tuning stage. In the decision stage, SAFM determines whether to add a new adapter, reuse an existing one, or add an empty adapter. The architecture search procedure prioritizes reusing or adding empty adapters, thereby minimizing parameter consumption and optimizing reuse. Once the adapter for the current task is determined, SAFM proceeds to the tuning stage, where it fine-tunes the model parameters using pseudo-replay and a layer-wise loss. Concretely, this layer-wise loss, defined as the cosine similarity between adapters in adjacent layers, maximizes information conveyance by enhancing the distinction between adapters within the same task, further improving performance. We highlight our key contributions as follows: • We propose the Sparse Adapter Fusion Method (SAFM) , a parameter-efficient continual learning method that mitigates CF by tending to reuse previously learned adapters or deploy empty adapters for new tasks. • SAFM introduces a layer-wise loss to promote the distinction adapters in adjacent layers within the same task, facilitating effective knowledge transfer and maximizing information conveyance with limited parameters. • Experimental results demonstrate the superior performance of our SAFM over SOTA methods, achieving comparable performance while utilizing less than 60% of the parameters employed by the SOTA models. 2 Related Work Continual learning aims to acquire knowledge from new tasks while maintaining proficiency in previously learned tasks ( 14 ; 25 ) . Various approaches have been proposed to mitigate the issue of catastrophic forgetting, including regularization-based, rehearsal-based, and architectural-based methods. These techniques have also been extended to apply in natural language processing, driving advancements in the field ( 2 ) . Regularization-based methods add constraints to the important parameters of previous tasks. For example, EWC ( 10 ) identifies crucial parameters and prevents substantial updates on them, thereby preserving performance on earlier tasks. ARPER ( 15 ) mitigates forgetting by combining prioritized exemplar replay with adaptive regularization inspired by EWC. Rehearsal-based methods mitigate forgetting by replaying real or pseudo-samples of previous tasks. For instance, LAMOL ( 23 ) utilizes a language model to generate pseudo-samples, thereby eliminating the need for additional memory storage to retain previous samples. DCL ( 32 ) introduces a novel generative-based rehearsal method in CL, assuming a Dirichlet distribution on the latent variables instead of the original Gaussian one applied in Conditional Variational Autoencoders. InsCL ( 27 ) dynamically replays previous data by leveraging task similarity measured via the Wasserstein Distance. It further prioritizes high-quality data using the Instruction Information metric (InsInfo), which evaluates instruction complexity and diversity. PCGR ( 5 ) proposes a Prototype Conditioned Generative Replay (PCGR) method, which enhances generative reply by incorporating task-level statistics through a Prototype Conditioned Variational Autoencoder (PCVAE). Architectural-based approaches reduce forgetting by modifying the network architecture. For example, AdapterCL ( 13 ) places a residual adapter layer ( 8 ) atop each transformer layer to approximate different tasks. More recently, CPT4DST ( 37 ) introduces prompt tuning to reduce forgetting in dialogue state tracking (DST), while ACM ( 34 ) reduces parameters by reusing modules from previous tasks. However, ACM does not reduce the number of adapter layers per task, which can lead to parameter redundancy and result in the same high training cost. SAPT ( 35 ) , on the other hand, employs a separate adapter for each new task and uses a shared attention framework to facilitate knowledge transfer between tasks. TCL ( 31 ) reduces parameters by employing Task-wrapped Adapters (TWAs) to jointly learn both global and task-specific local information across tasks. Overall, existing CL methods still face challenges related to parameter redundancy, resulting in high memory and computational costs. Therefore, it is essential to develop approaches that reduce model parameters while preserving performance across tasks. 3 Methodology 3.1 Task Definition The goal