Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learning

Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari · 2026

Read Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learning by Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17616.txt.

Introduction

Fatema Siddika and three co-authors present Split-on-Share for continual learning in language models. Continual learning adds new skills over time. The trap is catastrophic forgetting of old skills. Most methods treat all parameters the same. That misses a key split. Some knowledge is task-specific, and some is shared across tasks.

Their method uses a mixture of sparse experts. It splits updates along that specific-shared line. New tasks can then grow without wiping shared skill. Long-lived assistants need such care. They must learn new domains for years. The work targets the plasticity-stability dilemma head on. It offers a structure where growth and memory coexist.

Catalog record

CREATORFatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17616

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learning Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari Abstract Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of S parse E xperts for T ask A gnostic Continual Learning, referred to as SETA, a framework that resolves the plasticity-stability conflict by decomposing the model into modular subspaces. Unlike standard updates where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. This structure is maintained through elastic weight anchoring, which protects critical shared knowledge, enabling a unified gating network to automatically retrieve the correct expert combination for each task during inference. Extensive experiments across diverse domain-specific and general benchmarks demonstrate that SETA consistently outperforms state-of-the-art parameter-efficient fine-tuning-based continual learning methods. The paper code is available at Anonymous Repository 1 Introduction Large Language Models have demonstrated remarkable performance across diverse natural language processing tasks, ranging from text classification and reasoning to dialogue and summarization 3 ; 42 ; 40 . However, deploying LLMs in continual learning settings remains challenging 41 ; 7 as models must adapt to sequential tasks while preserving previously acquired knowledge 23 ; 30 . This challenge, known as catastrophic forgetting 30 ; 23 , arises from the fundamental trade-off between learning plasticity, the rapid adaptation to new tasks, and memory stability, the retention of learned knowledge. The problem is further exacerbated in parameter-efficient fine-tuning regimes, where sparsity for efficiency increases susceptibility to overwriting historical information 13 ; 8 . Existing continual learning strategies generally fall into three primary categories, including Replay based methods that rehearse historical data 6 , Regularization based techniques that penalize updates to critical parameters 23 , and Parameter Isolation approaches that physically separate task specific modules 49 . Despite their differences, these approaches share a common limitation: they treat parameters uniformly within their respective subspaces 47 . As a result, they fail to distinguish between generalizable features and task-specific requirements, leading to suboptimal trade-offs between stability and plasticity 44 . To address these limitations, Continual Sparse Fine-Tuning provides a natural mechanism for uncovering the internal knowledge structure of large language models. Unlike parameter-efficient fine-tuning (PEFT)-based low-rank adapters that project updates through opaque dense matrices, sparse tuning identifies specific attention sub-blocks, particularly in the Value projection, based on high-utility gradient signals. This unique property serves as the foundation for our proposed framework, Mixture of Sparse Experts for Task Agnostic Continual Learning, referred to as SETA . The design of SETA is driven by three fundamental research questions, each addressing a specific limitation in state-of-the-art literature: RQ1: How can sparse parameter patterns be leveraged to decouple the acquisition of novel features from the retention of historical representations? To address RQ1, SETA introduces a subspace-based expert decomposition that frames sparse continual learning within a Mixture-of-Experts paradigm. By exploiting the inherent block-wise sparsity of gradients, the model is separated into distinct expert modules, ensuring that conflicting updates operate in isolated subspaces rather than competing for a single monolithic block. RQ2: Is it possible to automatically differentiate between reusable shared knowledge and task-specific unique knowledge within a single unified framework? To answer RQ2, the Split-on-Share (SoS) mechanism combined with adaptive elastic anchoring dynamically assigns parameters based on their recurrence across tasks. Overlapping parameters are designated as shared experts and stabilized to prevent drift, while non-overlapping parameters become unique experts that are strictly frozen to act as immutable memory, ensuring historical knowledge is preserved. RQ3: How can the relevant expert modules be dynamically retrieved during inference without relying on external task identifiers? SETA resolves RQ3 by implementing adaptive gating expansion with logit invariance. This allows the router to evolve alongside the experts and dynamically select relevant parameters based solely on input tokens. This enables task-agnostic inference and supports scaling to large datasets and long task sequences without explicit task labels. The remainder of the paper is organized as follows. Section 2 formulates the problem and motivates SETA. Section 3 introduces the SETA framework and the Split-on-Share mechanism. Section 4 presents experimental results, and Section 5 concludes with key insights and future directions. 2 Problem Formulation and Motivation 2.1 Challenges of Continual Learning Continual learning aims to enable models to acquire new capabilities from a sequence of tasks \mathcal{T}=\{T_{1},\dots,T_{k}\} without forgetting previously learned knowledge 64 . For large language models, this creates a fundamental resource paradox. Full fine-tuning updates all parameters \theta , offers high plasticity, and allows the model to easily adapt to new data. However, full fine-tuning is computationally expensive and memory intensive 18 , and updating all parameters can severely disrupt the optimization landscape, leading to catastrophic forgetting 35 . Freezing the model preserves prior knowledge but eliminates plasticity, preventing the acquisition of new knowledge 41 . 2.2 PEFT in Continual Learning To address resource constraints, parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) freeze the backbone and train small modules, reducing memory overhead 20 . However, recent studies show that such approaches still face inherent limitations in continual multi-task settings. As Ren et al. 47 observed, there is a geometric disconnect between the loss minima of sequential tasks when updates are restricted to low-rank subspaces. Because PEFT limits updates to a narrow bottleneck dimension ( r\ll d ), it essentially provides a lossy approximation of the model’s weights 14 . This approximation introduces noise, hindering knowledge storage for long sequences. Under continuous updates, the optimization path is too narrow to satisfy conflicting requirements, causing a plasticity-stability failure where new learning overwrites historical parameters. Figure 1: Overview of the SETA Framework Architecture. (a–b) Sparse Subspace Selection: High-utility parameter blocks are identified from the pre-trained LLM using gradient magnitude to form the expert design. (c) Split-on-Share (SoS) Evolution: The SoS filter partitions parameters into plastic Shared ( E_{\text{s}} ) and frozen Unique ( E_{\text{u}} ) experts to resolve parameter collisions and retain knowledge. (d) Gating Evaluation: The gating network expands using logit invariance to strictly preserve decision boundaries during expert splitting. (e) Task-Agnostic Inference: A router network dynamically selects the top- k experts for input tokens, enabling automatic task processing without task identifications. 2.3 Shared Subspace Dilemma The core failure of continual learning arises from the Shared Subspace Dilemma . As tasks are learned sequentially, parameters often become important for multiple tasks. Let \mathcal{P}_{t} denote parameters for task t ; its intersection with the cumulative historical subspace \mathcal{P}_{1:t-1}=\bigcup_{k=1}^{t-1}\mathcal{P}_{k} defines a shared region \mathcal{I}=\mathcal{P}_{t}\cap\mathcal{P}_{1:t-1} . Here, updating weights risks overwriting prior knowledge, while freezing limits plasticity. Standard methods address this via uniform regularization 23 ; 4 : \mathcal{L}_{\text{reg, uniform}}=\alpha\cdot\|W-W^{*}_{1:t-1}\|^{2}, (1) where \alpha is a fixed penalty. However, uniform regularization is too rigid, treating parameters equally rather than distinguishing shared features requiring controlled plasticity from task-specific regions needing stability 50 . Since static penalties cannot simultaneously prevent semantic drift and enable new learning 9 , resolving this trade-off requires a structural solution rather than loss-based constraints alone. 2.4 Mixture of Sparse Experts for CL To address interference in

Read the complete text