CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning

Yang Liu, Toan Nguyen, Flora D. Salim · 2026

Read CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning by Yang Liu, Toan Nguyen, Flora D. Salim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20247.txt.

Introduction

Yang Liu, Toan Nguyen and Flora Salim present CP-MoE, a transient-expert framework. LoRA mixture-of-experts methods either isolate experts or let updates overwrite parameters, causing forgetting. CP-MoE's transient expert captures early updates and guides merging into experts. A routing bias uses the transient expert to gauge expert similarity and steer routing, while guided regularization shields parameters, cutting interference and forgetting while keeping cross-task transfer. On SuperNI, CP-MoE tops rivals with stronger zero-shot transfer; on VQA v2, it cuts forgetting and beats baselines.

Catalog record

CREATORYang Liu, Toan Nguyen, Flora D. Salim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56327
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20247

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning Yang Liu, Toan Nguyen, Flora D. Salim Abstract Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision–language models (VLMs). Although Mixture-of-Experts (MoE) architectures offer an efficient path to scaling, existing LoRA-based MoE continual learning methods still face a fundamental trade-off: they either isolate experts too aggressively, limiting knowledge transfer across tasks, or allow task-specific updates to overwrite important existing parameters, leading to severe forgetting. To address this, we propose CP-MoE , a continual learning framework built around a transient expert that captures early task-specific updates and guides their integration into stable experts. CP-MoE introduces a consistency-preserving routing bias, which uses the transient expert to estimate representation similarity with stable experts and steer routing towards more compatible expert selection, and a transient expert-guided regularisation mechanism, which selectively protects important historical parameters during merging. Together, these components reduce parameter interference and forgetting while preserving cross-task knowledge transfer. We validate CP-MoE on both unimodal and multimodal continual learning benchmarks with LLM-based and VLM-based MoE models. On SuperNI benchmark, spanning diverse sequential language tasks, CP-MoE achieves state-of-the-art performance and stronger zero-shot transfer to unseen tasks. On VQA v2 dataset, it scales effectively to multimodal visual reasoning, consistently reduces forgetting, and outperforms strong MoE baselines. 1 Introduction Large language models (LLMs) and vision–language models (VLMs) have demonstrated strong performance across a wide range of tasks ( 3 ; 22 ; 30 ) . Yet, unlike static benchmark settings, real-world deployment requires these models to continually adapt to non-stationary data streams, where tasks and data distributions evolve over time. This setting, known as continual learning (CL), requires a model to accumulate knowledge across a sequence of tasks without degrading previously acquired capabilities. Despite substantial progress, catastrophic forgetting remains a central challenge ( 25 ; 38 ) , as updates for new tasks can significantly overwrite earlier representations. To adapt large foundation models to new tasks efficiently, recent work has increasingly integrated sparse architectures, particularly Mixture-of-Experts (MoE) ( 29 ) , into parameter-efficient fine-tuning (PEFT) paradigms. In these adapted models, the backbone is typically frozen to preserve core pre-trained knowledge, while adaptation is handled by a small set of trainable expert modules. The appeal of incorporating MoE lies in its partitioned parameter space, which enables task-specific computation paths and provides a natural way to reduce interference across tasks ( 2 ) . By utilizing parameter-efficient methods such as LoRA ( 15 ) as the experts, this gives rise to LoRA-MoE frameworks ( 10 ) , which have emerged as a practical and scalable approach to continual adaptation. However, despite their structural sparsity and efficiency, such MoE-based adaptation methods still remain vulnerable to catastrophic forgetting under sequential task training. Despite their recent success, continual LoRA-MoE frameworks still face key limitations. Existing methods often prevent forgetting by isolating experts too aggressively ( 20 ) , typically through a learn-and-freeze strategy that adds new LoRA experts for each task and then keeps them fixed. As a result, the model behaves as a static ensemble with limited cross-task knowledge transfer, leading to weaker generalisation on unseen tasks and poor scalability as the task sequence grows. Second, the load-balancing constraint commonly used in MoE architectures for hardware efficiency can be problematic in continual learning, as it tends to enforce unnecessary uniformity across experts and can limit task-specific specialisation. This calls for routing constraints that account for the semantic alignment between task representations and experts. Finally, most current MoE-based continual learning methods lack mechanisms to protect important parameters during sequential updates, making them vulnerable to forgetting ( 16 ) . To address these challenges, we propose CP-MoE , a continual learning framework built around a transient expert that acts as a lightweight probe before updates are integrated into the stable expert pool. For each incoming task, the transient expert is briefly adapted on a small warm-up subset to capture early task-specific updates and assess its representation compatibility with existing experts. This design offers three key advantages. First, the transient expert provides a local look-ahead signal of task-induced parameter displacement, enabling more selective integration without dynamic expert expansion. Second, it supports a consistency-preserving routing bias that favours experts with more compatible representations, mitigating the excessive uniformity imposed by standard load-balancing constraints and promoting stronger expert specialisation. Third, it enables a representation-guided regularisation mechanism that selectively protects important historical parameters during integration, thereby reducing interference and forgetting. Together, these components allow CP-MoE to preserve prior knowledge while maintaining effective knowledge transfer across related tasks. We show that this idea is highly effective in practice. Although CP-MoE introduces only a transient expert for each task, this module is used only during the short warm-up stage and is not retained thereafter, allowing the framework to improve performance without incurring persistent computational overhead. We evaluate CP-MoE on both unimodal and multimodal continual learning benchmarks with LLM-based and VLM-based MoE models, where it achieves state-of-the-art performance and strong zero-shot generalisation to out-of-distribution tasks. Our contributions are summarised as follows: • Transient expert for selective consolidation. We introduce a transient expert that briefly adapts to each incoming task and serves as a lightweight probe of early task-specific updates and representation compatibility, enabling selective integration into a fixed pool of stable experts without dynamic expansion. • Consistency-preserving routing bias. We propose a routing mechanism that favours experts with more compatible representations, improving routing stability and promoting stronger expert specialisation beyond standard load balancing. • Representation-guided regularisation. We develop a dynamic regularisation scheme that selectively protects important historical parameters during integration, thereby reducing interference and forgetting. • Strong empirical performance. We validate CP-MoE on SuperNI and VQA v2 with LLM-based and VLM-based methods. CP-MoE achieves state-of-the-art performance, stronger zero-shot transfer on unseen tasks, and consistently lower forgetting than strong baselines. 2 Related Work Continual Learning. Continual learning (CL) methods for mitigating catastrophic forgetting are commonly grouped into three paradigms: regularisation-based methods, which constrain updates to parameters deemed important for previous tasks ( 17 ; 41 ; 27 ) ; memory-based methods, which replay stored samples to preserve past decision boundaries ( 24 ; 5 ; 26 ; 4 ) ; and expansion-based methods, which allocate task-specific parameters or network branches as new tasks arrive ( 28 ; 31 ) . While effective for smaller models, these approaches scale poorly to large language models (LLMs) and vision–language models (VLMs), where operating over the full parameter space is often prohibitively expensive in memory and computation. Moreover, many existing CL methods were developed primarily for image classification and do not transfer naturally to large-scale multimodal settings. Parameter-Efficient Fine-Tuning in Continual Learning. Recent work has shown that parameter-efficient fine-tuning (PEFT) can improve continual learning performance by adapting large models with a small number of trainable parameters ( 37 ; 36 ; 15 ) . In particular, many continual learning methods adopt LoRA and allocate a new branch for each task while freezing previous branches to reduce forgetting ( 21 ; 34 ; 43 ) . However, this design often results in a collection of isolated task-specific modules, limiting cross-task knowledge transfer and reducing adaptability. Recent approaches improve forgetting by combining parameter freezing with external gating to select transfer branches ( 20 ) .

Read the complete text