Mingle : Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging
Zihuan Qiu, Yi Xu, Chiyuan He, Fanman Meng, Linfeng Xu, Qingbo Wu, Hongliang Li · 2025
Read Mingle : Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging by Zihuan Qiu, Yi Xu, Chiyuan He, Fanman Meng, Linfeng Xu, Qingbo Wu, Hongliang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11883.txt.
Introduction
Zihuan Qiu and six co-authors set out MINGLE, a frame for joining models that were each fine-tuned for a task, one by one, at the time of use. Their paper is from two thousand twenty-five.
Such joins must work with no sight of the first training data. Two flaws block them: the tuned parts clash and cause loss of past skill, and the test data can drift. The team names this test-time task and meets it with a mix of small expert parts, each cheap to store. A gate built on null spaces curbs the clash. A small set of unlabeled test cases guides the join as it runs.
Catalog record
| CREATOR | Zihuan Qiu, Yi Xu, Chiyuan He, Fanman Meng, Linfeng Xu, Qingbo Wu, Hongliang Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 107 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11883
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mingle : Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging Zihuan Qiu, Yi Xu, Chiyuan He, Fanman Meng, Linfeng Xu, Qingbo Wu, Hongliang Li Abstract Continual model merging integrates independently fine-tuned models sequentially without access to the original training data, offering a scalable and efficient solution for continual learning. However, existing methods face two critical challenges: parameter interference among tasks, which leads to catastrophic forgetting, and limited adaptability to evolving test distributions. To address these issues, we introduce the task of Test-Time Continual Model Merging (TTCMM), which leverages a small set of unlabeled test samples during inference to alleviate parameter conflicts and handle distribution shifts. We propose Mingle , a novel framework for TTCMM. Mingle employs a mixture-of-experts architecture with parameter-efficient, low-rank experts, which enhances adaptability to evolving test distributions while dynamically merging models to mitigate conflicts. To further reduce forgetting, we propose Null-Space Constrained Gating, which restricts gating updates to subspaces orthogonal to prior task representations, thereby suppressing activations on old tasks and preserving past knowledge. We further introduce an Adaptive Relaxation Strategy that adjusts constraint strength dynamically based on interference signals observed during test-time adaptation, striking a balance between stability and adaptability. Extensive experiments on standard continual merging benchmarks demonstrate that Mingle achieves robust generalization, significantly reduces forgetting, and consistently surpasses previous state-of-the-art methods by 7–9% on average across diverse task orders. Our code is available at: https://github.com/zihuanqiu/MINGLE 1 Introduction Continual learning aims to incrementally adapt machine learning models to new tasks without forgetting previously learned knowledge, addressing the critical challenge of catastrophic forgetting [ 43 ] . However, conventional continual learning approaches typically require continuous access to original training data, raising significant concerns about privacy and substantial computational overhead due to retraining efforts, thus limiting their applicability in dynamic, data-sensitive environments. To address these limitations, recent works have explored an alternative paradigm known as continual model merging (CMM), which sequentially integrates independently fine-tuned models directly in parameter space, without revisiting any training data [ 38 , 51 , 70 ] . CMM typically operates under a "merge-to-transfer" paradigm: given a pretrained model \theta_{0} and independently fine-tuned models \{\theta_{t}\}_{t=1}^{T} , a unified model is constructed sequentially by combining task-specific weight updates \Delta\theta_{t}=\theta_{t}-\theta_{0} via weighted averaging or projection-based strategies [ 24 , 81 , 69 ] . Despite its advantages in scalability, data privacy, and distributed training capabilities [ 44 , 14 , 57 ] , existing CMM methods still encounter critical issues, notably severe parameter interference between tasks and limited adaptability to evolving test distributions. This parameter interference arises because, as fine-tuned models are incrementally merged, overlapping or conflicting parameter updates accumulate, resulting in severe forgetting of previously learned tasks. To mitigate this interference, recent methods introduce structural constraints such as orthogonal projection [ 70 , 78 ] , model linearization [ 38 , 69 ] , and pruning-based sparsification [ 86 , 90 ] . However, their effectiveness diminishes as task count grows and interference becomes increasingly entangled. Moreover, models merged across tasks often fail to generalize effectively, particularly when facing unseen or shifting task conditions. These flaws result in severe forgetting of earlier tasks and substantial performance gaps compared to the upper bound achieved by individually fine-tuned models. As shown in Fig. 1 , TA [ 24 ] suffers from large performance gaps and strong forgetting, reflected by low accuracy and negative backward transfer. OPCM [ 70 ] improves over TA via orthogonalized merging but still shows notable degradation. Figure 1 : After 8-task continual merging: accuracy on first four tasks and overall BWT. To overcome these limitations, we propose a novel continual merging paradigm— Test-Time Continual Model Merging (TTCMM) —which explicitly introduces the concept of test-time adaptation (TTA) [ 75 , 63 ] into model merging. Unlike prior TTA-based multi-task merging methods [ 87 , 68 , 88 ] , which assume simultaneous availability of models and test data from all tasks, TTCMM utilizes only a small set of unlabeled samples from the current task, making it uniquely suited for realistic continual scenarios where revisiting historical data is often infeasible. In this paper, we propose Mingle ( MI xture of N ull-Space G ated L ow-Rank E xperts), a method designed to continually merge independently fine-tuned models at test-time while preserving prior knowledge. Mingle employs a mixture-of-experts architecture [ 27 , 45 ] composed of lightweight LoRA-based [ 21 ] experts, enabling efficient and flexible test-time adaptation. To robustly prevent interference from previously learned tasks, we introduce a novel Null-Space Constrained Gating mechanism, restricting gating updates to task-orthogonal subspaces. Additionally, we propose an Adaptive Relaxation Strategy to dynamically modulate constraint strength based on test-time interference feedback during adaptation. Extensive experiments on standard continual learning benchmarks show that Mingle consistently outperforms previous state-of-the-art approaches by 7–9% on average, achieving robust generalization and strong resistance to catastrophic forgetting across diverse continual learning scenarios. Remarkably, these improvements are achieved entirely without any access to original training data, demonstrating the effectiveness of our TTCMM paradigm and the power of test-time adaptation in continual learning. Our contributions are summarized as follows: • We formalize test-time continual model merging (TTCMM), a novel task that leverages unlabeled test samples to merge independently fine-tuned models. • We propose Mingle , a TTCMM framework with Adaptive Null-Space Constrained Gating to effectively balance stability and plasticity. • Extensive experiments show that Mingle achieves state-of-the-art performance, consistently outperforming prior methods in accuracy, robustness and resistance to forgetting. 2 Related Work Continual Learning. Continual learning (CL) seeks to mitigate catastrophic forgetting [ 43 ] , where learning new tasks overwrites prior knowledge. Regularization-based methods constrain updates with importance weights [ 31 , 92 , 2 , 30 , 82 ] , while distillation aligns outputs to preserve knowledge [ 20 , 12 , 60 , 52 ] . Replay methods store exemplars or generate surrogates with prompts, prototypes, or generators [ 55 , 39 , 79 , 61 , 53 ] , and dynamic architectures expand capacity via growth or ensembling [ 35 , 98 , 42 ] . Recent work leverages lightweight adapters or prompts in pre-trained models for efficient transfer [ 89 , 23 , 80 ] . Model merging offers an alternative route. Some methods remain close to conventional CL by sequentially fine-tuning and merging models to reduce forgetting [ 41 , 42 , 16 ] , typically requiring training data. In contrast, continual model merging [ 28 , 38 , 5 , 51 , 70 ] merges independently fine-tuned models without revisiting training data, enabling greater scalability and privacy. Model Merging. Early work merged models via direct parameter averaging [ 72 , 58 ] , later refined by linear mode connectivity [ 13 , 1 ] . Wortsman et al. [ 81 ] showed that weight averaging can also enhance robustness and out-of-distribution generalization. Task Arithmetic (TA) [ 24 ] views models as task vectors to be summed, but relies on weight disentanglement [ 49 ] , often violated under standard fine-tuning, motivating structured training [ 28 , 65 ] . Beyond averaging, interference-aware methods reweight or sparsify parameters [ 86 , 90 ] , or fuse models via distillation and clustering [ 88 , 76 ] . LoRA-based tuning