Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning

Xi Wang, Cheng Deng · 2026

Read Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning by Xi Wang, Cheng Deng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08311.txt.

Introduction

Xi Wang and Cheng Deng merge continually learned models with no old data kept. They keep no old checkpoints either. Model merging can fold specialist models into one. That suits continual learning well. But merging expects diverse stored knowledge. That clashes with the storage limits of the continual setting.

The authors analyze current merging methods under those limits. They find two faults. Global alignment lets task-specific errors build up and amplify down the stream. Vanishing gradients at the start of each new task stall optimization. Their Trajectory Regularized Merging reformulates the merge phase to counter both faults. The goal is a merged model in a better state when the next task begins.

Catalog record

CREATORXi Wang, Cheng Deng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08311

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning Xi Wang, Cheng Deng Abstract Model merging provides a compelling paradigm for integrating specialized expertise into a unified multi-task model, a goal that aligns naturally with the sequential knowledge acquisition in continual learning (CL). However, the requirement for preserving diverse forms of previous knowledge conflicts with the storage limitations inherent to CL. In this paper, we systematically analyze existing model merging methods under the constraints of CL. We find that current methods prioritize global alignment, which often leads to the accumulation and amplification of task-specific errors within the continuous data stream; and the vanishing gradients at the onset of subsequent tasks frequently cause optimization to stagnate. These leave the merged model in a suboptimal state at the beginning of the next training phase. To address these challenges, we propose Trajectory Regularized Merging (TRM), a framework that reformulates the merging phase as an optimization process within an augmented trajectory subspace. Our framework integrates three synergistic objectives including task alignment, prediction consistency, and gradient responsiveness to concurrently preserve merged model’s historical stability and re-activate optimization dynamics. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple benchmarks. 1 Introduction The rapid evolution of data in the real world presents ongoing challenges for deep learning models. Even for pre-trained models (PTMs), training on dynamic data streams often biases them toward new task data, leading to progressive degradation in performance on previously learned knowledge, a phenomenon known as catastrophic forgetting ( 44 ) . Continual Learning (CL) seeks to mitigate this by navigating the stability-plasticity dilemma, ensuring the retention of historical expertise while facilitating the acquisition of new capabilities. Recent advancements in CL for PTMs have predominantly leveraged Parameter-Efficient finetuning (PEFT), employing modular components such as prompts ( 35 ; 30 ) or adapters ( 9 ) to isolate task-specific updates. More recently, model merging ( 10 ; 36 ) has emerged as a training-free paradigm for model adapting, task-specific models are integrated into a unified multi-task model by selecting or interpolating task vectors, which capture the parameter differences between finetuned and initial model across tasks. However, the most existing merging methods ( 36 ; 12 ; 10 ; 38 ) require access to all previous task vectors and the initial model during the merging phase. This operational requirement is functionally equivalent to storing the entire ensemble of prior experts, which fundamentally contradicts the memory constraints of CL that prohibit the retention of historical data or models to ensure scalability and privacy. Although methods specifically designed for CL, such as MagMax ( 23 ) and BECAME ( 19 ) , attempt to mitigate this storage overhead, they still retain accumulated task vectors or Fisher information matrices, which still presents the aforementioned problems in terms of memory. Our experiments reveal that under a strictly CL constraint, where only the current expert and the immediately merged model are available, existing methods suffer from a catastrophic performance degradation of up to 6\% . (See the supplementary material for detailed results.) These findings prompted us to further investigate whether there are more effective model merging methods for continual learning under strict data storage constraints. In alignment with the insights from ( 5 ) , performance variation of merged model is primarily influenced by the initial model in each training task. In this paper, we first design a series of experiments to examine the stability and plasticity of merged models. We reveal that existing merging methods predominantly optimize for task-agnostic global alignment while neglecting task-specific local optimality. This oversight triggers a progressive representational drift, where localized errors are compounded throughout sequential training, ultimately destabilizing the model’s retention of prior knowledge. Furthermore, the merged models frequently exhibit optimization stagnation during the early phases of new task adaptation, indicating a loss of parameter plasticity. To jointly address these challenges, we propose Trajectory Regularized Merging (TRM). Our framework reformulates the merging phase as a guided optimization problem within an augmented trajectory subspace, which is spanned solely by the task vectors of the models before and after training. And we introduce a multi objective supervisory signal composed of three synergistic constraints, task alignment for localized precision, prediction consistency for structural stability, and gradient responsiveness for kinetic re-activation. By navigating this regularized trajectory, TRM identifies an optimal merging point that harmonizes historical stability with future plasticity. This enables robust knowledge integration across dynamic data streams without any reliance on any stored models or data replay. Our main contributions can be summarized as follows: • We pointed out that existing model merging methods do not fully adhere to the principles of continual learning, and their performance degraded significantly when knowledge storage is disallowed. • We analyzed the cause of this performance gap and reformulated the merging phase as an optimal point search problem within an orthogonally augmented trajectory subspace. • We proposed an objective composed of three constraints to guide the optimize for the optimal merging point. • Our method achieves state-of-the-art performance across multiple benchmarks. 2 Related Work 2.1 Continual Learning Continual learning aims to acquire new knowledge from a never-ending data stream continuously ( 33 ; 45 ; 42 ; 32 ) . The primary challenge is learning without catastrophic forgetting: as new data arrives, the model’s performance on previously learned tasks should not degrade significantly ( 20 ; 28 ) . Traditional Regularization based methods ( 41 ; 1 ) penalize changes to important parameters for previous tasks to mitigate forgetting. Conversely, rehearsal based methods ( 29 ; 22 ; 4 ) maintain a small episodic memory of past data or employ generative models to replay synthetic samples. Recently, with the widespread adoption of pre-trained models, many continual learning methods have been developed as extensions of parameter-efficient finetuning (PEFT) methods. Prompt-based methods ( 35 ; 34 ; 30 ; 26 ; 18 ) have demonstrated the effectiveness of migrating pre-trained models into continuous data streams, adapter-based methods ( 9 ; 6 ; 31 ; 39 ; 21 ) , achieved high performance by training only a small number of parameters. Additionally, there are methods ( 14 ; 40 ) that consider the knowledge of language modality to aid in modeling learning. 2.2 Model Merging Model merging has recently gained significant attention as a practical technique for aggregating multiple models by performing linear interpolation in parameter space ( 37 ) . The core idea traces back to ensemble learning methods such as Bagging Predictors ( 3 ) , which improve generalization by averaging the outputs of diverse models. Stochastic Weight Averaging ( 11 ) aggregates gradients over training to yield wider optima and improved robustness. Subsequently, recent methods shift the focus from output space to a weight space combination. numerous methods ( 36 ; 12 ; 10 ; 38 ) have explored weight combination to achieve model merging, and MagMax ( 23 ) extends model merging to the continual learning, achieving excellent performance through the appropriate storage of model parameters. 3 Background and Motivation 3.1 Problem Formulation We consider a supervised continual learning based on pre-trained models, where a pre-trained model (PTM) f(:,\theta_{init}) parametrized by \theta_{init} is required to learn a sequence of \mathcal{T} tasks in order. Each task

Read the complete text