MAGIC: Achieving Superior Model Merging via Magnitude Calibration
Yayuan Li, Jian Zhang, Jintao Guo, Zihan Cheng, Lei Qi, Yinghuan Shi, Yang Gao · 2025
Read MAGIC: Achieving Superior Model Merging via Magnitude Calibration by Yayuan Li, Jian Zhang, Jintao Guo, Zihan Cheng, Lei Qi, Yinghuan Shi, Yang Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19320.txt.
Introduction
Yayuan Li, Jian Zhang, Jintao Guo, and Zihan Cheng fix a blind spot in model merging, the craft of fusing specialist models into one. Merging usually aligns feature directions and calls the job done. Features, though, also carry magnitude, and common merge steps quietly distort it, dragging performance down.
Their plug-in framework, MAGIC, recalibrates magnitudes layer by layer in both feature and weight space. Three variants cover different needs, and no real retraining is required. By restoring the scale that fusion perturbs, the merged model keeps more of each specialist's behavior. The work argues magnitude deserves the same care that direction has long received.
Catalog record
| CREATOR | Yayuan Li, Jian Zhang, Jintao Guo, Zihan Cheng, Lei Qi, Yinghuan Shi, Yang Gao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-19320
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MAGIC: Achieving Superior Model Merging via Magnitude Calibration Yayuan Li, Jian Zhang, Jintao Guo, Zihan Cheng, Lei Qi, Yinghuan Shi, Yang Gao Abstract The proliferation of pre-trained models has given rise to a wide array of specialised, fine-tuned models. Model merging aims to merge the distinct capabilities of these specialised models into a unified model, requiring minimal or even no additional training. A core objective of model merging is to ensure the merged model retains the behavioural characteristics of the specialised models, typically achieved through feature alignment. We identify that features consist of two critical components: direction and magnitude. Prior research has predominantly focused on directional alignment, while the influence of magnitude remains largely neglected, despite its pronounced vulnerability to perturbations introduced by common merging operations ( e.g. , parameter fusion and sparsification). Such perturbations to magnitude inevitably lead to feature deviations in the merged model from the specialised models, resulting in subsequent performance degradation. To address this, we propose MAGnItude Calibration (MAGIC), a plug-and-play framework that rectifies layer-wise magnitudes in feature and weight spaces, with three variants. Specifically, our Feature Space Calibration (FSC) realigns the merged model’s features using a small set of unlabelled data, while Weight Space Calibration (WSC) extends this calibration to the weight space without requiring additional data. Combining these yields Dual Space Calibration (DSC). Comprehensive experiments demonstrate that MAGIC consistently boosts performance across diverse Computer Vision tasks (+4.3% on eight datasets) and NLP tasks (+8.0% on Llama) without additional training. Our code is available at: https://github.com/lyymuwu/MAGIC I Introduction As deep learning flourishes, abundant fine-tuned models have been open-sourced on the Internet ( e.g. , HuggingFace and Civitai) over the past few years [ 15 , 68 , 71 , 64 ] . Since models specialised in different tasks may come from the same pre-trained model ( e.g. , Llama [ 60 ] or CLIP [ 47 ] ), an interesting question arises whether there is a way to directly merge task-specific knowledge into a single model without introducing extra parameters and retraining costs. Therefore, we can combine different styles of a generation model into a mixed style [ 40 , 53 ] or merge specialised models into a single model with multi-task processing ability [ 21 , 43 ] . Fig. 1: Current merging methods seek to approximate specialised models, whose features serve as the optimal reference. While features consist of both direction and magnitude, existing approaches overlook magnitude alignment. Our calibration method improves performance consistently by aligning feature magnitudes more closely with the optimal. Fig. 2: An illustration of task vectors based model merging methods. (a) Definition of task vectors, where \theta_{\text{pre}} is the parameter of a pre-trained model, and \theta_{k} is the model’s weight fine-tuned on the k -th task. (b) Task Arithmetic [ 21 ] merges models via the arithmetic mean of task vectors. (c) Layer-wise AdaMerging [ 70 ] uses unlabelled data to learn different merge coefficients \lambda_{k}^{l} for the task vector of each task k and layer l . (d) Our method applies post-calibration after model merging, using adaptive estimation calibration coefficients \xi^{l} to adjust both the task vector \Delta\theta_{\mathrm{merge}}^{l} and task feature \Delta h_{\mathrm{merge}}^{l}(x_{k}) at each layer. Fortunately, model merging [ 1 , 66 ] serves as a promising method for integrating the weights of task-specialised models into a single model. A prior work, Weight Averaging [ 22 ] , achieves this by simply averaging their weights. In contrast, Task Arithmetic (TA) [ 21 ] introduces task vectors , defined as the difference between fine-tuned and pre-trained weights, to isolate task-specific knowledge. This formulation not only provides an easier way to manipulate the contribution of each specialised model, but also discovers the inherent problem of model merging, where the conflicts of specialised models’ task vectors significantly impair the merged model performance. Subsequent works [ 67 , 12 , 13 , 72 ] try to restore the specialised model performance of merged models by aligning their intermediate feature directions with those of the specialised models. However, they overlook another equally critical aspect: feature magnitude. Through theoretical and experimental analysis, we observe that when only aligning the directions, various merging operations ( e.g. , parameter fusion or sparsification), inevitably make the feature magnitudes of the merged model deviate from those of the specialised model with significant performance degradation. As shown in Fig. 1 , features from an intermediate layer of the merged model are projected onto two dimensions. A recent method, e.g. , Iso-C [ 36 ] yields a smaller angle with the optimal feature ( i.e. , feature of the specialised model) compared to the conventional method, e.g. , TA [ 21 ] , thereby achieving stronger directional alignment. However, features from the merged model generally exhibit significant “Magnitude Deviation” from the optimal feature and fail to achieve perfect alignment with the optimal. This insight motivates our proposed plug-and-play MAGnItude Calibration (MAGIC) framework. Specifically, we develop MAGIC in both feature and weight space with three variants. Our Feature Space Calibration (FSC) method preserves feature direction while rescaling the deviated magnitude of the features induced by the task vector in the merged model to match those of the specialised models. This enables the merged model to achieve performance comparable to the specialised models. Since FSC requires additional input data to calculate features and the magnitude variation between weights and features exhibits a linear correlation, the method is naturally extended to data-free Weight Space Calibration (WSC), which rescales the task vector’s magnitude. Finally, Dual Space Calibration (DSC) is introduced to rescale in both spaces. Furthermore, it is observed that different layers exhibit varying degrees of sensitivity to rescaling-based calibration. Certain layers that reside in sharp minima of the loss landscape are extremely sensitive to naive rescaling, particularly to scaling up, which leads to considerable performance degradation. These extremely sensitive layers are referred to as “Magnitude-Sensitive Layers” . To enhance the robustness of calibration, additional constraints are incorporated to suppress these magnitude-sensitive layers, thereby improving merging performance. The main research contributions are as follows: • Theoretical and empirical analyses expose severe magnitude deviation in merged models, clarify its impact on performance, and motivate effective mitigation strategies. • A simple yet efficient post-calibration framework, MAGIC, is proposed with three variants in weight space (WSC), feature space (FSC) and dual space (DSC) to address magnitude deviation from the specialised models. • Our proposed training-free and plug-and-play post-merging strategies demonstrate significant performance improvements across diverse scenarios ( e.g. , 8% improvement on Llama-2 for SOTA methods). II Revisit Model Merging Model merging exhibits promising performance in multi-task learning (MTL). Nevertheless, previous studies insufficiently addressed the issue of magnitude. This section first presents the necessary notation and outlines the model merging process in Section II-A , then analyzes the factors contributing to magnitude deviation in Section II-B , and theoretically examines the impact of magnitude deviation on performance as well as mitigation strategies in Section II-C and Section II-D . II-A Preliminary Problem Setting . Let \theta_{\mathrm{pre}} denote the parameters of a pre-trained model, and consider K fine-tuned models derived from it. Each fine-tuned model consists of up to L layers, where \theta_{k}^{l} represents the parameters of the l -th layer in the k -th fine-tuned model.