Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging
Zhixiang Wang, Zhenyu Mao, Yixuan Qiao, Yunfang Wu, Biye Li · 2025
Read Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging by Zhixiang Wang, Zhenyu Mao, Yixuan Qiao, Yunfang Wu, Biye Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12217.txt.
Introduction
Zhixiang Wang and co-authors merge language models while taming the clash between their parts. Merging offers a cheap way to blend skills without fresh training. But weights from unlike models fight each other, and quality drops. Their method, OBIM, fights the clash on two fronts.
First, each weight is graded by how much the loss moves when it is nudged. Only high grade weights are kept, cutting fights inside one model. Then models are folded in one at a time under a mask that bars direct averaging. Tests on tuned and pretrained models show clear wins over prior merging tricks. The authors pitch it as a practical recipe for model blending.
Catalog record
| CREATOR | Zhixiang Wang, Zhenyu Mao, Yixuan Qiao, Yunfang Wu, Biye Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12217
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging Zhixiang Wang, Zhenyu Mao, Yixuan Qiao, Yunfang Wu, Biye Li Abstract Large Language Models (LLMs) have demonstrated impressive capabilities, but their high computational costs pose challenges for customization. Model merging offers a cost-effective alternative, yet existing methods suffer from interference among parameters, leading to performance degradation. In this work, we propose O ptimal B rain I terative M erging (OBIM), a novel method designed to mitigate both intra-model and inter-model interference. OBIM consists of two key components: (1) A saliency measurement mechanism that evaluates parameter importance based on loss changes induced by individual weight alterations, reducing intra-model interference by preserving only high-saliency parameters. (2) A mutually exclusive iterative merging framework , which incrementally integrates models using a binary mask to avoid direct parameter averaging, thereby mitigating inter-model interference. We validate OBIM through experiments on both Supervised Fine-Tuned (SFT) models and post-pretrained checkpoints. The results show that OBIM significantly outperforms existing merging techniques. Overall, OBIM provides an effective and practical solution for enhancing LLM merging. We will publicly release our code upon the acceptance of this paper. 1 Introduction Existing research Akiba et al. 2025 ; Dekoninck et al. 2023 ; Wan et al. 2024a has demonstrated that a composite LLM can be constructed by merging the parameters of different expert LLMs. Traditional approaches Wortsman et al. 2022 ; Matena and Raffel 2022 ; Jin et al. 2022 employ matrices to determine task-specific coefficients and perform a weighted average based on these coefficients. Methods grounded in task arithmetic Ilharco et al. 2023 leverage task vectors, defined as the difference between the parameter values of a fine-tuned model and those of its pre-trained counterpart, to effectively manipulate and integrate the knowledge embedded within the models. State-of-the-art model merging methods Yadav et al. 2024 ; Wang et al. 2024 ; Yu et al. 2024b have shown that task performance degradation is primarily caused by interference between parameter values, as aggregation operations, such as averaging, can alter the parameter distribution Yu et al. 2024a . The interference can be categorized into two types: intra-model interference and inter-model interference. Figure 1: Illustration of inter-model interference. The dotted box highlights cases where TIES fails to resolve interference. Approximately 46\% of parameters deviate from the original models due to task vector averaging in the absence of sign conflicts. Intra-model interference arises from redundant parameters within a single model. Due to the over-parameterized nature of neural networks Choudhary et al. 2020 ; He and Xiao 2023 , removing a significant portion of the parameters often has little impact on model performance Sun et al. 2023 ; Kim et al. 2024 . However, these redundant parameters introduce noise during the model merging process, adversely affecting the outcome. To address this, it is crucial to identify parameters that are closely related to the target task. Existing approaches, however, primarily rely on magnitude-based methods, assuming that parameter magnitude directly correlates with saliency. For instance, TIES Yadav et al. 2024 trims the parameters with the smallest magnitudes, the Model Breadcrumbs Davari and Belilovsky 2025 highlight the importance of removing the parameters with the largest weights to further reduce noise. While these methods demonstrate effectiveness, they fall short of fully revealing the true saliency of the parameters. Interference between models arises due to variations in parameter distributions Shoemake 1985 ; Jang et al. 2024 . Directly averaging these parameters can lead to performance degradation. TIES addresses this issue by resolving sign conflicts in parameter values, aligning them based on the direction of the largest total movement across models. Similarly, TALL-Mask Wang et al. 2024 is designed to exclude parameters that are relevant only to a subset of tasks. While these methods effectively mitigate inter-model interference under certain conditions, their effectiveness diminishes when parameter distributions deviate from expected patterns, causing them to revert to simple averaging. As shown in Figure 1 , when there is no sign conflict, TIES yields the same result as simple averaging, deviating from both input models and leading to suboptimal performance. To address the interference problem in model merging, we propose a novel method for LLMs called Optimal Brain Iterative Merging (OBIM). Our approach comprises two core components: a saliency measurement mechanism to filter intra-model interference and a mutually exclusive iterative merging framework to prevent inter-model interference. In detail, our approach measures the saliency of parameters within a single model by evaluating the loss change induced by altering each parameter. Inspired by layer-wise model pruning methods Frantar and Alistarh 2022 ; Frantar et al. 2022 , we forgo reliance on the overall model loss during training and instead independently apply the Mean Square Error (MSE) to each linear weight. This enables calculation of the output distribution difference between the trained weight and the original weight, providing a more precise and efficient measure of parameter saliency. By retaining parameters with high saliency, we effectively reduce intra-model interference in model merging. Subsequently, we design an iterative merging framework to integrate models step by step in a mutually exclusive manner, mitigating inter-model interference. Specifically, we employ a binary mask to track the positions that have already been merged. At each step, parameters with the highest saliency, which are not yet recorded in the mask, are selected from a model and integrated into the base model. This ensures that each position is occupied by only one parameter, thereby eliminating the need for averaging operations. In summary, we propose a novel method, OBIM, to mitigate both intra-model and inter-model interference in LLM merging. To validate the effectiveness of our method, we conducted model merging experiments on Supervised Fine-Tuned (SFT) models of Llama2 Touvron et al. 2023 for multi-task merging and post-pretrained checkpoints of Qwen2 Yang et al. 2024a for catastrophic forgetting recovery. The results of both experiments demonstrate that OBIM significantly outperforms existing approaches. In addition, extensive ablation studies and analyses of key factors provide a comprehensive understanding of OBIM. Figure 2: An overview of the proposed method. The left part depicts the iterative merging process, while the right part details how parameters are selected at each iteration step through the cooperation of parameter saliency and the merged mask. 2 Preliminaries 2.1 Model Merging Problem In this paper, we focus on merging models that are optimized from the same backbone. Given K models with parameters \{\theta^{1},\theta^{2},\dots,\theta^{K}\}\in\mathbb{R}^{d} , each trained on a distinct task or setting \{t_{1},t_{2},\dots,t_{K}\} from a shared base model \theta^{B}\in\mathbb{R}^{d} . Model merging aims to fuse these parameters into a single model with parameters \theta^{M}\in\mathbb{R}^{d} , and enable \theta^{M} to effectively handle all K tasks simultaneously. 2.2 Task Vector A task vector \delta^{k}\in\mathbb{R}^{d} for model \theta^{k} is defined as the delta weights between the trained model’s parameters and those of the backbone: \delta^{k}=\theta^{k}-\theta^{B}, (1) which represents both the direction and magnitude of parameter updates during training. By merging the task vectors \{\delta^{1},\delta^{2},\dots,\delta^{K}\} of K models into a single task vector \delta^{M} , the parameters of the merged model can be expressed as \theta^{M}=\theta^{B}+\delta^{M}. (2) 3 Methodology The proposed merging method comprises two key components: Optimal Brain Merging (OBM), a saliency-based mechanism that selects high-saliency parameters for merging, and Iterative Merging (IM), an iterative framework designed to mitigate interference between models. Figure 2 illustrates