Multi-Level Collaboration in Model Merging
Qi Li, Runpeng Yu, Xinchao Wang · 2025
Read Multi-Level Collaboration in Model Merging by Qi Li, Runpeng Yu, Xinchao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01268.txt.
Introduction
Qi Li, Runpeng Yu, and Xinchao Wang study when merging models can match ensembling them, in a 2025 paper. Merging joins model weights into one model, while ensembling averages the predictions of many models and is seen as the upper bound. Past links between the two held only under narrow conditions, such as two models from one shared start.
The authors prove a performance link that holds even when those conditions drop. They then build a validation frame, Neural Ligand, with a loss shaped by that theory. Tests show merging can equal ensembling across model sizes and counts of models. The work gives multi task merging a firmer base.
Catalog record
| CREATOR | Qi Li, Runpeng Yu, Xinchao Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01268
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-Level Collaboration in Model Merging Qi Li, Runpeng Yu, Xinchao Wang Abstract Parameter-level model merging is an emerging paradigm in multi-task learning with significant promise. Previous research has explored its connections with prediction-level model ensembling—commonly viewed as the upper bound for merging—to reveal the potential of achieving performance consistency between the two. However, this observation relies on certain preconditions, such as being limited to two models, using ViT-based models, and all models are fine-tuned from the same pre-trained checkpoint. To further understand the intrinsic connections between model merging and model ensembling, this paper explores an interesting possibility: If these restrictions are removed, can performance consistency still be achieved between merging and ensembling? To answer this question, we first theoretically establish a performance correlation between merging and ensembling. We find that even when previous restrictions are not met, there is still a way for model merging to attain a near-identical and superior performance similar to that of ensembling. To verify whether our findings are practical, we introduce a validation framework termed Neural Ligand (NeuLig). The learning process of NeuLig is meticulously designed with a specialized loss function supported by theoretical foundations. Experimental results demonstrate the robust resilience of NeuLig in terms of both model scale and the number of collaborating models. For instance, for the case involving 5 CLIP-ViT-B/32 models, parameter-level merging achieves the same performance as prediction-level ensembling (merging: 95.44% vs. ensembling: 95.46%). Please check our repo here . 1 Introduction Exploring the intrinsic connections between multiple models to enable efficient reuse and collaboration has long been a core issue in multi-task learning Zhang and Yang 2021 ; Chen et al. 2024 ; Li et al. 2024 ; Li et al. 2023 . Recently, a new paradigm named model merging Wortsman et al. 2022a ; Ilharco et al. 2023 ; Yadav et al. 2024 ; Jin et al. 2023 ; Yang et al. 2024 ; Tang et al. 2024 has emerged. This paradigm merges single-task models at the parameter-level using task vectors Ilharco et al. 2023 to create a unified multi-task model. It enables the seamless integration of specialized knowledge from each single-task model, resulting in enhanced efficiency and adaptability across tasks. Besides model merging, another widely-used paradigm in multi-task learning called model ensembling Dong et al. 2020 ; Sagi and Rokach 2018 ; Polikar 2012 ; Zhou and Zhou 2021 ; Zhou et al. 2021 ; Zhang and Ma 2012 improves multi-task learning performance by aggregating outputs from multiple models rather than merging parameters. Recent studies Wortsman et al. 2022b ; Jacot et al. 2018 ; Fort et al. 2020 ; Wortsman et al. 2022a have started to explore the connections between these two paradigms. For example, results in Wortsman et al. 2022a suggest that when two models collaborate, parameter-level merging and prediction-level ensembling can yield nearly comparable performance at the data level. Despite the promising results, prior related work primarily focuses on collaborations that (1) center around ViT-based models, (2) are fine-tuned from the same pre-trained checkpoint, and (3) are limited to only two models. We argue that a deeper investigation into this phenomenon is both necessary and valuable, as current model merging methods—typically relying on task vectors or their variations—often yield performance that falls short of standalone models Ilharco et al. 2023 ; Yadav et al. 2024 ; Jin et al. 2023 ; Yang et al. 2024 and ensembling is often considered an upper bound on merging performance Stoica et al. 2023 . An open question remains as to whether it is theoretically feasible for merging to match the performance of ensembling in cases involving multiple models (more than two, which is the primary focus of merging) and, if so, under what specific conditions such consistency could be achieved. Achieving this can greatly liberate the potential of multi-model collaboration, especially when multiple entities collaborate while privacy constraints vary across stages—sometimes allowing model parameter sharing, while at other times only permitting model output sharing Yao 1982 ; Gentry 2009 ; Smith et al. 2017 . Grounded in this significance, we take an initial step toward exploring this intriguing question: When multiple models collaborate, can we find an inherent link between parameter-level merging and prediction-level ensembling that could enable performance consistency for both sides? We first theoretically demonstrate that merging and ensembling can yield a performance difference of the second order of smallness under certain conditions. This holds regardless of model scale and the number of collaborating models. Figure 1 : An illustration of Portland, which consists of a linear layer followed by a softmax function. Furthermore, to verify it in practice, we design a validation framework called Neu ral Lig and (NeuLig) . In NeuLig , we train an extremely small (single-layer) neural network termed Portable Ligand (Portland). As shown in Figure 1 , Portland takes the outputs of several models on each data as input and generates a corresponding Cooperative Vector (CoopVec), with one entry per model for each data. These vectors can then be used for either parameter-level merging or prediction-level ensembling. The learning process of Portland is meticulously designed with a specialized loss function, which is backed by our theoretical findings. The loss function comprises two terms: the boosting term and the alignment term. We adopt an ensemble-driven merging strategy, where the boosting term guides Portland to enhance prediction-level ensembling performance, while the alignment term is designed to bring the performance of merging in line with that of ensembling. Ultimately, this enables the realization of performance consistency between merging and ensembling. Results on multi-model collaboration (e.g., five CLIP-RN-50 models or five CLIP-ViT-B/32 models) demonstrate the effectiveness of NeuLig . Furthermore, we find that the performance consistency persists even when models are trained from random initialization rather than fine-tuned from the same pre-trained checkpoint. We also explore the behavior of NeuLig across different scenarios and provide relevant insights. Our contributions are summerized as follows: • We explore whether parameter-level merging and prediction-level ensembling can achieve performance consistency in multi-model collaboration at the data level, offering theoretical support for this potential. • We develop a validation framework named Neu ral Lig and (NeuLig) to further verify our findings. In this framework, we introduce a single layer network termed Portland to perform merging and ensembling in a unified manner. • Extensive experimental results validate the feasibility of NeuLig . For instance, we observe an almost negligible performance gap (0.02%) between merging (95.44%) and ensembling (95.46%) in the collaboration of five ViT-based models. We further explore the nature of NeuLig from various perspectives and provide detailed discussions, offering a new perspective on understanding model merging and ensembling. 2 Related Work Model Merging for Multi-Task Learning. Model merging has received significant attention for its storage- and computation-efficient properties, showing promise in improving model generalization and supporting multi-task learning (MTL) Zhang and Yang 2021 ; Evgeniou and Pontil 2004 ; Sener and Koltun 2018 ; Li et al. 2023 . From a task scope perspective, current merging methods can be divided into two categories Yang et al. 2024 ; Zhang et al. 2024 : single-task and multi-task model merging. The former merges multiple models trained on the same task, either to improve generalization Gupta et al. 2020 ; Cha et al.