Understanding Model Merging: A Unified Generalization Framework for Heterogeneous Experts
Qinglun Li, Anke Tang, Miao Zhang, Mengzhu Wang, Quanjun Yin, Li Shen · 2026
Read Understanding Model Merging: A Unified Generalization Framework for Heterogeneous Experts by Qinglun Li, Anke Tang, Miao Zhang, Mengzhu Wang, Quanjun Yin, Li Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21690.txt.
Introduction
Qinglun Li, Anke Tang, Miao Zhang, Mengzhu Wang, Quanjun Yin, and Li Shen build a unified theory for merging trained models. Merging blends several fine-tuned models into one, working only on their weights. It needs no original data and no costly retraining, yet it often works surprisingly well.
A general theory for why it works was missing, above all when the merged models were tuned with different settings. Learning rates and batch sizes vary widely in practice. The authors' framework explains merging across that heterogeneity. A firm theory tells builders when a merge is safe and when it will quietly average away the very skills they wanted.
Catalog record
| CREATOR | Qinglun Li, Anke Tang, Miao Zhang, Mengzhu Wang, Quanjun Yin, Li Shen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 107 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21690
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding Model Merging: A Unified Generalization Framework for Heterogeneous Experts Qinglun Li, Anke Tang, Miao Zhang, Mengzhu Wang, Quanjun Yin, Li Shen Abstract Model merging efficiently aggregates capabilities from multiple fine-tuned models into a single one, operating purely in parameter space without original data or expensive re-computation. Despite empirical successes, a unified theory for its effectiveness under heterogeneous finetuning hyperparameters (e.g., varying learning rates, batch sizes) remains missing. Moreover, the lack of hyperparameter transparency in open-source fine-tuned models makes it difficult to predict merged-model performance, leaving practitioners without guidance on how to fine-tune merge-friendly experts. To address those two challenges, we employ L 2 L_{2} -Stability theory under heterogeneous hyperparameter environments to analyze the generalization of the merged model 𝒙 a v g \boldsymbol{x}_{avg} . This pioneering analysis yields two key contributions: (i) A unified theoretical framework is provided to explain existing merging algorithms, revealing how they optimize specific terms in our bound, thus offering a strong theoretical foundation for empirical observations. (ii) Actionable recommendations are proposed for practitioners to strategically fine-tune expert models, enabling the construction of merge-friendly models within the pretraining-to-finetuning pipeline. Extensive experiments on the ResNet/Vit family across 20/8 visual classification tasks, involving thousands of finetuning models, robustly confirm the impact of different hyperparameters on the generalization of 𝒙 a v g \boldsymbol{x}_{avg} predicted by our theoretical results. 1 Introduction The paradigm of pre-training followed by fine-tuning has become the dominant paradigm in computer vision Yuan et al. 2021 ; Chung et al. 2024 ; Wortsman et al. 2022b , leading to an explosion of powerful foundation models Zheng et al. 2025b ; Dosovitskiy 2020 ; Brown et al. 2020 and a vast ecosystem of specialized “expert” models fine-tuned for specific downstream tasks Guo et al. 2019 . Although these experts perform well on their respective tasks, selecting a specific expert model for each downstream task is cumbersome. A critical challenge has emerged: how can we efficiently aggregate the distinct capabilities of multiple expert models into a single, multi-talented model? Model merging offers a compelling answer. By operating directly in the parameter space, methods like Model Soups Wortsman et al. 2022a ; Rame et al. 2023 ; Chegini et al. 2024 and Task Arithmetic Ilharco et al. 2023 can fuse models without needing access to their original training data or incurring the prohibitive costs of re-training from scratch. Moreover, model merge techniques are also crucial for the advancement of current LLMs Tam et al. 2024 ; Fu et al. 2025 ; Wang et al. 2025c . Since model merging methods can operate directly in the parameter space without accessing the original training data and are also computationally efficient, researchers have proposed a wide range of effective techniques from different perspectives Yadav et al. 2023 ; Yu et al. 2024 ; Ilharco et al. 2023 ; Yang et al. 2024c ; Yang et al. 2024b . However, these methods are often explained heuristically, and there is still no unified theoretical framework to explain why and when they work. This theoretical gap also leads to significant practical challenges. For example, in open-source communities such as Hugging Face , practitioners now have access to a large number of pretrained and fine-tuned models ( Wolf et al. 2020 ) . However, the key hyperparameters used during fine-tuning are often unavailable or non-transparent. This lack of transparency introduces two main difficulties: (i) The performance of model merging becomes highly unpredictable ( Yang et al. 2024c ) , and (ii) The trial-and-error process to identify suitable combinations greatly increases experimental cost ( Huang et al. 2024a ; Zheng et al. 2025a ) . Furthermore, without theoretical guidance, practitioners attempting to merge multiple “expert models” face two central questions: (i) Since the underlying mechanisms that determine the effectiveness of different model merging methods remain unclear, it is difficult to select an appropriate merging strategy for a given set of models; and (ii) Without knowing how fine-tuning processes and their hyperparameters influence the final merging performance, it is challenging to fine-tune new models that can serve as good candidates for future merging—so-called merge-friendly experts (i.e., expert models with good merging compatibility) ( Wu et al. 2023 ; Dziadzio et al. 2025 ) . This leaves the community with an unclear theoretical framework to guide the full pretrain-to-merge pipeline. Table 1 : Summary of challenges in model merging and our contributions. Aspect Prevailing Challenges & Gaps Our Contribution Theoretical - Lack of a unified theory to explain various methods. - Effectiveness is explained by heuristics, not principles. - Unclear why merging works in heterogeneous settings. A Unified Generalization Framework : We derive the first excess error bound via L_{2} -Stability, explaining why different methods are effective. Practical - Merged model performance is highly unpredictable. - No clear guidance on how to fine-tune models for merging. - High trial-and-error costs for practitioners. Actionable Guidance for Practitioners : Our bound predicts hyperparameter impact, providing a guide to create ‘merge-friendly’ experts. Empirical - Lack of systematic, large-scale studies connecting theory to practice under varied hyperparameters. Extensive Empirical Validation : We finetune thousands of models across 20 tasks, showing our theory strongly aligns with real-world results. To bridge these critical gaps in both theory and practice, our work seeks to answer two fundamental questions: (1) What makes different model merging algorithms effective, and can these methods be unified within a single framework? (2) How do the hyperparameters of these algorithms affect their performance, and how can practitioners adjust them to improve the performance of \boldsymbol{x}_{avg} ? To answer these questions, we provide a rigorous theoretical analysis of model merging through the lens of L_{2} -Stability Lei and Ying 2020 ; Li et al. 2025 . Our primary contribution is the first excess error bound for models merged from experts trained in heterogeneous hyperparameter environments. This pioneering theoretical result yields two key insights. First, it provides a unified framework that explains the effectiveness of diverse merging algorithms. We theoretically demonstrate how different methods, from simple averaging Ilharco et al. 2023 to sparsity-based approaches Yu et al. 2024 , can be interpreted as strategies to optimize distinct terms within our bound, thus providing a strong theoretical foundation for previously empirical observations. Second, our analysis yields actionable recommendations for practitioners. By elucidating the precise role each hyperparameter (e.g., learning rate \eta_{l} , batch size b_{i} , training steps K_{i} ) plays in the generalization of the final merged model, we offer a strategic guide for constructing merge-friendly experts. Empirically, we validate our theory through extensive experiments, performing thousands of fine-tuning and merging trials on the ResNet family across diverse image classification tasks Wang et al. 2024a ; Tang et al. 2025 . The results demonstrate a remarkable alignment between our theory and empirical generalization trends. Moreover, this work establishes the first unified and predictive mathematical framework for model merging, transforming it from an empirical craft into a principled and systematic discipline, and offering theory-grounded insights to guide future developments in the field. Table 1 summarizes the main challenges and our contributions. 2 Related Works This section reviews diverse model merging methods and the current theoretical advancement in the field. At the end of each subsection, we explicitly outline the