From Task-Specific Models to Unified Systems: A Review of Model Merging Approaches

Wei Ruan, Tianze Yang, Yifan Zhou, Tianming Liu, Jin Lu · 2025

Read From Task-Specific Models to Unified Systems: A Review of Model Merging Approaches by Wei Ruan, Tianze Yang, Yifan Zhou, Tianming Liu, Jin Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08998.txt.

Introduction

Wei Ruan and colleagues surveyed model merging, the craft of fusing trained networks into one, in a 2025 review. Thousands of fine-tuned models now sit public, but their training data often stays private for legal or commercial reasons, ruling out classic multi-task training on shared data. Merging the models' parameters directly is the alternative, and methods have multiplied without a common language: terminologies clash and categories overlap, making fair comparison hard.

The review establishes a fresh taxonomy of merging approaches, compares them systematically, and maps the field's key developments and open directions. It argues merging is becoming the practical route to unified systems with broad generalization when data cannot move. For practitioners drowning in model zoos, the survey offers both a map and a vocabulary.

Catalog record

CREATORWei Ruan, Tianze Yang, Yifan Zhou, Tianming Liu, Jin Lu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08998

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Task-Specific Models to Unified Systems: A Review of Model Merging Approaches Wei Ruan, Tianze Yang, Yifan Zhou, Tianming Liu, Jin Lu Abstract Model merging has achieved significant success, with numerous innovative methods proposed to enhance capabilities by combining multiple models. However, challenges persist due to the lack of a unified framework for classification and systematic comparative analysis, leading to inconsistencies in terminologies and categorizations. Meanwhile, as an increasing number of fine-tuned models are publicly available, their original training data often remain inaccessible due to privacy concerns or intellectual property restrictions. This makes traditional multi-task learning based on shared training data impractical. In scenarios where direct access to training data is infeasible, merging model parameters to create a unified model with broad generalization across multiple domains becomes crucial, further underscoring the importance of model merging techniques. Despite the rapid progress in this field, a comprehensive taxonomy and survey summarizing recent advances and predicting future directions are still lacking. This paper addresses these gaps by establishing a new taxonomy of model merging methods, systematically comparing different approaches, and providing an overview of key developments. By offering a structured perspective on this evolving area, we aim to help newcomers quickly grasp the field’s landscape and inspire further innovations. 1 Introduction The increasing prevalence of open-source models trained on diverse tasks provides unprecedented opportunities to utilize pre-trained weights for various applications. However, access to the original training data is often restricted due to privacy concerns, proprietary limitations, or other constraints, posing significant challenges for tasks requiring cross-domain capabilities Jin et al. 2022 . Model merging techniques address this issue by enabling the combination of model weights without relying on original data, thereby equipping models with the ability to handle multiple tasks effectively. Currently, model merging has emerged as a promising solution, enabling the combination of multiple models with similar architectures to harness complementary strengths. This approach not only enhances task-specific performance but also fosters greater adaptability across tasks Yang et al. 2024a ; Tam et al. 2024 . Model merging provides several key advantages Yang et al. 2024a ; Yu et al. 2024b ; Zhao et al. 2024 . Firstly, it allows for the aggregation of knowledge across multiple models without requiring extensive retraining, thereby offering a more resource-efficient alternative to traditional fine-tuning and transfer learning. Additionally, model merging can mitigate issues like catastrophic forgetting and offers a pathway to create models that encapsulate the strengths of multiple training regimes. For instance, weight average Wortsman et al. 2022 ; Choshen et al. 2022 and task arithmetic merging Ilharco et al. 2022 are widely adopted methods for retaining model capabilities while maintaining robustness across varied domains. In recent years, model merging techniques have evolved from simple linear interpolation methods or weight averaging to more sophisticated approaches Yang et al. 2024a ; Sung et al. 2023 . These include weight interference suppression, parameter freezing, and decoupling parameters for old and new tasks, allowing for the fine-tuning of specific model aspects while preserving core functionalities. There is also growing interest in integrating merging methods with Mixture of Experts (MoE) frameworks, where specialized ”experts” are dynamically engaged based on the task requirements. These advancements underscore the potential of model merging as a versatile solution, capable of adapting to a range of tasks and minimizing resource demands. Although model merging has achieved remarkable results and numerous innovative methods have been developed, drawing increasing attention from researchers, this field is still in its early stages and faces several challenges. First, existing methods lack a unified framework for classification and comparative analysis. There is no comprehensive perspective to systematically understand and define model merging as a discipline, leaving inconsistencies in terminologies and categorizations. Second, the field lacks in-depth surveys that provide a thorough investigation of existing approaches. This absence makes it difficult for newcomers to quickly and comprehensively grasp the methodologies in this area. Moreover, there is a lack of systematic and forward-looking thinking about the future trends and potential development of this domain. This paper seeks to address these gaps by presenting a new taxonomy of model merging methods (Figure 1 ). We categorize and analyze various approaches, examine the distinct challenges faced in model merging, and propose a comprehensive framework for advancing this research area . Our aim is to provide a foundational reference for researchers interested in model merging and to catalyze future innovations in this field, bridging the existing knowledge gaps and helping newcomers rapidly understand this domain. Figure 1: An overview of common model merging methods and their main distinctions. 2 Classification of Model Merging 2.1 Permutation type Model merging was initially introduced to address challenges arising from Linear Mode Connectivity (LMC) barriers and Permutation Invariance Garipov et al. 2018 . LMC barriers refer to the obstacles encountered when attempting to linearly interpolate between two independently trained neural networks, which often results in suboptimal performance due to differences in their parameter spaces Zhou et al. 2023 ; Singh et al. 2024 . These barriers highlight the difficulty in aligning the solutions found by stochastic gradient descent (SGD), as they typically reside in separate loss basins Freeman and Bruna 2016 . To overcome LMC barriers, various approaches have been proposed, including post-hoc neuron alignment techniques, permutation symmetry reduction methods, and training-time alignment strategies, each aiming to improve the connectivity and compatibility of independently trained models Li et al. 2024a . In neural network training, independently trained models with the same architecture often exhibit unaligned weights, a phenomenon stemming from permutation invariance. This invariance implies that the order of neurons within a layer does not affect the overall functionality of the network. As a result, when two models trained separately are merged, their corresponding weights do not naturally align, making direct merging ineffective or suboptimal Ainsworth et al. 2022 ; Zhou et al. 2023 . Without alignment, merging such models can result in high loss barriers between their weights, which hinders Linear Mode Connectivity (LMC) Zhou et al. 2023 . LMC refers to the existence of a smooth path in the loss landscape that connects two models. When weights are misaligned, naive merging can degrade model performance due to incompatibilities in the combined representations Li et al. 2024b . Permutation-type model merging assumes that the models are trained on the same dataset but with different initializations and training parameters. Currently, several commonly used methods exist to align the weights before merging neural networks: Weight Matching: Due to the permutation invariance property of neural networks, neurons within a layer can be arbitrarily reordered without altering the model’s functionality. Weight Matching leverages this property by searching for an optimal permutation matrix that minimizes the L2 distance between the parameters of two models. This is achieved by aligning corresponding neurons in a way that ensures the functional similarity of the models is preserved Ito et al. 2024 .This method solves a Linear Assignment Problem (LAP) to match the rows and columns of weight matrices between two models. It minimizes the difference between weights by finding the best permutation Ainsworth et al. 2022 ; Mena et al. 2018 ; Cai et al. 2024 ; Ito et al. 2024 .Neural Functional Networks (NFNs) Zhou et al. 2024a leverage the permutation invariance property

Read the complete text