FedMerge: Federated Model Merging for Personalization
Shutong Chen, Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi Zhang · 2025
Read FedMerge: Federated Model Merging for Personalization by Shutong Chen, Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06768.txt.
Introduction
Shutong Chen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang propose a new federated learning method called FedMerge in this 2025 arXiv paper. Federated learning trains models across many devices without pooling their data. One shared model often fits clients poorly when their data differ, yet training many models still leaves each client needing local fine-tuning.
FedMerge instead keeps a few global models and blends them with weights tuned for each client, making a personal model by simple merging. The server then sends each client its own blended model rather than the same one for all. In tests across three varied setups, the method beat clustering and mixture-of-experts rivals. It also smoothed the drift that builds when local training restarts from a distant global model.
Catalog record
| CREATOR | Shutong Chen, Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06768
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FedMerge: Federated Model Merging for Personalization Shutong Chen, Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi Zhang Abstract One global model in federated learning (FL) might not be sufficient to serve many clients with non-IID tasks and distributions. Despite recent advances in FL to train multiple global models for better personalization, they only provide limited model choices to clients, so local finetuning of multiple models is still indispensable. This paper proposes a novel “FedMerge” approach that can create a single personalized model per client by simply merging multiple global models with automatically optimized and customized weights. We formulate this problem as a joint optimization of global models and the merging weights per client. Unlike existing FL approaches, where the server broadcasts one or multiple global models to all clients, the server only needs to send a customized, merged model to each client. Moreover, instead of periodically interrupting the local training and re-initializing it to a global model, the merged model aligns better with each client’s task and data distribution, smoothing the local-global gap between consecutive rounds caused by client drift. We evaluate FedMerge on different non-IID settings applied to various domains with diverse tasks and data types, in which FedMerge consistently outperforms existing FL approaches, including clustering-based and mixture-of-experts (MoE) based methods. 1 University of Technology Sydney, Australia 2 University of Maryland, College Park, USA 3 The Hong Kong Polytechnic University, China [email protected], [email protected], [email protected], [email protected], [email protected] Code — https://github.com/shutong043/FedMerge Introduction Federated learning (FL) enables decentralized and collaborative learning of models across various clients without sharing their local data. By exchanging only local models instead of raw data, FL ensures data privacy and security while leveraging distributed datasets for improved performance. In particular, local datasets across clients are often non-identically and independently distributed (non-IID), making the design of algorithms to address such non-IIDness a fundamental challenge. (a) The architecture of Federated Learning with Mixture of Experts . Each client receives d global models from the server and optimizes them separately. The client’s resource consumption is proportional to the number of global models, making it inefficient. (b) The architecture of the proposed Federated Learning with Model Merging (FedMerge) . The merging operation (weight averaging) is performed on the server side instead of on each client. Only the merged models, \theta , are communicated and optimized, making the cost irrelevant to the number of global models. Figure 1: Comparison between MoE-like FL methods and the proposed FedMerge. Both methods perform weight averaging over global models. MoE-like FL performs weight averaging on each client, while FedMerge performs weight averaging on the server. Model merging ( Yadav et al. 2024 ) is a knowledge ensemble technique that integrates multiple models into a single unified model. It has recently gained increasing attention in the context of foundation models ( Huang et al. 2023 ; Chronopoulou et al. 2023 ; Yang et al. 2024a ; Kim et al. 2025 ; Wang et al. 2025 ) , as it enables efficient knowledge integration across diverse domains. Instead of training a large model from scratch, model merging combines existing pre-trained models into an enriched knowledge space, allowing effective reuse of previously trained experts. This flexibility naturally extends to parameter-efficient tuning methods, such as Low-Rank Adaptation (LoRA), which can be merged with minimal computational overhead. Due to these efficiency advantages, LoRA has been widely explored in FL settings ( Yi et al. 2023 ; Yang et al. 2024b ; Sun et al. 2024 ; Cho et al. 2024 ) . These properties make model merging a promising approach for scaling and personalizing federated learning. Federated learning with multiple models is an important direction for addressing client heterogeneity ( Marfoq et al. 2021 ; Wu et al. 2023 ) , and a recent dominant approach in this line of work is Mixture-of-Experts (MoE) ( Luo et al. 2024 ; Feng et al. 2025 ; Jiang et al. 2025 ; Radwan et al. 2025 ) . In these MoE-like FL approaches, the server maintains a “model soup” of diverse experts that are shared across clients. As illustrated in Fig. 1(a) , each client downloads multiple (or all) expert models and combines them locally to form a personalized model. However, sharing multiple models across clients introduces an intrinsic conflict: the desire for a rich, diverse set of global models conflicts with the limited computational and communication resources of individual clients. The cost for each client scales with the number of models it downloads, making it impractical to leverage a large model soup. While selecting a subset of models for each client can alleviate this issue ( Chronopoulou et al. 2023 ) , it often requires task-specific knowledge, which limits its generalization ability. Model merging provides an elegant solution to the scalability issues faced by MoE-like FL methods. By integrating multiple expert models into a single personalized model, model merging allows each client to benefit from the richness and diversity of the model soup without having to download or train multiple models locally. This fundamentally eliminates the linear growth of communication and computation costs on the client side, making it feasible to leverage a large number of global models even in resource-constrained environments. Building on this insight, we propose Federated Learning with Model Merging (FedMerge) . As shown in Fig. 1(b) , the server merges global models using client-specific weights to construct a unique personalized model for each client. Clients then only need to communicate and optimize a single merged model, while the server handles the merging operation with its typically greater resources. A challenge introduced by this design is how to update both the global models and the merging weights when only merged models are trained on clients. To address this, we formulate FedMerge as a joint optimization problem and derive a backpropagation-like update strategy to enable end-to-end training for both components. Experimental results on various non-IID settings show that FedMerge consistently outperforms clustering-based and MoE-based FL approaches. We summarize the main contributions of this paper: • We propose a novel FL framework to solve the conflict between using multiple global models for personalization and the limited resources of clients. • We introduce the Federated Merging (FedMerge) algorithm, where the server provides a personalized model to each client by merging a set of global models with client-specific weights. • Convergence analysis is provided in the Appendix E . • Experimental results demonstrate that FedMerge offers a cost-effective way to improve performance by supporting multiple global models while maintaining a constant and low client-side cost. Related Work Model Merging Model merging refers to methods aimed at reusing the knowledge contained in multiple models. The fundamental technique of model merging is to integrate existing models to form new ones. There have been a number of works focusing on merging methodologies, such as simple averaging ( Wortsman et al. 2022 ) , merging with Fisher Information ( Matena and Raffel 2022 ) , and merging with task vectors ( Ilharco et al. 2022 ) . Recently, model merging has become closely related to the development of foundation models ( Lu et al. 2023 ; Lu et al. 2024 ; Ostapenko et al. 2024 ; Zhou et al. 2025 ; Cheng et al. 2025 ; Du et