Moss : Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models
Yifeng Cai, Ziqi Zhang, Ding Li, Yao Guo, Xiangqun Chen · 2025
Read Moss : Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models by Yifeng Cai, Ziqi Zhang, Ding Li, Yao Guo, Xiangqun Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10218.txt.
Introduction
Yifeng Cai, Ziqi Zhang, Ding Li, Yao Guo, and Xiangqun Chen propose Moss in a 2025 paper on federated learning. In that setup, phones and other devices train models on their own data and share updates instead of raw data. Devices differ a lot in size and power, so most methods merge only part of each model, which can hurt accuracy and waste time.
Moss takes a different path by merging all weights, using a shared proxy model to line up models that do not match. In tests across several tasks, the method trained faster than leading rivals. It also cut the training time and energy used on each device, raised accuracy, and used less network traffic. The paper argues that full merging can work even when device models differ a great deal.
Catalog record
| CREATOR | Yifeng Cai, Ziqi Zhang, Ding Li, Yao Guo, Xiangqun Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10218
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Moss : Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models Yifeng Cai, Ziqi Zhang, Ding Li, Yao Guo, Xiangqun Chen Abstract . Modern Federated Learning (FL) has become increasingly essential for handling highly heterogeneous mobile devices. Current approaches adopt a partial model aggregation paradigm that leads to sub-optimal model accuracy and higher training overhead. In this paper, we challenge the prevailing notion of partial-model aggregation and propose a novel "full-weight aggregation" method named Moss , which aggregates all weights within heterogeneous models to preserve comprehensive knowledge. Evaluation across various applications demonstrates that Moss significantly accelerates training, reduces on-device training time and energy consumption, enhances accuracy, and minimizes network bandwidth utilization when compared to state-of-the-art baselines. Journal: IMWUT Volume: 9 1 211 3 DOI: 10.1145/3712274 CCS: Human-centered computing Ubiquitous and mobile computing systems and tools CCS: Computing methodologies Neural networks CCS: Security and privacy Privacy protections 1. Introduction Federated learning (FL) ( McMahan et al., 2017 ; Zhao et al., 2018 ) has been applied to various mobile/IoT applications, such as human activity recognition ( Ouyang et al., 2021 ; Tu et al., 2021 ) , mobile vision ( Li et al., 2021a ; Li et al., 2021c ) , and voice assistance ( Leroy et al., 2019 ) . The key challenge for adopting FL in modern mobile/IoT environments is how to deploy models on heterogeneous devices while achieving desirable accuracy. Conventional FL requires that all mobile devices have the same model ( McMahan et al., 2017 ; Li et al., 2020b ; Li et al., 2020c ) . However, this requirement is not practical for mobile/IoT environments since devices can be highly heterogeneous ( Horvath et al., 2021 ) . The heterogeneity of devices leads to heterogeneity in the deployment of deep neural network (DNN) models. First, manufacturers and developers can select or customize design models to fit hardware characteristics ( Lee et al., 2020 ; Hussain et al., 2022 ; Hao et al., 2019 ) , resulting in heterogeneous model deployments across devices. Second, developers may choose models with varying architectures and parameter sizes for devices with different computational capacities to optimize performance ( Caldas et al., 2018 ) . Moreover, in the context of model heterogeneity, employing FL systems becomes imperative. This enables collaborative model training while ensuring privacy protection, ultimately enhancing model accuracy. Therefore, to address the heterogeneity between devices, researchers have developed multiple solutions that support developers to deploy heterogeneous models with different architectures to different devices in the FL system ( Li et al., 2021b ; Li et al., 2021a ; Li et al., 2021c ; Deng et al., 2022 ; Luo et al., 2021 ; Diao et al., 2020 ; Horvath et al., 2021 ; Li and Wang, 2019 ; Lin et al., 2020 ; Itahara et al., 2020 ; Cheng et al., 2021 ; Cho et al., 2022 ) . The key challenge for heterogeneous FL is how to aggregate weights across models with different architectures . Unlike conventional homogeneous FL, it is impossible to directly have the conventional full-weight aggregation (i.e., model aggregation based on the index of weight) between heterogeneous models. To address this challenge, researchers developed several approaches following the three types of solutions ( Fan et al., 2023 ) : pruning-based solutions ( Li et al., 2021b ; Li et al., 2021a ; Li et al., 2021c ; Deng et al., 2022 ; Luo et al., 2021 ; Diao et al., 2020 ; Horvath et al., 2021 ) , distillation-based solutions ( Li and Wang, 2019 ; Lin et al., 2020 ; Itahara et al., 2020 ; Cheng et al., 2021 ; Cho et al., 2022 ) , and alternative solutions ( Litany et al., 2022 ; Rapp et al., 2022 ; Yao et al., 2021 ) . We illustrate the common design of pruning-based and distillation-based solutions in Figure 1 . Figure 1 . Illustration of pruning-based solutions and distillation-based solutions. Specifically, pruning-based solutions ( Li et al., 2021b ; Li et al., 2021a ; Li et al., 2021c ; Deng et al., 2022 ; Luo et al., 2021 ; Diao et al., 2020 ; Horvath et al., 2021 ) capitalize on the concept of model pruning, which posits that existing models are over-parameterized. Therefore, these solutions first initialize an original model and then remove some of the less crucial weights within the model to tailor it to adapt the computational capabilities of the devices. During the aggregation phase of federated learning, only the weights of the pruned sub-models in heterogeneous models are aggregated and updated; the weights that have been pruned are not updated. However, when models are highly heterogeneous, these solutions may fail. If the original model is excessively pruned to adapt devices with lower computational power ( Chen et al., 2024 ) , it can disrupt the original model architecture, leading to significant reductions in accuracy and preventing the model from converging ( Liu et al., 2018 ; Zhu and Gupta, 2017 ) . Moreover, many devices may have model architectures that are custom-designed for specific devices ( Lee et al., 2020 ; Hussain et al., 2022 ; Hao et al., 2019 ) , making it difficult to find a consistent original architecture across different device models, thus pruning-based solutions have significant limitations. Distillation-based solutions ( Li and Wang, 2019 ; Lin et al., 2020 ; Itahara et al., 2020 ; Cheng et al., 2021 ; Cho et al., 2022 ) are based on the principle that the output dimensions of heterogeneous models within the same task are identical. These solutions first utilize a public dataset as input to generate outputs from heterogeneous models, which are then aggregated. Subsequently, model distillation techniques are employed to compute the differences between the aggregated outputs and the original outputs, thereby optimizing each heterogeneous model. However, these solutions overlook the knowledge in the intermediate layers of the models, and the outputs they utilize contain less information ( Romero et al., 2014 ; Heo et al., 2019 ; Tung and Mori, 2019 ) , which extends the convergence time of training. For devices with limited computational capabilities, this significantly increases computational overhead and reduces the model accuracy. Alternative solutions ( Fan et al., 2023 ) also tried to achieve FL with heterogeneous models. HAFL-GHN ( Litany et al., 2022 ) employs a Graph HyperNetwork to adapt to heterogeneous client architectures while preserving privacy. However, it does not establish correspondences between different architectures and avoids aggregating layers without matches, making it hard to apply in the real-world. DISTREAL ( Rapp et al., 2022 ) introduces a distributed resource-aware learning pattern that allows clients to adapt by omitting dynamic filters. Yet, like pruning-based methods, it requires an initial large model, limiting its use in highly heterogeneous environments. FEDHM ( Yao et al., 2021 ) decomposes model parameters into shared and individual components, reducing communication costs. However, it incurs significant computational overhead and demands shared parameters that exceed the capabilities of low-end devices, hindering its adaptability in diverse settings. In summary, all of the solutions suffer from sub-optimal accuracy and training efficiency. The reason is that these approaches take a partial model aggregation paradigm, thus can only aggregate a subset