SparsyFed : Sparse Adaptive Federated Training

Adriano Guastella, Lorenzo Sani, Alex Iacob, Alessio Mora, Paolo Bellavista, Nicholas D. Lane · 2025

Read SparsyFed : Sparse Adaptive Federated Training by Adriano Guastella, Lorenzo Sani, Alex Iacob, Alessio Mora, Paolo Bellavista, Nicholas D. Lane (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05153.txt.

Introduction

Adriano Guastella and co-authors present a 2025 paper on training AI models across many weak devices. In federated learning, phones and sensors train a shared model without sending private data away. Sparse models, with most weights switched off, cut traffic and compute. Yet sparse federated training is rarely used, for three reasons the team names.

Mixed data makes devices disagree on which weights matter. Fixed sparsity masks cannot adapt to varied data. Extra tuning knobs are painful to set across a fleet. Their method, SparsyFed, answers all three at once. It reaches 95 percent sparsity with barely any accuracy loss and one tuning knob. Masks regrow 200 times less per round than in prior work, and adapt to skewed data.

Catalog record

CREATORAdriano Guastella, Lorenzo Sani, Alex Iacob, Alessio Mora, Paolo Bellavista, Nicholas D. Lane
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05153

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SparsyFed : Sparse Adaptive Federated Training Adriano Guastella, Lorenzo Sani, Alex Iacob, Alessio Mora, Paolo Bellavista, Nicholas D. Lane Abstract Sparse training is often adopted in cross-device federated learning (FL) environments where constrained devices collaboratively train a machine learning model on private data by exchanging pseudo-gradients across heterogeneous networks. Although sparse training methods can reduce communication overhead and computational burden in FL, they are often not used in practice for the following key reasons: (1) data heterogeneity makes it harder for clients to reach consensus on sparse models compared to dense ones, requiring longer training; (2) methods for obtaining sparse masks lack adaptivity to accommodate very heterogeneous data distributions, crucial in cross-device FL; and (3) additional hyperparameters are required, which are notably challenging to tune in FL. This paper presents SparsyFed , a practical federated sparse training method that critically addresses the problems above. Previous works have only solved one or two of these challenges at the expense of introducing new trade-offs, such as clients’ consensus on masks versus sparsity pattern adaptivity. We show that SparsyFed simultaneously (1) can produce 95% sparse models, with negligible degradation in accuracy, while only needing a single hyperparameter, (2) achieves a per-round weight regrowth 200 times smaller than previous methods, and (3) allows the sparse masks to adapt to highly heterogeneous data distributions and outperform all baselines under such conditions. † † footnotetext: * Equal Contribution, correspondence to [email protected] 1 Dipartimento di Informatica - Scienza e Ingegneria, Università di Bologna 2 Department of Computer Science and Technology, University of Cambridge 3 Flower Labs, UK 1 Introduction Federated Learning ( McMahan et al., 2017 ) has become a standard technique for distributed training on private data ( Yang et al., 2018 ; Ramaswamy et al., 2019 ; Pati et al., 2022 ; Wang et al., 2023 ; Huba et al., 2022 ; Bonawitz et al., 2019 ) , particularly on edge devices. Given its application to constrained hardware, mitigating communication and computational overheads—significant in standard FL infrastructures ( Kairouz et al., 2021 ; Bellavista et al., 2021 ) —remains a key field focus. Practical cross-device FL methods typically assume stateless clients with imbalanced, heterogeneous datasets and constrained, diverse hardware ( Wang et al., 2021 ) . Restricted client hardware and low communication bandwidth significantly increase training time compared to centralized methods, elongating hyperparameter tuning ( Khodak et al., 2021 ) . Additionally, unknown data distributions and dynamic client availability demand robust optimization methods that can handle these variations. When device availability is constrained, the federated orchestrator may struggle to sample a representative client subset ( Eichner et al., 2019 ; Cho et al., 2020 ; Li et al., 2020b ) , inducing trade-offs between sampling ratio and efficiency ( Charles et al., 2021b ) . Sparse training methods improve computational and communication efficiency by reducing (a) memory footprint and FLOPs during training ( Raihan & Aamodt, 2020 ) , and (b) the communication costs ( Bibikar et al., 2022 ) . However, applying these methods in cross-device FL is challenging due to client availability and data heterogeneity, which can disrupt the binary mask structure across clients ( Qiu et al., 2022 ) . Such inconsistencies hinder consensus on the binary mask, lowering global model performance ( Babakniya et al., 2023 ) . Recent approaches address these issues using fixed sparse masks ( Huang et al., 2022 ; Qiu et al., 2022 ) or dynamic methods involving mask warmup and refreshing ( Babakniya et al., 2023 ) . However, fixed masks reduce the adaptability to unseen distributions, while dynamic methods require careful tuning of additional hyperparameters, like warmup duration and refresh interval. Fixed-mask methods also limit model plasticity—the ability to rewire and adapt to diverse distributions ( Lyle et al., 2022 ; Lyle et al., 2023 ) . For example, it is known that in a multi-task or continual learning setting, neural networks can be iteratively pruned to build task-specific sub-networks ( Mallya & Lazebnik, 2018 ) , a lost ability when adopting a fixed-mask method. This lack of adaptability makes fixed-mask approaches unsuitable for cross-device FL, where unseen distributions frequently arise. Thus, we argue that sparse training methods for cross-device FL should (a) adopt dynamic masking and (b) remain agnostic to optimization and selection methods while minimizing hyperparameter complexity. Figure 1: SparsyFed pipeline. (1) Server broadcasts the global model \omega_{t} . (2) Client i re-parameterizes local weights. (3) Executes a forward pass on batch \mathcal{B} . (4a) Computes layer-wise sparsity s_{t} . (4b) Prunes activations using s_{t} and stores them. (5) Computes grads. (6) Applies grads. (7) Computes model updates and applies Top-K pruning. (8) Sends sparse updates \Delta\tilde{\omega}_{i}^{t} back to the server. (9) Apply server optimizer to obtain the global model. Steps (2-6) repeat until convergence. Our work addresses these challenges by introducing SparsyFed , a sparse training method for training global models in cross-device federated learning. During local training, as illustrated in Fig. 1 , SparsyFed uses an easy-to-tune approach to prune (1) activations with an adaptive layer-wise method and (2) model weights before communication using the target sparsity. These strategies can reduce FLOPs and memory usage, with suitable hardware support ( Raihan & Aamodt, 2020 ) , and the communication costs by transmitting sparse updates. Our layer-wise approach prunes based on each layer’s parameter proportion, unlike previous methods ( Babakniya et al., 2023 ; Qiu et al., 2022 ) , which applied fixed global sparsity across layers. This approach removes more capacity from dense layers while preserving parameter-efficient ones. SparsyFed also prunes models at the end of each round, allowing the complete flexibility of the local optimization. SparsyFed provides robust initialization for crucial layers, such as embeddings, without excluding layers to preserve performance as done in past works ( Qiu et al., 2022 ; Raihan & Aamodt, 2020 ) . Additionally, SparsyFed employs a sparsity-inducing weight re-parameterization ( Schwarz et al., 2021 ) based on a single parameter, enhancing the model’s resilience to sparsity and improving adaptivity, which enables changing the global mask for new clients with diverse data distributions. SparsyFed is agnostic to the choice of the outer optimizer, treating sparse model updates as pseudo-gradients. It is also compatible with biased client selection policies, allowing training masks to adapt to the training data utilized. Our work’s contributions are: 1. We introduce SparsyFed , a method which accelerates on-device training in FL. SparsyFed achieves high sparsity (up to 95\% ) without sacrificing accuracy through a novel approach combining hyperparameter-free activation pruning and weight re-parameterization. 2. We compare SparsyFed against the latest state-of-the-art techniques, demonstrating superior accuracy over sparse training baselines (including those using fixed sparse masks) and, in some cases, surpassing non-pruned baselines at extreme sparsity levels. 3. SparsyFed quickly achieves consensus on client sparse model masks, enabling faster global convergence and significantly reducing downlink communication costs (up to 19.29\times ). 4. We evaluate SparsyFed with ablation studies on typical cross-device FL datasets, including CIFAR-10/100 ( Krizhevsky, 2012 ) and Speech Commands ( Warden, 2018 ) , under various data heterogeneity conditions. We provide the developed code publicly available in this repository to facilitate result reproducibility and for the community of researchers in the field. 2 Background In the following, we describe

Read the complete text