Hybrid-Regularized Magnitude Pruning for Robust Federated Learning under Covariate Shift
Özgü Göksu, Nicolas Pugeault · 2024
Read Hybrid-Regularized Magnitude Pruning for Robust Federated Learning under Covariate Shift by Özgü Göksu, Nicolas Pugeault (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15010.txt.
Introduction
Özgü Göksu and Nicolas Pugeault tackle uneven data in federated learning in this 2024 paper. In federated learning, many devices train a shared model without pooling their data. When each device's data skews differently, the shared model often learns poorly. The authors show that such skews in how classes look, not just which classes appear, hurt results across test sets.
Their fix prunes weak connections while regularizing training on each client. The aim is a sparser model whose parts combine more cleanly. They also built a new benchmark from celebrity faces, with gender-based shifts inside each class. Across image tests the method beat standard federated baselines and held up better on data unlike the training mix.
Catalog record
| CREATOR | Özgü Göksu, Nicolas Pugeault |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15010
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hybrid-Regularized Magnitude Pruning for Robust Federated Learning under Covariate Shift Özgü Göksu, Nicolas Pugeault Abstract Federated Learning offers a solution for decentralised model training, addressing the difficulties associated with distributed data and privacy in machine learning. However, the fact of data heterogeneity in federated learning frequently hinders the global model’s generalisation, leading to low performance and adaptability to unseen data. This problem is particularly critical for specialised applications such as medical imaging, where both the data and the number of clients are limited. In this paper, we empirically demonstrate that inconsistencies in client-side training distributions substantially degrade the performance of federated learning models across multiple benchmark datasets. We propose a novel FL framework using a combination of pruning and regularisation of clients’ training to improve the sparsity, redundancy, and robustness of neural connections, and thereby the resilience to model aggregation. To address a relatively unexplored dimension of data heterogeneity, we further introduce a novel benchmark dataset, CelebA-Gender, specifically designed to control for within-class distributional shifts across clients based on attribute variations, thereby complementing the predominant focus on inter-class imbalance in prior federated learning research. Comprehensive experiments on many datasets like CIFAR-10, MNIST, and the newly introduced CelebA-Gender dataset demonstrate that our method consistently outperforms standard FL baselines, yielding more robust and generalizable models in heterogeneous settings. I Introduction Federated Learning (FL) offers an efficient paradigm for collaboratively training a shared model across multiple users while preserving client data privacy. Consequently, FL is particularly well-suited for real-world applications where data privacy is critical, such as medical imaging, remote sensing, or processing personal data (eg, faces). FL process is typically initiated by broadcasting an initialised global model from the server to participating clients. Each client independently updates the model using its local data, and the resulting parameters are transmitted back to a central server, where they are aggregated to refine the global model [ 1 ] , [ 2 ] , [ 3 ] . In principle, the global model in FL should outperform local models by leveraging diverse client data for improved generalisation. This assumes that local updates can be aggregated effectively to form a stable and performant global model. However, non-IID (non-independent and identically distributed) data often induces substantial model drift, leading to divergent local optima and impairing the stability and effectiveness of global aggregation. Addressing this challenge is central to advancing the robustness of federated learning in heterogeneous settings. We propose FedMPR ( F ederated Learning with M agnitude P runing and R egularization), a novel framework designed to enhance federated learning in the presence of data heterogeneity. FedMPR promotes robustness in local models by integrating three key components: (1) magnitude-based pruning to eliminate redundant parameters at the client level; (2) dropout to introduce functional redundancy in decision pathways; and (3) noise injection to regularise model responses and improve resilience to weight perturbations during aggregation. We demonstrate that this framework outperforms standard FL approaches on benchmarks, in particular in datasets with large covariate shifts between clients. Additionally, we introduce CelebA-Gender, a novel benchmark dataset for heterogeneous FL. Derived from CelebA [ 4 ] , it is specifically designed to evaluate FL methods under challenging conditions where inter-client distribution shifts arise not only from class imbalance, but also from substantial within-class distribution variations. Client data shift in our study arises from attribute-based gender classification, unlike existing literature [ 5 ] , which typically focuses on examining one attribute at a time for binary classification. CelebA-Gender benchmark provides a more complex data distribution, enabling the evaluation of varying levels of attribute overlap in the image content, ranging from high to low overlap scenarios. Our framework presents several key contributions: • FedMPR : We propose a novel framework for addressing scenarios with significant covariate shifts across clients’ data distributions. • Data Heterogeneity Scenarios: We evaluate FL approaches under different levels of covariate shift, testing both low and high shift scenarios, including varying numbers of clients (both limited and large populations), as well as imbalanced and limited data, to assess the adaptability and robustness of FL methods across diverse settings. • Novel Classification Dataset: We introduce CelebA-Gender, a reconstructed gender classification dataset derived from attribute-based labels, designed to facilitate a comprehensive evaluation of our framework and enable comparison with existing methods. In FL, cross-silo methods typically involve a smaller number of clients or devices (ranging from 2 to 100), compared to cross-device approaches that can involve millions or even billions of devices [ 6 ] . In practice, issues such as data connection loss or slow clients (stragglers) often reduce the effective number of participating clients. This effect is especially pronounced in scenarios with 90 \% or more stragglers, resulting in a drastically reduced subset of active clients, sometimes as few as 10, 100, or even fewer. Combined with data heterogeneity, this further impedes model convergence and degrades performance. In this work, we investigate the effects of straggler behaviour, characterised by a limited subset of participating clients, combined with significant data heterogeneity in federated learning. We emphasise the challenges that arise when only a small fraction of clients are active with highly diverse, non-IID data distributions. II Background II-A Federated Learning FL frameworks such as FedAvg [ 1 ] typically involve three main steps: broadcasting , local training , and model aggregation . After each communication round, the central server broadcasts the current global model to participating clients. Each client then performs local training on its private data, ensuring data privacy by keeping raw data on-device. Once local updates are completed (e.g., after a fixed number of epochs), clients send their updated model parameters back to the server. The server performs model aggregation, typically by averaging the received parameters, to form a new global model, which is then broadcast in the next round. Following the typical FL approach, the data D is distributed across N clients and subset of K clients is randomly selected. Let D_{k} be the local dataset at client k , with n_{k}=|D_{k}| denoting the number of data points at client k . The global objective function in FL is then a weighted average of the local objectives \min_{w}F(w)=\sum_{k=1}^{K}\frac{n_{k}}{n}F_{k}(w) (1) where n=\sum_{k=1}^{K}n_{k} is the total number of data points across clients and F_{k}(w) is the local objective function at a client k \quad F_{k}(w)=\frac{1}{n_{k}}\sum_{i\in\mathcal{D}_{k}}f_{i}(w) (2) Current research predominantly relies on FedAvg-based model aggregation and training procedures [ 7 ] , [ 8 ] [ 9 ] that can broadly be categorised into two main directions: enhancing local training and refining the model aggregation strategy. Our work aligns with the first direction, targeting the local training phase. By improving local optimisation, we aim to reduce divergence across client models, thereby indirectly mitigating challenges typically addressed at the aggregation stage. Fig. 1 : FedMPR framework, each client applies a tailored regularisation that combines dropout during forward passes with Gaussian noise injection inside each basic block. At the i th iteration, the central server broadcasts the global model weights to clients, which then perform local training before model aggregation. Specifically, w denotes the original weights, wi the weights perturbed by Gaussian noise, and wd the zeroed (pruned) weights. II-B Related Work Federated Parameter Selection FL approaches encounter several critical challenges, particularly poor convergence