FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learning
Chaimaa MEDJADJI, Sadi ALAWADI, Feras M. Awaysheh, Guilain LEDUC, Sylvain KUBLER, Yves Le Traon · 2025
Read FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learning by Chaimaa MEDJADJI, Sadi ALAWADI, Feras M. Awaysheh, Guilain LEDUC, Sylvain KUBLER, Yves Le Traon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05591.txt.
Introduction
Chaimaa Medjadji and co-authors present FedSparQ, a way to shrink the traffic in federated learning, where many devices train one model without sharing raw data. The method cuts the heavy flow of model updates that such training sends over weak networks. It needs no hand tuning of its settings.
FedSparQ drops small gradient values using a threshold that adapts on its own, stores what remains at half precision, and feeds the leftover error back so nothing is lost. In vision tests it cut bytes sent by about ninety percent versus a standard baseline. Accuracy held or rose, and training stayed steadier, which the authors say suits real deployments on tight bandwidth.
Catalog record
| CREATOR | Chaimaa MEDJADJI, Sadi ALAWADI, Feras M. Awaysheh, Guilain LEDUC, Sylvain KUBLER, Yves Le Traon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-05591
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learning Chaimaa MEDJADJI, Sadi ALAWADI, Feras M. Awaysheh, Guilain LEDUC, Sylvain KUBLER, Yves Le Traon Abstract Federated Learning (FL) enables collaborative model training across decentralized clients while preserving data privacy by keeping raw data local. However, FL suffers from significant communication overhead due to the frequent exchange of high-dimensional model updates over constrained networks. In this paper, we present FedSparQ , a lightweight compression framework that dynamically sparsifies the gradient of each client through an adaptive threshold, applies half-precision quantization to retained entries and integrates residuals from error feedback to prevent loss of information. FedSparQ requires no manual tuning of sparsity rates or quantization schedules, adapts seamlessly to both homogeneous and heterogeneous data distributions, and is agnostic to model architecture. Through extensive empirical evaluation on vision benchmarks under independent and identically distributed (IID) and non-IID data, we show that FedSparQ substantially reduces communication overhead (reducing by 90% of bytes sent compared to FedAvg) while preserving or improving model accuracy (improving by 6% compared to FedAvg non-compressed solution or to state-of-the-art compression models) and enhancing convergence robustness (by 50%, compared to the other baselines). Our approach provides a practical, easy-to-deploy solution for bandwidth-constrained federated deployments and lays the groundwork for future extensions in adaptive precision and privacy-preserving protocols. I Introduction Federated Learning (FL) [ 1 ] enables distributed edge nodes (clients and workers) to collaboratively train machine learning models locally, preserving data privacy by sharing only model updates instead of raw data. However, in practical deployments (e.g. mobile phones, IoT sensors, or geographically dispersed edge servers, …), communication bandwidth is severely constrained and often the dominant bottleneck [ 2 ] . Transmitting full-precision gradients or parameters each round can exhaust network quotas, increase latency, and even drain device batteries. Moreover, client-side heterogeneity (in compute power, data distribution, and network connectivity) exacerbates these challenges, since a “one-size-fits-all" compression strategy can either underutilize bandwidth when updates are small or discard important information when updates are large. Fig. 1 : Taxonomy of communication-efficient federated learning (FL). The top node states the goal; the middle layer distinguishes three complementary families: (i) Quantization ; (ii) Sparsification ; (iii) Compression (scheduling/physical-layer) . Arrows to Hybrid Techniques indicate methods that combine multiple techniques to obtain multiplicative savings while preserving convergence. Several approaches exist to mitigate this communication overhead problem. Fig. 1 summarizes the three main approaches: To overcome this limitation, we propose FedSparQ , a unified compression framework that: (1) performs FedProx-regularized local updates to avoid client’s model drift on non-IID data, (2) uses EMA-smoothed, layer-wise thresholds to select exactly the most informative parameters each round, (3) quantizes only those retained updates to float16, halving the per-value byte cost, and (4) uses error-feedback with residual accumulation to compensate for the bias introduced by dropped updates. Our Key contributions are: • Communication-efficient: using an adaptive compression strategy that combines exponentially smoothed, layer-wise sparsification with float16 quantization. This compound approach selects only the most informative gradient coordinates and encodes them with reduced precision, achieving significant bandwidth savings without manual tuning. • Robustness to non-IID data: by integrating FedProx regularization to mitigate client drift and stabilize optimization in the presence of skewed and heterogeneous data distributions, a common challenge in real-world FL settings. • Unbiased convergence via residual feedback: by employing an error-feedback mechanism to accumulate and re-inject discarded gradient information, ensuring convergence remains unbiased despite aggressive gradient pruning. Several experiments have been conducted across various datasets, including MNIST, FashionMNIST, and CIFAR-10. Our results show that FedSparQ achieves up to a 90% reduction in communication overhead without hurting and, in some cases, improving model accuracy across varied non-IID settings and heterogeneous devices. The remainder of this paper is organised as follows: Section II provides a general background about FedAvg, Quantization, Sparsification and error-feedback residual. Section III provides a comprehensive overview of previous research addressing FL communication efficiency challenges. The proposed FedSparQ approach is introduced and detailed in Section IV , including its functional mechanism and theoretical motivation. Section V presents the formal convergence analysis of the proposed approach. Experimental evaluations and found results across three benchmark datasets and six baseline methods are reported in Section VI followed by a discussion of the implications and insights derived from the results, and the main conclusions and future directions are summarized in Section VII . II Background This section introduces FL as a privacy-by-design [ 6 ] and FedAvg, the foundational FL algorithm that works without compression techniques. In addition, we briefly explain key compression techniques such as quantisation, sparsification, and error-feedback, which aim to reduce communication overhead in FL systems. These techniques form the conceptual basis for the proposed FedSparQ approach. II-A FedAvg Consider a federation of K clients, where client k holds a local dataset \mathcal{D}_{k} of size n_{k} , and the total number of examples is N=\sum_{k=1}^{K}n_{k} . Federated learning seeks to minimize the global empirical risk: F(w)\;=\;\frac{1}{N}\sum_{k=1}^{K}\sum_{(x,y)\in\mathcal{D}_{k}}\ell(w;(x,y))\;=\;\sum_{k=1}^{K}\frac{n_{k}}{N}\,F_{k}(w)\,, where F_{k}(w) is client k ’s local loss. The canonical FedAvg [ 1 ] algorithm proceeds in communication rounds t=1,2,\dots . At round t : 1. The server broadcasts the current global model w_{t} to a subset S_{t} of clients. 2. Each client k\in S_{t} performs one or more steps of SGD on its local objective F_{k} , yielding an updated model w_{t}^{k} . 3. Clients return their updates \Delta_{t}^{k}=w_{t}-w_{t}^{k} to the server. 4. The server aggregates: w_{t+1}=w_{t}-\eta\sum_{k\in S_{t}}\frac{n_{k}}{\sum_{j\in S_{t}}n_{j}}\,\Delta_{t}^{k}, where \eta is the global learning rate. While FedAvg preserves data privacy, each round incurs a communication cost of O(d) parameters (32-bits each) per participating client, which can overwhelm bandwidth-constrained devices. II-B Quantization Quantization compresses each gradient or parameter update by reducing its bit-width. A common unbiased quantizer Q_{s}:\mathbb{R}^{d}\to\mathbb{R}^{d} (e.g. QSGD [ 12 ] ) operates coordinate-wise: [Q_{s}(x)]_{i}\;=\;\|x\|_{2}\,\mathrm{sign}(x_{i})\,\xi_{i},\quad\mathbb{E}[\xi_{i}]=\frac{|x_{i}|}{\|x\|_{2}}, ensuring \mathbb{E}[Q_{s}(x)]=x . Mixed-precision quantization simply casts float32 values to float16: Q_{16}(x_{i})=\mathrm{float16}(x_{i}), halving per-coordinate bytes at the cost of added rounding noise. II-C Sparsification Sparsification sends only a subset of coordinates. Two primary schemes are: • Static top- k : retain the k largest-magnitude entries: [S_{k}(x)]_{i}=\begin{cases}x_{i},&i\in\mathrm{TopK}(|x|),\\ 0,&\text{otherwise},\end{cases} guaranteeing exactly k values per update. • Threshold sparsification [ 4 ] : choose a threshold \tau , often \tau=\frac{1}{d}\sum_{i}|x_{i}| , and keep [S_{\tau}(x)]_{i}=x_{i}\,\mathbf{1}\{|x_{i}|\geq\tau\}. This dynamic sparsifier adapts the support size |I(x)| to the gradient distribution each round. Sparsification reduces communication to O(|I|) non-zero entries, but discarding small entries introduces a bias that can slow or even prevent convergence. II-D Error-Feedback Residuals Error-feedback (EF) addresses the bias of compression by accumulating the dropped “residual”: r_{t+1}=x_{t}-C(x_{t})\,,\quad\tilde{x}_{t}=C(x_{t}+r_{t}), where C is a compressor (quantizer or sparsifier). In the next round, r_{t} is added back into the input, ensuring that \sum_{t}\tilde{x}_{t}=\sum_{t}x_{t} in expectation, and restoring the convergence properties of uncompressed SGD [ 4 ] . II-E Hybrid Compression Methods Hybrid FL methods combine sparsification and quantization, often with periodic averaging or control variates. However, these methods typically rely on multiple hyperparameters and do not adapt their compression levels to per-round gradient statistics. FedSparQ builds on these foundations by unifying adaptive threshold sparsification, float16 quantization, and error-feedback residual accumulation into a single, hyperparameter-light framework that automatically balances communication reduction against unbiased convergence. III Related Work To mitigate the communication bottleneck in FL, researchers have developed a spectrum of compression techniques that