Stabilizing Federated Learning under Extreme Heterogeneity with HeteRo-Select
Md. Akmol Masud, Md Abrar Jahin, Mahmud Hasan · 2025
Read Stabilizing Federated Learning under Extreme Heterogeneity with HeteRo-Select by Md. Akmol Masud, Md Abrar Jahin, Mahmud Hasan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06692.txt.
Introduction
Akmol Masud, Abrar Jahin, and Mahmud Hasan propose in 2025 a way to steady federated learning when client data differ wildly. The usual rule hands out compression by link speed, but under skewed data that rule can squeeze the most useful updates hardest. Their framework scores each client by how informative its data is. That score drives who joins each round, how much each update is compressed, and how much weight it gets at the server.
Proofs back the gains, and tests under a fixed protocol showed a near doubling of speed with a clear cut in traffic. The same setup scaled from a tiny model to a large image network. It held up even when fast links carried the least useful data.
Catalog record
| CREATOR | Md. Akmol Masud, Md Abrar Jahin, Mahmud Hasan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06692
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stabilizing Federated Learning under Extreme Heterogeneity with HeteRo-Select Md. Akmol Masud, Md Abrar Jahin, Mahmud Hasan Abstract Federated Learning (FL) is a machine learning technique that often suffers from training instability due to the diverse nature of client data. Although utility-based client selection methods like Oort are used to converge by prioritizing high-loss clients, they frequently experience significant drops in accuracy during later stages of training. We propose a theoretical HeteRo-Select framework designed to maintain high performance and ensure long-term training stability. We provide a theoretical analysis showing that when client data is very different (high heterogeneity), choosing a smart subset of client participation can reduce communication more effectively compared to full participation. Our HeteRo-Select method uses a clear, step-by-step scoring system that considers client usefulness, fairness, update speed, and data variety. It also shows convergence guarantees under strong regularization. Our experimental results on the CIFAR-10 dataset under significant label skew ( α = 0.1 \alpha=0.1 ) support the theoretical findings. The HeteRo-Select method performs better than existing approaches in terms of peak accuracy, final accuracy, and training stability. Specifically, HeteRo-Select achieves a peak accuracy of 74.75%, a final accuracy of 72.76%, and a minimal stability drop of 1.99%. In contrast, Oort records a lower peak accuracy of 73.98%, a final accuracy of 71.25%, and a larger stability drop of 2.73%. The theoretical foundations and empirical performance in our study make HeteRo-Select a reliable solution for real-world heterogeneous FL problems. I Introduction Federated Learning (FL) is an important method in machine learning that enables model training on decentralized datasets while preserving data privacy. It allows multiple clients to collaboratively build a shared model without sharing their local data. However, a major limitation in FL is statistical heterogeneity, where non-independent and identically distributed (non-IID) data across clients leads to conflicting updates and model divergence. This often results in unstable training and presents challenges in achieving consistent performance. Intelligent client selection addresses this issue by choosing a subset of clients in each round who are most likely to contribute positively to the global model. The present study on client selection methods often focuses on immediate utility to accelerate convergence. The authors in [ 1 ] and Oort [ 2 ] studied techniques like Power-of-Choice that prioritize clients with high local loss for their informative updates. Although these methods are effective in the early stages of training, they often suffer from performance instability and struggle to maintain strong performance in later rounds. Such instability shows a critical challenge for real-world applications that require reliable and consistent model performance. To address this, we present the HeteRo-Select framework, designed for long-term stability over short-term gains. Unlike utility-focused methods, HeteRo-Select employs a multi-phase scoring system that evaluates clients on various dimensions. It balances immediate utility with factors like historical performance trends (momentum), participation equity (fairness), and data diversity to ensure a robust selection strategy. This approach prevents overfitting to a narrow subset of high-loss clients, mitigating performance degradation in later training stages and promoting consistent, dependable outcomes. Our main contributions are: 1. We present a theoretical framework that shows intelligent client selection under extreme heterogeneity performs better than full participation in terms of communication complexity and convergence, with proven results. 2. A novel HeteRo-Select framework with a multi-phase additive scoring system, theoretically supported to ensure fairness, diversity, and stability. 3. Formal analysis demonstrates the necessity of strong FedProx regularization ( \mu=0.1 ) in explorative selection policies. 4. Empirical validation on CIFAR-10 under extreme label skew ( \alpha=0.1 ) demonstrates HeteRo-Select’s better performance over Oort in terms of peak accuracy (74.75% vs. 73.98%), final accuracy (72.76% vs. 71.25%), and stability (1.99% vs. 2.73% drop). 5. A comprehensive study that supports theoretical insights into scoring methods, regularization techniques, and hyperparameter effects. The paper is structured as follows: Section II reviews related works. Section III details the HeteRo-Select framework. Section IV outlines the experimental setup, followed by results and analysis in Section V . Section VI concludes with future directions. II Related Works The challenge of communication efficiency in FL, particularly under statistical heterogeneity, has been addressed primarily through two approaches: (1) reducing the size of the updates sent by clients, and (2) intelligently selecting which clients should send updates. Our work focuses on the latter, proposing a novel client selection framework that complements existing methods for managing heterogeneity. II-A Client Selection Strategies The foundational FedAvg algorithm [ 3 ] proposed random client selection, which remains a common baseline. However, subsequent research has demonstrated that more intelligent client selection strategies can lead to significant performance improvements. Early studies have focused on utility-based selection. For instance, the Power-of-Choice approach proposed by the authors in [ 1 ] selects clients with a probability proportional to their local loss, thereby prioritizing those with more to learn. More recently, Oort in [ 2 ] introduced a sophisticated client selection method that balances statistical utility (based on local loss) with system efficiency (e.g., training time), aiming to maximize progress per unit of time. While Oort serves as a strong baseline, our experiments highlight a critical limitation: its emphasis on high-utility clients results in performance instability, failing to match the peak accuracy and long-term stability. This limitation represents a significant practical concern, one that our HeteRo-Select framework is specifically designed to overcome. Recent advances since 2021 have further diversified client selection strategies in Federated Learning. Clustered client selection methods, such as those proposed in [ 4 ] , group clients based on data similarity before selection. Reinforcement learning-based approaches in [ 5 ] learn optimal selection policies through trial-and-error interactions. Additionally, gradient diversity methods aim to select clients with diverse gradient directions to enrich learning dynamics. Other recent studies explore multi-objective optimization techniques that balance competing goals such as model accuracy and fairness during client selection. Key Distinctions of HeteRo-Select : Unlike these approaches, our HeteRo-Select framework uniquely combines: (1) momentum tracking that considers client improvement trends over time, (2) explicit fairness and staleness factors that ensure long-term participation equity, directly mitigating the stability issues observed in utility-focused methods like Oort, (3) dynamic temperature adjustment for a principled exploration-to-exploitation transition, and (4) update norm penalties to prevent destabilizing contributions. Most importantly, our work demonstrates the critical synergy between this sophisticated selection mechanism and strong regularization ( \mu=0.1 ), a factor not explicitly optimized in prior selection strategies. II-B Update Compression and Heterogeneity Many methods have been proposed to reduce the size of model updates, serving as a complementary approach to client selection in Federated Learning. These include techniques like quantization in [ 6 ] , sparsification, and knowledge distillation in [ 7 ] . These approaches focus on how an update is transmitted, whereas our work focuses on who should transmit it. Integrating update compression with intelligent selection could further improve efficiency, an area we leave for future exploration. A significant body of work has analyzed the impact of heterogeneity on FL convergence in [ 8 , 9 ] . Within this context, FedProx in [ 10 ] emerged as a critical method for mitigating client drift by adding a proximal term to the local objective. In our work, FedProx is leveraged not merely as a baseline but as an integral component of our proposed