A Thorough Assessment of the Non-IID Data Impact in Federated Learning

Daniel M. Jimenez-Gutierrez, Mehrdad Hassanzadeh, Aris Anagnostopoulos, Ioannis Chatzigiannakis, Andrea Vitaletti · 2025

Read A Thorough Assessment of the Non-IID Data Impact in Federated Learning by Daniel M. Jimenez-Gutierrez, Mehrdad Hassanzadeh, Aris Anagnostopoulos, Ioannis Chatzigiannakis, Andrea Vitaletti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17070.txt.

Introduction

Daniel Jimenez-Gutierrez and four co-authors measure how non-IID data harms federated learning. In federated learning, many clients train a shared model while keeping their data private. Client data are rarely alike, and that mismatch can cut accuracy and slow convergence. The team uses the Hellinger Distance to gauge how far client distributions diverge.

Their tests benchmark four leading strategies under label, feature, quantity, and spatiotemporal skewness. The study is the first full analysis of spatiotemporal skew in this setting. Label and spatiotemporal skew hurt most, with sharp drops at certain distance thresholds. Extreme mismatch does the worst damage, and the authors give practical advice for handling it.

Catalog record

CREATORDaniel M. Jimenez-Gutierrez, Mehrdad Hassanzadeh, Aris Anagnostopoulos, Ioannis Chatzigiannakis, Andrea Vitaletti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17070

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Thorough Assessment of the Non-IID Data Impact in Federated Learning Daniel M. Jimenez-Gutierrez, Mehrdad Hassanzadeh, Aris Anagnostopoulos, Ioannis Chatzigiannakis, Andrea Vitaletti Abstract Federated learning (FL) allows collaborative machine learning (ML) model training among decentralized clients’ information, ensuring data privacy. The decentralized nature of FL deals with non-independent and identically distributed (non-IID) data. This open problem has notable consequences, such as decreased model performance and longer convergence times. Despite its importance, experimental studies systematically addressing all types of data heterogeneity (a.k.a. non-IIDness) remain scarce. This paper aims to fill this gap by assessing and quantifying the non-IID effect through an empirical analysis. We use the Hellinger Distance ( HD ) to measure differences in distribution among clients. Our study benchmarks five state-of-the-art strategies for handling non-IID data, including label, feature, quantity, and spatiotemporal skews, under realistic and controlled conditions. This is the first comprehensive analysis of the spatiotemporal skew effect in FL. Our findings highlight the significant impact of label and spatiotemporal skew non-IID types on FL model performance, with notable performance drops occurring at specific HD thresholds. The FL performance is also heavily affected, mainly when the non-IIDness is extreme. Thus, we provide recommendations for FL research to tackle data heterogeneity effectively. Our work represents the most extensive examination of non-IIDness in FL, offering a robust foundation for future research. † † highlights: Label skew and spatiotemporal skew have the most significant impact on the model’s performance. The drop in the model’s performance for label skew appears in a double threshold. A notable performance decline is immediately evident when the Hellinger Distance exceeds 0.5 and 0.75. Feature skew does not alter the model’s performance nor the convergence point. The quantity skew in the client’s data does not affect the model’s performance. The higher the non-IIDness level in time and space (spatiotemporal skew), the worse the model’s performance. keywords Federated Learning ,Machine Learning ,Non-IID data ,Data heterogeneity quantification ,Spatiotemporal skew 1 Introduction In today’s digital age, the interaction of machine learning (ML) and healthcare or financial data holds immense promise for improving disease diagnosis [ 57 ] and combating financial crimes [ 55 ] . These advancements have traditionally relied on centralized learning (CL), such as Machine Learning as a Service (MLaaS) platforms—including AWS, Azure, and Google Cloud [ 2 ] —where raw data is aggregated on a central server for model training. . However, it raises critical questions about data privacy when dealing with sensitive information from hospitals or banks. In this context, trusted research environments emerge as a mechanism for balancing ML research and protecting individual privacy [ 23 , 80 ] . Federated learning (FL) [ 47 ] has emerged as a transformative approach for training ML models across decentralized data sources, preserving data privacy and security. This paradigm is particularly beneficial in cross-silo settings, where entities such as hospitals, banks, and other organizations collaborate without sharing sensitive data. However, a significant challenge inherent in FL is the variation in data distributions across clients, referred to as non-IID (Non-Independent and Identically Distributed) data. This non-IID data (i.e., non-IIDness, data heterogeneity) hinders model performance and convergence during training [ 66 , 48 ] . Such non-IID data is classified into four categories: label, feature, quantity, and spatiotemporal skew [ 83 ] . Spatiotemporal skew presents unique challenges that are particularly critical yet underexplored in FL research. This skew occurs when data distributions vary across both geographical locations (spatial) and periods (temporal) [ 15 ] . Such skew fundamentally differs from the label, feature, or quantity skew by introducing dynamic variations that standard FL aggregation algorithms often fail to address [ 4 ] . Understanding spatiotemporal skew is crucial because it directly impacts the model’s ability to generalize across diverse real-world environments while maintaining temporal relevance, making it a key frontier for robust FL systems. Furthermore, diagnosing and quantifying the level of non-IID data in FL is a significant challenge, as emphasized by Pei et al. [ 56 ] and Li et al. [ 41 ] , who identify critical research directions in this domain. Numerous studies have introduced metrics to quantify the level of non-IID data in FL [ 33 , 19 , 52 , 67 ] , with the Hellinger Distance ( HD ) [ 22 ] emerging as one of the most reliable options. HD = 0.0 corresponds to fully IID data, while higher values (e.g., 0.25, 0.5, 0.75, 0.9) represent increasing degrees of non-IID data, with HD = 0.9 approaching the most non-IID scenario considered in our study. HD offers a fine-grained measurement of distribution differences, achieving values close to 1 under extreme non-IID conditions, unlike the Jensen-Shannon Distance (JSD), which tends to plateau at lower levels. Furthermore, HD is versatile and applicable across various types of skews. Motivation. Recent advances in FL research have significantly advanced our understanding of non-IID data challenges, with notable progress in addressing isolated aspects of heterogeneity such as label skew [ 64 , 69 , 49 ] . Based on this foundation, there is now a timely opportunity to unify these insights through comprehensive empirical benchmarks that span the full spectrum of non-IID skews. While pioneering theoretical frameworks [ 44 , 56 ] have established critical mitigation principles, translating these into practice requires systematic quantification of how diverse non-IID data types—from feature skew to spatiotemporal drift—affect real-world FL performance metrics. Closing this knowledge gap through rigorous experimental analysis will empower the community to develop FL systems that are theoretically sound and empirically robust across application domains. Our study addresses this gap by using the HD to quantify differences in client data distributions, enabling a rigorous empirical analysis of non-IID effects across multiple dimensions, including label, feature, quantity, and spatiotemporal skews. Throughout the assessment of spatiotemporal skew, we capture the impact of dynamic data shifts over time and space, which are particularly relevant in real-world applications such as banking credit risk [ 81 ] and personalized healthcare [ 25 ] . This approach ensures that our conclusions are robust and generalizable across diverse scenarios. Contribution. The subsequent points encapsulate the contributions of our study: 1. We benchmark five of the most employed state-of-the-art aggregation and client selection of FL algorithms to tackle non-IID data distributions among clients under realistic, controlled, and quantifiable methods for synthetic data partitioning and all non-IID types (label, feature, quantity, and spatiotemporal skews). Previous empirical works have focused on label skew [ 64 , 69 , 49 ] or in combinations of label, feature, and quantity skew [ 38 ] (see Section 2.1 for more details). Thus, this is the first study empirically analyzing how the spatiotemporal skew affects the performance of FL models. 2. We motivate using HD to quantify the differences among data distributions, standardizing the guidelines for systematic studies of non-IID data in FL. This is the first work to demonstrate that the effect of the non-IID data is not the same under all levels of heterogeneity. We use HD to quantify differences in distribution as it provides more granular information, and we leave as future work the exploration of other measures; see our section on design insights and opportunities. 3. We provide a reference to researchers about which methods are robust to

Read the complete text