ProFed: a Benchmark for Proximity-based non-IID Federated Learning

Davide Domini, Gianluca Aguzzi, Mirko Viroli · 2025

Read ProFed: a Benchmark for Proximity-based non-IID Federated Learning by Davide Domini, Gianluca Aguzzi, Mirko Viroli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20618.txt.

Introduction

Davide Domini, Gianluca Aguzzi, and Mirko Viroli build a test bed for federated learning, in a 2025 paper. Federated learning trains one model across many devices without gathering their data. It struggles when each device's data looks different, which often happens along geographic lines, since nearby devices share language or traffic patterns.

Existing tests usually scatter the differences at random, ignoring that real skew follows maps. Their benchmark, ProFed, splits four well-known image datasets with graded levels of regional skew drawn from published methods. The point is a shared yardstick: researchers can run their algorithms against the same skewed splits and compare with known baselines fairly. The authors aim to steer the field toward tests that honor how data is truly spread across places.

Catalog record

CREATORDavide Domini, Gianluca Aguzzi, Mirko Viroli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20618

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ProFed: a Benchmark for Proximity-based non-IID Federated Learning Davide Domini, Gianluca Aguzzi, Mirko Viroli Abstract In recent years, fl ( fl ) has gained significant attention within the machine learning community. Although various fl algorithms have been proposed in the literature, their performance often degrades when data across clients is non-independently and identically distributed (non-IID). This skewness in data distribution often emerges from geographic patterns, with notable examples including regional linguistic variations in text data or localized traffic patterns in urban environments. Such scenarios result in IID data within specific regions but non-IID data across regions. However, existing fl algorithms are typically evaluated by randomly splitting non-IID data across devices, disregarding their spatial distribution. To address this gap, we introduce ProFed, a benchmark that simulates data splits with varying degrees of skewness across different regions. We incorporate several skewness methods from the literature and apply them to well-known datasets, including MNIST, FashionMNIST, CIFAR-10, and CIFAR-100. Our goal is to provide researchers with a standardized framework to evaluate FL algorithms more effectively and consistently against established baselines. I Introduction fl ( fl ) has gained significant interests in the last years. It has been introduced to address privacy problems during learning from users data. In fact, this framework allows the training of a shared global model without the need of collecting the data on a central server. Different studies [ 1 , 2 ] have shown that while fl achieve good learning performance compared to classical learning approaches on homogeneously distributed data, it drops performance when data are non-independently and identically distributed (Non-IID). For instance, in urban traffic prediction scenarios, data patterns exhibit strong spatial correlations: traffic flows within specific city districts often share common characteristics, while differing significantly from patterns observed in other areas. This geographical dependency implies that models trained on data from one district typically achieve higher accuracy when predicting traffic patterns within the same area compared to predictions in different districts. In the literature, various algorithms–like Scaffold [ 3 ] and FedProx [ 4 ] –have been proposed to tackle data heterogeneity. These approaches typically assume that client data are distributed without considering specific patterns or structures. However, in real world scenarios, particularly in the case of highly distributed systems [ 5 ] (e.g., in edge computing or spatial-aware scenarios), it is common for data from geographically close devices to be more similar to each other compared to data from devices that are farther apart. This phenomenon is driven by the fact that devices in the same region often experience similar environments and make comparable observations in the above described scenarios [ 6 ] . Several studies (e.g., [ 7 , 8 , 9 ] ) have attempted to tackle this scenario by proposing algorithms that cluster clients based on similarity metrics, under the assumption that clients within the same cluster have IID data while clusters themselves exhibit Non-IID properties. Nevertheless, the lack of standardized benchmarks for evaluating such approaches remains a significant limitation. Existing benchmarks–like [ 10 , 11 ] –often rely on synthetic data splits or arbitrary partitioning schemes that fail to capture the realistic geographic clustering observed in practice. To bridge this gap, we introduce ProFed 1 1 1 https://github.com/davidedomini/ProFed , a novel benchmark designed specifically for proximity-based Non-IID fl providing a more realistic and complete evaluation setting. ProFed leverages well-known computer vision datasets from PyTorch [ 12 ] and TorchVision [ 13 ] –like MNIST [ 14 ] , CIFAR10 [ 15 ] and CIFAR100 [ 16 ] –and incorporates established data partitioning methods from the literature, such as Dirichlet distribution-based splits [ 17 , 18 ] . Moreover, by enabling researchers to control the degree of data skewness, this approach allows for fine-grained experimentation and analysis. To demonstrate the utility and usability of ProFed, we conducted experiments using three state-of-the-art algorithms, namely: FedAvg [ 19 ] , FedProx [ 4 ] and Scaffold [ 3 ] . The remainder of this paper is organized as follows: Section II provides a background and review of related work. Section III-A formalizes the problem statement. Section III-B details the structure and implementation of the proposed benchmark. Section IV presents an evaluation of our benchmark running experiments with multiple algorithms. Finally, Section V concludes the paper and outlines paths for future research. II Background and Related Work II-A Federated Learning fl [ 19 ] is a machine learning framework that aims at collaboratively train a joint global model from distributed datasets. This technique has been introduced to enable the training of models without the need of collecting and merging multiple datasets into a single central server, thus making it possible to work in contexts with strong privacy concerns—for instance, hospitals [ 20 , 21 ] or banks [ 22 , 23 ] . Moreover, in highly distributed systems, given the large volume of generated data, it becomes inconvenient, or also infeasible, to move all the data to a central server [ 24 ] . In the literature, most fl algorithms share a common learning flow (see Figure 1 ): 1. Model initialization : the central server initializes a common base model that is shared with each client; 2. Local Learning : each client performs one or more steps of local learning on its own dataset; 3. Local models sharing : each client sends back to the central server the new model trained on its own data (i.e. the local model); 4. Local models aggregation : the local models collected by the central server are aggregated to obtain the new global model. This process is carried out iteratively for a predefined number of global rounds. The most common and simple models aggregation method is called FedAvg; it was introduced in [ 19 ] and consists in performing an average of the local models to obtain the next global model. The classical client-server architecture presents some limitations, namely: (i) in federations with many clients, the server may be a bottleneck; (ii) the server is a single point of failure, and if it fails to communicate with the clients, the entire the learning process is interrupted; and (iii) the server must be a trusted entity, which could be a challenging constraint in some scenarios. For these reasons, despite the commonly adopted learning flow, in recent years alternative algorithms based on peer-to-peer or hierarchical networks have also been proposed [ 25 , 26 , 27 ] . Fig. 1 : A graphical representation of the Federated Learning process. The numbers (1-4) present the steps of the learning flow discussed above. II-B Learning from Heterogeneous Data Federated learning algorithms achieve remarkable performance when data are homogeneously distributed among clients [ 28 ] . However, in real-life datasets, data are often heterogeneous, namely non-independently and identically distributed (Non-IID). The skewness of data can be categorized in various ways based on how they are distributed among the clients [ 10 , 29 ] . The main categories include: (i) feature skew : all clients have the same labels but different feature distributions (e.g., in a handwritten text classification task, we may have the same letters written in different calligraphic styles); (ii) label skew : each client has only a subset of the classes;

Read the complete text