Conditioning on Local Statistics for Scalable Heterogeneous Federated Learning
Rickard Brännvall · 2025
Read Conditioning on Local Statistics for Scalable Heterogeneous Federated Learning by Rickard Brännvall (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00378.txt.
Introduction
Rickard Brännvall proposes a privacy-friendly fix for uneven data in federated learning, in a 2025 paper. In federated learning, many clients train one model together without sharing their data. When each client's data looks different, the shared model can wobble and converge poorly.
His fix lets each client compute summary statistics of its own data, such as averages and spreads, and keep them private. During training, the model learns to adjust itself based on those local summaries. At use time, the same summaries steer each client's predictions. Experiments show the approach handles uneven data well and scales better than methods that hunt for similar peers. No extra information leaves the client, so privacy holds while accuracy improves.
Catalog record
| CREATOR | Rickard Brännvall |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00378
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Conditioning on Local Statistics for Scalable Heterogeneous Federated Learning Rickard Brännvall Abstract Federated learning is a distributed machine learning approach where multiple clients collaboratively train a model without sharing their local data, which contributes to preserving privacy. A challenge in federated learning is managing heterogeneous data distributions across clients, which can hinder model convergence and performance due to the need for the global model to generalize well across diverse local datasets. We propose to use local characteristic statistics, by which we mean some statistical properties calculated independently by each client using only their local training dataset. These statistics, such as means, covariances, and higher moments, are used to capture the characteristics of the local data distribution. They are not shared with other clients or a central node. During training, these local statistics help the model learn how to condition on the local data distribution, and during inference, they guide the client’s predictions. Our experiments show that this approach allows for efficient handling of heterogeneous data across the federation, has favorable scaling compared to approaches that directly try to identify peer nodes that share distribution characteristics, and maintains privacy as no additional information needs to be communicated. Table 1: Performance comparison of conditional models with reference models on three tasks. \csvautobooktabular Analysis/tables/comparison_3.csv 1 Introduction. Figure 1: Conditional CNN performs better on similar characters compared to global and client unconditional reference models. Its accuracy is at par with the cluster oracle model. To address heterogeneous data distributions in federated learning, we propose conditioning the model on local statistics that characterize each client’s joint data distribution, estimated from its own training data. Our approach relates to Personalized Federated Learning (PFL), which uses meta-learning and fine-tuning to tailor the global model to each client’s data. However, PFL can be computationally intensive, lead to overfitting, and increase communication overhead [ 2 , 6 ] . Clustered Federated Learning (CFL) groups clients based on similar data distributions, allowing each cluster to train a specialized model. While this improves performance, it poses challenges in determining optimal clusters and adds communication and computation overhead [ 8 , 4 ] . Unlike PFL and CFL, our approach requires no modifications to the aggregation process and does not increase communication overhead. The set-up consists of three stages: 1. Preparation: Each client calculates local statistics independently using their own training data. Clients have agreed on a method, but the resulting statistics are not shared. 2. Training: During the training phase of federated learning, each client feeds in their own local statistics as input to the model in parallel to the other training data so that the model can learn how to condition on the local data distribution. FedAvg or FedSGD can be used. 3. Inference: The local client uses its own static local characteristics to guide its predictions in the inference phase so that they are tailored to the specific data distribution of each client. As many multivariate distributions are uniquely determined by their moments, we propose to use means, covariances, and higher moments to characterize the local joint distribution of features and labels. We also consider compressed statistics, e.g., by Principal Component Analysis. 2 Preliminaries Moments in statistics, such as the mean (first moment), variance (second moment), skewness (third moment), and kurtosis (fourth moment), are quantitative measures related to the shape of a distribution’s probability density function. A result from multivariate statistics holds that many multivariate distributions are uniquely determined by their moments. This property aids in statistical estimation and hypothesis testing by allowing distributions to be characterized and compared based on their moments. For instance, the multivariate normal distribution is fully specified by its mean vector and covariance matrix (first and second moments). Principal Component Analysis (PCA) is related to moments, particularly the second moment (covariance), as it transforms data into a new coordinate system where the greatest variances lie on the principal components. It is achieved through eigenvalue decomposition of the covariance matrix and reduces dimensionality while preserving the essential dependence structure. Federated learning is a distributed machine learning approach where multiple clients collaboratively train a model without sharing their local data, preserving privacy. This involves initializing a global model, performing local training on each client, and aggregating updates on a central server using techniques like federated averaging (FedAvg) [ 7 ] . A significant challenge is dealing with heterogeneous data distributions across clients, leading to issues in model convergence and performance. This heterogeneity includes covariate shift, label shift, and concept shift [ 3 ] . Personalized Federated Learning (PFL) leverages Model-Agnostic Meta-Learning (MAML) [ 2 ] to tailor the global model to each client’s data. Clustered Federated Learning (CFL) groups clients into clusters based on data similarity, allowing each cluster to train a specialized model, improving performance and handling non-convex objectives [ 8 ] . The Iterative Federated Clustering Algorithm (IFCA) alternately estimates cluster identities and optimizes model parameters for user clusters, demonstrating efficient convergence even with non-convex problems [ 4 ] . Recent advancements include PyramidFL, [ 5 ] , a client selection framework that fully exploits data and system heterogeneity within selected clients, and FedGH [ 9 ] , which focuses on sharing a generalized global header and training it with local average representations. Similarly, [ 1 ] introduced FedPAC, which aligns local representations to the global feature centroid. 3 Experiments 3.1 Synthetic Tasks We assumed a set-up of multiple clusters, which each has multiple clients sharing the same data distribution. We generated synthetic data to evaluate the performance of our method in handling heterogeneous data distributions. For linear regression, the feature vectors X were drawn from a multivariate normal distribution. The true regression coefficients \theta were drawn from a multivariate uniform distribution [-10,10] independently for each cluster and then shared among clusters. True labels for the regression problem were obtained from y=X^{T}\theta+\epsilon , adding a small noise \epsilon\sim N(0,0.1) . For logistic regression, we created binary classification data with varying class distributions, where the true labels were obtained by thresholding X^{T}\theta at zero. We evaluated three models that incorporate local statistics: a conditional linear model that combines features x and local node stats \mu by matrix multiplication (Model 1); an ensemble of regression models on x weighted by a softmax function that depends on \mu_{i} (Model 2); and a fully connected neural network (multi-level perceptron) that takes x and \mu_{i} as input (Model 3), where i denotes the client that evaluates the method. Model 1 (Conditional linear) . \hat{y}=x^{T}W\mu_{i} (1) Model 2 (Ensemble regression) . u=\mu^{T}_{i}W_{u}\\ (2) v=x^{T}W_{v}\\ (3) \hat{y}=v^{T}\mathrm{softmax}(u) (4) Model 3 (Multi-level perceptron) . \hat{y}=\mathrm{MLP}(x,\mu_{i};\eta) (5) Here W , W_{u} , W_{v} , and \eta are the parameters of the models that must be learned. These are all global models, in the sense that all clients learn the same model with the same parameters – only \mu_{i} differ between clients and was calculated as the covariance between X and y on the (local) training data in our experiments. For comparison, we also train three conventional regressions models, \hat{y}=x^{T}\beta , where the regression weights \beta were globally fitted to data from all clients, fitted separately to data from clients belonging to the same cluster, or fitted individually to each local client. We denote