Information-Geometric Barycenters for Bayesian Federated Learning

Nour Jamoussi 2 , Giuseppe Serra 2 , Photios A. Stavrou 2 , Marios Kountouris 2 3 · 2024

Read Information-Geometric Barycenters for Bayesian Federated Learning by Nour Jamoussi 2 , Giuseppe Serra 2 , Photios A. Stavrou 2 , Marios Kountouris 2 3 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11646.txt.

Introduction

Nour Jamoussi, Giuseppe Serra, Photios Stavrou, and Marios Kountouris rethink federated learning through geometry in a 2024 paper. Federated learning trains models across many clients and merges them by averaging. Bayesian inference, though, treats models as distributions, not points. Averaging points may not fit that space well.

The authors recast merging as finding the barycenter, or center point, of local posterior distributions under a chosen distance. That view unifies many past methods, they show. Their algorithm keeps the convergence traits of standard federated averaging. Tests under uneven client data match leading statistical methods, with a clearer geometric story.

Catalog record

CREATORNour Jamoussi 2 , Giuseppe Serra 2 , Photios A. Stavrou 2 , Marios Kountouris 2 3
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Information-Geometric Barycenters for Bayesian Federated Learning Nour Jamoussi 2 , Giuseppe Serra 2 , Photios A. Stavrou 2 , Marios Kountouris 2 3 Abstract Federated learning (FL) is a widely used and impactful distributed optimization framework that achieves consensus by averaging locally trained models. While effective, this approach may not align well with Bayesian inference, where the model space is more naturally represented as a distribution space. Taking an information-geometric perspective, we reinterpret FL aggregation as the problem of finding the barycenter of local posteriors using a predefined divergence metric, minimizing the average discrepancy across clients. This perspective provides a unifying framework that generalizes many existing methods and offers crisp insights into their theoretical underpinnings. We then propose BA-BFL, an algorithm that retains the convergence properties of Federated Averaging in non-convex settings. In non-independent and identically distributed scenarios, we conduct extensive comparisons with statistical aggregation techniques, showing that BA-BFL achieves performance comparable to state-of-the-art methods while also providing a geometric interpretation of the aggregation phase. Additionally, we extend our analysis to Hybrid Bayesian Deep Learning, exploring the impact of Bayesian layers on uncertainty quantification and model calibration. Fig. 1 : The BA-BFL framework. Deterministic Arithmetic Average Aggregations FedAvg [ 1 ] , FedProx [ 2 ] , PropFair [ 3 ] , GiFair [ 4 ] , FedLaw [ 5 ] Multiplicative Aggregations of Posteriors FedPA [ 6 ] , FOLA [ 7 ] , FedEP [ 8 ] , FedIvon [ 9 ] WB / RKLB with variance \to 0 RKLB Fig. 2 : Mapping of various aggregation methods to their corresponding barycenter formulations. I Introduction Federated Learning (FL) has emerged as the de facto standard for decentralized learning, particularly in scenarios that demand strong privacy guarantees. As originally introduced in [ 1 ] , an FL system consists of a central server that maintains a global model and interacts with multiple clients (end-user devices), each of which holds private local data. FL schemes typically operate in two phases. In the local learning phase, each client trains a model on its own private data. In the aggregation phase, the locally updated models are transmitted to the server and merged according to a predetermined rule. These phases repeat iteratively, with the global model from the previous iteration distributed to clients as the starting point for their local models in the current iteration. Aggregation plays a central role in FL, allowing individual client contributions to be combined into a global model while preserving data privacy and ensuring communication efficiency. Although various aggregation strategies have been proposed (e.g., [ 10 ] ), most aggregation methods rely on variants of weighted averaging. Notable examples include FedAvg [ 1 ] and FedProx [ 2 ] , which aggregate local models by computing a weighted average of their parameters. The choice of weights is an important design parameter, as it can encode auxiliary attributes such as the relative importance of each client’s model to the overall objective or reflect the amount of data available to each client. A key challenge in FL, and generally in distributed learning systems, is the statistical heterogeneity among participating clients. In real-world scenarios, client datasets rarely satisfy the idealized assumption of independent and identically distributed (i.i.d.) data. Instead, they often exhibit significant heterogeneity and distributional shifts across clients. As reviewed in [ 11 , 2 ] , five major forms of heterogeneity are typically identified: label distribution skew : differences in the frequency of specific labels across clients (e.g., under- or over-represented classes); feature distribution skew : differences in the feature distributions associated with the same label; concept drift : cases where the same label is associated with different feature distributions across clients; concept shift : cases where identical samples receive different labels from different clients, and quantity skew : differences in the number of data samples held by each client. While these heterogeneities are of high practical relevance, addressing all of them simultaneously remains a significant challenge. As a result, most of the existing FL research is focused on only a subset of these challenges [ 12 , 13 , 14 ] . Given the growing applicability of FL in real-world settings, uncertainty quantification and model calibration are central to building trustworthy and reliable models. Nonetheless, these aspects remain largely underexplored in existing research on deterministic FL. A preliminary study on the topic, with a specific application to healthcare, is presented in [ 15 ] . It provides an overview of various uncertainty quantification methods for deterministic FL, which were later implemented in [ 16 ] . However, it is important to note that the techniques discussed are primarily inspired by Bayesian approaches, such as Bayesian ensembles and Monte Carlo dropout. Bayesian learning excels at improving model reliability, as Bayesian methods enable more accurate uncertainty quantification and calibration, making them a compelling solution for FL in non-i.i.d. contexts. FedPPD [ 17 ] introduces an FL framework with built-in uncertainty quantification: in each round, each client estimates both the posterior distribution over its parameters and the posterior predictive distribution (PPD). The PPD is subsequently distilled into a single deep neural network, which is then sent to the server. pFedBayes [ 18 ] and Fedpop [ 19 ] are also Bayesian approaches with a focus on uncertainty quantification aspects, proposed in the context of personalized Bayesian Federated Learning (BFL). Nonetheless, we emphasize that many of the existing methods [ 18 , 20 , 17 , 21 ] rely on variations of weighted averaging of the posteriors’ parameters. Contributions This work introduces a unifying perspective on aggregation methods in BFL through the lens of barycentric aggregation (BA-BFL). Given a divergence metric, we interpret the aggregation process as a geometric problem, where the global model is identified as the barycenter of the local posteriors. Unlike existing methods that often rely on heuristic variations of parameter averaging, our approach is theoretically grounded: it explicitly minimizes the average divergence between the global posterior and the local posteriors. We show that this methodology generalizes several aggregation strategies previously proposed in the literature. Furthermore, the proposed methods preserve the convergence properties of FedAvg, even in non-convex settings (see Theorem III.1 ). We evaluate our approach against state-of-the-art Bayesian aggregation methods, comparing accuracy and uncertainty quantification in heterogeneous settings. The results demonstrate that our method achieves performance comparable to existing statistical aggregation techniques. To bridge gaps in the BFL literature and buildinsights from Hybrid Bayesian Deep Learning (HBDL) [ 22 ] , we further examine how limiting the number of Bayesian layers affects the performance of different Bayesian aggregation methods. Notation Table I summarizes the key symbols and their meanings for clarity and convenience of reference. Symbol Meaning \mathcal{D}_{k} Local dataset of client k \mathcal{D} Union of all datasets \bigcup_{k=1}^{N}\mathcal{D}_{k} D_{\alpha} \alpha -divergence \KL Kullback–Leibler divergence \W 2 Squared 2-Wasserstein distance w_{k} Weight of client k f_{k} Local objective of client k f Weighted sum \sum_{k=1}^{N}w_{k}f_{k} \psi_{k} Posterior parameters of client k ’s model \psi_{g} Posterior parameters of the global model TABLE I: Summary of notation. II Background and Related Work Federated Learning An FL system [ 1 ] consists of a central server and N clients that engage in an iterative learning process through server-client communication. For

Read the complete text