Connecting Federated ADMM to Bayes

Siddharth Swaroop, Mohammad Emtiyaz Khan, Finale Doshi-Velez · 2025

Read Connecting Federated ADMM to Bayes by Siddharth Swaroop, Mohammad Emtiyaz Khan, Finale Doshi-Velez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17325.txt.

Introduction

Siddharth Swaroop, Mohammad Emtiyaz Khan, and Finale Doshi-Velez uncover a hidden link between two federated learning methods long seen as unrelated, in a paper from 2025. Federated learning trains one model across many sites that keep their data local. One camp uses an alternating optimization scheme; the other uses a Bayesian approximation.

The team shows the helper variables of the first method are exactly the site parameters of the second, under simple spread assumptions. That bridge yields two new hybrid methods, one with flexible spread and one with a functional penalty. Tests confirm the hybrids perform better than the parents. The work merges two fields thought to be fundamentally apart and improves learning across sites.

Catalog record

CREATORSiddharth Swaroop, Mohammad Emtiyaz Khan, Finale Doshi-Velez
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-17325

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Connecting Federated ADMM to Bayes Siddharth Swaroop, Mohammad Emtiyaz Khan, Finale Doshi-Velez Abstract We provide new connections between two distinct federated learning approaches based on (i) ADMM and (ii) Variational Bayes (VB), and propose new variants by combining their complementary strengths. Specifically, we show that the dual variables in ADMM naturally emerge through the “site” parameters used in VB with isotropic Gaussian covariances. Using this, we derive two versions of ADMM from VB that use flexible covariances and functional regularisation, respectively. Through numerical experiments, we validate the improvements obtained in performance. The work shows connection between two fields that are believed to be fundamentally different and combines them to improve federated learning. 1 Introduction The goal of federated learning is to train a global model in the central server by using the data distributed over many local clients ( McMahan et al., 2016 ) . Such distributed learning improves privacy, security, and robustness, but is challenging due to frequent communication needed to synchronise training among nodes. This is especially true when the data quality differs drastically from client to client and needs to be appropriately weighted. Designing new methods to deal with such challenges is an active area of research in federated learning. We focus on two distinct federated-learning approaches based on the Alternating Direction Method of Multipliers (ADMM) and Variational Bayes (VB), respectively. The ADMM approach synchronises the global and local models by using constrained optimisation and updates both primal and dual variables simultaneously. This includes methods like FedPD ( Zhang et al., 2021 ) , FedADMM ( Gong et al., 2022 ; Wang et al., 2022 ; Zhou and Li, 2023 ) and FedDyn ( Acar et al., 2021 ) . The VB approach, on the other hand, uses local posterior distributions as messages and multiplies them to compute an accurate estimate of the global posterior ( Ashman et al., 2022 ) . This follows the more general Bayesian framework which has a long history in distributed and decentralised computations ( Mutambara, 1998 ; Tresp, 2000 ; Durrant-Whyte, 2001 ) . Despite decades of work, the two fields remain disconnected and are thought to be fundamentally different: there is no notion of duality in VB, and posterior distributions are entirely absent in ADMM. Little has been done to connect the two at a fundamental level and our goal here is to address this gap. In this paper, we provide new connections between ADMM and VB-based approaches for federated learning. Our main result shows that the dual-variables used in ADMM naturally emerge through the “site” parameters in VB. We show this for a specific case where isotropic-covariance Gaussian approximations are used. For this case, we get a close line-by-line correspondence between the ADMM and VB updates. The expectations of the local likelihood approximations, also known as the sites, yield the dual terms used in the local updates of ADMM. We use this connection to derive new federated learning algorithms. First, we extend the isotropic case to learn full-covariances through the federated VB approach; see Sec. 3.3 . This leads to an ADMM-like approach with an additional dual variable to learn covariances, which are then used as preconditioners for the global update. Second, building upon functional regularisation approaches from Bayesian literature, we propose to use similar extensions to handle unlabelled data, providing a relationship to federated distillation methods ( Seo et al., 2020 ; Li and Fedmd, 2019 ; Lin et al., 2020a ; Seo et al., 2020 ; Wu et al., 2021 ) , as well as work in function-space continual learning ( Buzzega et al., 2020 ; Kirkpatrick et al., 2017 ; Pan et al., 2020 ; Titsias et al., 2020 ) ; see Sec. 3.4 . Our empirical results in Sec. 4 show that the new algorithms lead to improved performance (improved convergence) across a range of different models, datasets, numbers of clients, and dataset heterogeneity levels. 2 Federated Learning with ADMM and Bayes We start by introducing federated ADMM and Bayesian federated learning approaches, and then, in Secs. 3 and 4 , we will provide new connections along with new algorithms and empirical results. The goal of federated learning is to train a global model (parameters denoted by \bm{w}_{g} ) by using the data distributed across K clients. Ideally, we want to recover the solution that minimises the loss over all the aggregated data, for instance, we may aim for \bm{w}_{g}^{*}=\arg\min_{\bm{w}}\,\,\sum_{k=1}^{K}\ell_{k}(\bm{w})+\delta\mathcal{R}(\bm{w}), (1) where \ell_{k} is the loss defined at the k ’th client using its own local data \mathcal{D}_{k} and \mathcal{R} is a regulariser with \delta\geq 0 as a scalar (set to 0 when no explicit regularisation is used). However, because the data is distributed, the server needs to query the clients to get access to the losses \ell_{k} . Then the goal is to estimate \bm{w}_{g}^{*} while also minimising the communication cost. Typically, instead of communicating \ell_{k} or its gradients, local clients train their own models (denoted by \bm{w}_{k} for the k ’th client) and communicate them (or changes to them) to the server. The server then updates \bm{w}_{g} and sends it back to the clients. This process is iterated with the aim of eventually recovering \bm{w}_{g}^{*} . The Alternating Direction Method of Multipliers (ADMM) is a popular framework for distributed and federated learning, where the local and global optimisation are performed with constraints that they must converge to the same model. That is, we use constraints \bm{w}_{k}=\bm{w}_{g} enforced through the dual variables, denoted by \bm{v}_{k} . The triple (\bm{w}_{k},\bm{v}_{k},\bm{w}_{g}) are updated as follows, \begin{split}\text{Client updates:}\quad\bm{w}_{k}&\leftarrow\underset{\bm{w}}{\text{argmin}}\;\;\bar{\ell}_{k}(\bm{w})+\bm{v}_{k}^{\top}\bm{w}+\mbox{$\frac{1}{2}$}\alpha\|\bm{w}-\bm{w}_{g}\|^{2},\\ \quad\bm{v}_{k}&\leftarrow\bm{v}_{k}+\alpha(\bm{w}_{k}-\bm{w}_{g}),\quad\text{ for all }k,\\ \text{Server update:}\quad\bm{w}_{g}&\leftarrow\frac{1}{K}\sum_{k=1}^{K}\left[\bm{w}_{k}+\frac{1}{\alpha}\bm{v}_{k}\right].\end{split} (2) Here, \alpha>0 is a scalar and we denote \bar{\ell}_{k}=\ell_{k}/N_{k} where N_{k} is the size of the data at the k ’th client. Upon convergence, we get \bm{w}_{g}^{*}=\bm{w}_{k}^{*} and \bm{v}_{k}^{*}=-\nabla\bar{\ell}_{k}(\bm{w}_{g}^{*}) for all k . Such updates are instances of primal-dual algorithms and many variants have been proposed for federated learning ( Zhang et al., 2021 ; Gong et al., 2022 ; Wang et al., 2022 ; Zhou and Li, 2023 ) . For example, FedDyn ( Acar et al., 2021 ) is perhaps the best-performing variant which uses a client-specific \alpha_{k}\propto\alpha/N_{k} . It also adds an additional hyperparameter through a local weight-decay regulariser added to each local client’s loss. There are also simpler versions (not based on ADMM), for instance, FedProx ( Li et al., 2020 ) where \bm{v}_{k} is not used (or equivalently fixed to 0 in Eq. 2 ), and even simpler Federated Averaging or FedAvg ( McMahan et al., 2016 ) , where the proximal constraint \bm{w}-\bm{w}_{g} is also removed. Such methods do not ensure that \bm{w}_{k}^{*}=\bm{w}_{g}^{*} at convergence. The use of dual-variables is a unique feature of ADMM to synchronise the server and clients. The Bayesian framework for federated learning takes a very different approach to ADMM. Instead of a point estimate \bm{w}_{g}^{*} , the goal for Bayes is to target a global posterior p_{g}(\bm{w})=p(\bm{w}|\mathcal{D}_{1},\mathcal{D}_{2},\ldots,\mathcal{D}_{K}) . Instead of loss functions for each \mathcal{D}_{k} , we use likelihoods denoted by p(\mathcal{D}_{k}|\bm{w}) . The solution \bm{w}_{g}^{*} in Eq. 1 can be seen as the mode of the posterior p_{g} whenever there exists likelihood such that \log p(\mathcal{D}_{k}|\bm{w})=-\ell_{k}(\bm{w}) and a prior \log p_{0}(\bm{w})=-\delta\mathcal{R}(\bm{w}) , where both equations need to be true only

Read the complete text