Task Diversity in Bayesian Federated Learning: Simultaneous Processing of Classification and Regression
Junliang Lyu, Yixuan Zhang, Xiaoling Lu, Feng Zhou · 2024
Read Task Diversity in Bayesian Federated Learning: Simultaneous Processing of Classification and Regression by Junliang Lyu, Yixuan Zhang, Xiaoling Lu, Feng Zhou (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10897.txt.
Introduction
Junliang Lyu and co-authors present a federated learning method that can handle classification and regression tasks at the same time on different devices, in a 2024 paper. Federated learning trains a shared model while data stay on local devices. Most methods assume every device does the same kind of task, which real settings rarely satisfy.
Their design pairs a multi-output Gaussian process on each device with federated pooling at a central server. The process handles linked tasks and reports its own uncertainty in a principled way. Tricks for fast local inference speed the training. On synthetic and real data, the method beat rivals on accuracy, uncertainty fit, and out-of-sample detection.
Catalog record
| CREATOR | Junliang Lyu, Yixuan Zhang, Xiaoling Lu, Feng Zhou |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10897
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Task Diversity in Bayesian Federated Learning: Simultaneous Processing of Classification and Regression Junliang Lyu, Yixuan Zhang, Xiaoling Lu, Feng Zhou Abstract . This work addresses a key limitation in current federated learning approaches, which predominantly focus on homogeneous tasks, neglecting the task diversity on local devices. We propose a principled integration of multi-task learning using multi-output Gaussian processes (MOGP) at the local level and federated learning at the global level. MOGP handles correlated classification and regression tasks, offering a Bayesian non-parametric approach that naturally quantifies uncertainty. The central server aggregates the posteriors from local devices, updating a global MOGP prior redistributed for training local models until convergence. Challenges in performing posterior inference on local devices are addressed through the Pólya-Gamma augmentation technique and mean-field variational inference, enhancing computational efficiency and convergence rate. Experimental results on both synthetic and real data demonstrate superior predictive performance, OOD detection, uncertainty calibration and convergence rate, highlighting the method’s potential in diverse applications. Our code is publicly available at https://github.com/JunliangLv/task_diversity_BFL . Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (KDD ’25), August 3–7, 2025, Toronto, ON, Canada DOI: 10.1145/3690624.3709341 ISBN: 979-8-4007-1245-6/25/08 CCS: Mathematics of computing Bayesian computation CCS: Mathematics of computing Variational methods CCS: Computing methodologies Multi-task learning CCS: Computing methodologies Distributed algorithms 1. Introduction Over the past few years, artificial intelligence has experienced tremendous growth. Traditional machine learning methods often necessitated centralizing datasets for training. However, with the proliferation of edge devices like smartphones and Internet of Things (IoT) devices, there is a strong demand for machine learning models to be trained on dispersed data. Therefore, federated learning (FL) ( Yang et al., 2019 ) has emerged as a concept in recent years, aiming to train models using data scattered across multiple local devices, thus avoiding large-scale data transfers and enhancing data privacy ( Zhang et al., 2021 ) . While FL has seen considerable advancement, it is known that most current FL efforts focus on homogeneous tasks on local devices, either exclusively for classification or solely for regression tasks. However, this contradicts real-world scenarios, where local devices often gather data for both types of tasks. Taking the health monitoring application on a smartphone as an example: it collects various health metrics such as heart rate, step count, and sleep quality. Suppose the application aims to classify the user’s movement states, such as stationary or walking, using sensor data like step count. Simultaneously, it can utilize heart rate and sleep duration for regression analysis, predicting trends in specific indicators. It is evident that this example involves both classification and regression tasks, and they are closely correlated. This implies a need to adopt multi-task learning (MTL) approaches to simultaneously handle both types of tasks on the local device. Furthermore, numerous existing FL frameworks rely on deterministic methods, suffering from overfitting when data is limited and providing predictions without uncertainty estimation, restricting their application in high-risk domains. For example, in high-risk domains, encountering decisions with high uncertainty indicates a need for caution, prompting a shift towards conservative strategies rather than complete reliance on algorithmic outputs. Similarly, in the context of out-of-distribution (OOD) detection, leveraging uncertainty helps identify OOD samples as they tend to exhibit higher uncertainty compared to in-distribution data. This article aims to integrate multi-task learning at the local level and federated learning at the global level in a principled probabilistic manner. Specifically, on each local device, we employ multi-output Gaussian processes (MOGP) ( Álvarez et al., 2012 ) to jointly model multiple correlated classification and regression tasks. As a Bayesian framework, MOGP naturally quantifies uncertainty through posterior inference. On the central server, we aggregate the posteriors uploaded by local devices to obtain an updated global MOGP prior. This updated global prior is then redistributed to local devices to train new local models. This iteration continues until the global convergence. It is worth noting that performing posterior inference on local devices presents a challenge due to the non-conjugacy of classification likelihood with MOGP prior, requiring approximation methods like Markov chain Monte Carlo (MCMC) ( Neal, 1993 ) or variational inference (VI) ( Blei et al., 2017 ) . While VI is computationally efficient, the standard methods, which assume a Gaussian variational distribution and optimize a tractable evidence lower bound (ELBO), often suffer from slow convergence ( Wenzel et al., 2019 ) . To address this challenge, this work employs the Pólya-Gamma augmentation technique ( Polson et al., 2013 ) , crafting a mean-field VI with closed-form expressions. Specifically, we make the following contributions: (1) at the local level, we extend from single-task to multi-task settings, empowering local device to handle correlated classification and regression tasks concurrently; (2) as a Bayesian approach, our local model not only provides predictions but also characterizes uncertainty, a crucial factor in OOD detection and model calibration; (3) by enhancing local posterior inference using Pólya-Gamma augmentation, we derive a completely analytical mean-field VI method, significantly boosting convergence; (4) across synthetic and real datasets, our method outperforms baselines in predictive performance, and demonstrates superior OOD detection, uncertainty calibration, and fast convergence. Lastly, we conduct ablation studies to explore the robustness of our method concerning various components. 2. Related Works In this section, we discuss pertinent research on FL, Bayesian FL, and multi-task learning. 2.1. Federated Learning In FL, collaboration among clients is pivotal for addressing learning tasks while upholding data privacy. Google introduced the initial FL algorithm, FedAvg, to safeguard client privacy in distributed learning ( McMahan et al., 2017 ) . Subsequent advancements encompass a range of methods to enhance convergence ( Li et al., 2018 ; Stich, 2018 ; Haddadpour and Mahdavi, 2019 ) , fortify data privacy ( Agarwal et al., 2018 ; Truex et al., 2019 ; Wei et al., 2020 ) , and improve communication efficiency ( Chen et al., 2021 ; Sattler et al., 2019 ; Reisizadeh et al., 2020 ) . Personalized federated learning (PFL) has gained traction in recent years, overcoming the suboptimal performance of early FL methods when confronted with heterogeneous datasets ( Rothchild et al., 2020 ) . Recent methods include local customization ( T Dinh et al., 2020 ; Huang et al., 2021 ; Tan et al., 2022 ) , meta-learning techniques ( Fallah et al., 2020b ; Fallah et al., 2020a ; Jiang et al., 2019 ) , and other strategies. Our method can be considered as a form of the meta-learning approach. 2.2. Bayesian Federated Learning To address uncertainty estimation and overfitting with limited data, some studies have proposed Bayesian federated learning (BFL) ( Cao et al., 2023 ) . In BFL, incorporating suitable priors on model parameters, as regularization, mitigates overfitting with limited data. Additionally, the posterior equips the model with the capability to capture uncertainty. Consequently, BFL facilitates more robust and well-calibrated predictions ( Triastcyn and Faltings, 2019 ; Zhang et al., 2022 ; Liu et al., 2023 ; Mou et al., 2023 ) . Recently, a cohort of BFL methods based on GPs has emerged ( Achituve et al., 2021 ; Dai et al., 2020 ; Yin et al.,