FedAH: Aggregated Head for Personalized Federated Learning
1 st Pengzhan Zhou 1, 2 nd Yuepeng He, 3 rd Yijun Zhai, 4 th Kaixin Gao, 5 th Chao Chen, 6 th Zhida Qin, 7 th Chong Zhang, 8 th Songtao Guo · 2024
Read FedAH: Aggregated Head for Personalized Federated Learning by 1 st Pengzhan Zhou 1, 2 nd Yuepeng He, 3 rd Yijun Zhai, 4 th Kaixin Gao, 5 th Chao Chen, 6 th Zhida Qin, 7 th Chong Zhang, 8 th Songtao Guo (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01295.txt.
Introduction
Pengzhan Zhou and co-authors present a 2024 method for training models across many sites while letting each site keep a personal touch. In federated learning, sites train a shared model without pooling private data. The personal variant splits the model into a shared feature extractor and a local head, the last part that makes the final call, which stays at each site.
Keeping the head purely local has a cost: the head never learns from the wider group. Their method, FedAH, starts each round with a head built by merging all sites' heads. Each site then adapts that merged head to its own data. The design keeps personal fit while letting global knowledge flow into the head. Tests in the paper judge the gain over the standard local-head scheme.
Catalog record
| CREATOR | 1 st Pengzhan Zhou 1, 2 nd Yuepeng He, 3 rd Yijun Zhai, 4 th Kaixin Gao, 5 th Chao Chen, 6 th Zhida Qin, 7 th Chong Zhang, 8 th Songtao Guo |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01295
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FedAH: Aggregated Head for Personalized Federated Learning 1 st Pengzhan Zhou 1, 2 nd Yuepeng He, 3 rd Yijun Zhai, 4 th Kaixin Gao, 5 th Chao Chen, 6 th Zhida Qin, 7 th Chong Zhang, 8 th Songtao Guo Abstract Recently, Federated Learning (FL) has gained popularity for its privacy-preserving and collaborative learning capabilities. Personalized Federated Learning (PFL), building upon FL, aims to address the issue of statistical heterogeneity and achieve personalization. Personalized-head-based PFL is a common and effective PFL method that splits the model into a feature extractor and a head, where the feature extractor is collaboratively trained and shared, while the head is locally trained and not shared. However, retaining the head locally, although achieving personalization, prevents the model from learning global knowledge in the head, thus affecting the performance of the personalized model. To solve this problem, we propose a novel PFL method called Federated Learning with Aggregated Head (FedAH), which initializes the head with an Aggregated Head at each iteration. The key feature of FedAH is to perform element-level aggregation between the local model head and the global model head to introduce global information from the global model head. To evaluate the effectiveness of FedAH, we conduct extensive experiments on five benchmark datasets in the fields of computer vision and natural language processing. FedAH outperforms ten state-of-the-art FL methods in terms of test accuracy by 2.87%. Additionally, FedAH maintains its advantage even in scenarios where some clients drop out unexpectedly. Our code is open-accessed at https://github.com/heyuepeng/FedAH. I Introduction The traditional centralized training approach in machine learning is facing significant challenges due to the increasing importance of user data privacy [ 1 , 2 , 3 ] . On the other hand, it is difficult to achieve well-performing models on individual clients due to the sparsity of data on each client [ 4 , 5 , 6 ] . Federated Learning (FL), as a popular distributed machine learning paradigm, offers excellent privacy protection and collaborative learning capabilities [ 7 ] . The learning tasks in FL are coordinated by a server and solved collaboratively by a network of multiple participating devices (clients). FedAvg [ 8 ] is the original FL method and serves as the fundamental framework for subsequent FL methods. Its iterative process can be described in five steps: (1) The server randomly selects a subset of clients to join FL and distributes the same global model to them for initialization; (2) Clients overwrite their local model parameters with the parameters of the downloaded global model to acquire global knowledge; (3) Clients train their local models on their private local data; (4) Clients upload their trained local models to the server; (5) The server receives the local models from clients and performs weighted averaging on the model parameters to obtain a new global model. FedAvg aims to learn a single global model that performs well across all clients. However, this approach often suffers in statistically heterogeneous environments, such as when facing not independent and identically distributed (Non-IID) and unbalanced data [ 5 , 9 ] , leading to degraded model performance [ 4 , 10 , 11 ] . Personalized Federated Learning (PFL) has been proposed to address statistical heterogeneity and achieve personalization in FL [ 6 ] . PFL focuses on learning personalized models rather than a single global model [ 11 ] . For each client participating in FL, the global model distributed by the server contains global information, which can enhance the local model and address the data scarcity issue of clients. Most existing PFL methods use the global model as a container for global information and exploit global/personalized information by leveraging the parameters of the global/local models [ 3 ] . Specifically, meta-learning-based PFL methods (such as Per-FedAvg [ 12 ] and FedMeta [ 13 ] ) adapt the global model parameters to heterogeneous client data through fine-tuning. Regularization-based PFL methods (such as FedProx [ 5 ] , pFedMe [ 11 ] , and Ditto [ 14 ] ) regularize the model parameters during local training. Personalized-aggregation-based PFL methods (such as FedFomo [ 15 ] , APPLE [ 16 ] , FedAMP [ 10 ] , and FedALA [ 17 ] ) achieve better local initialization by aggregating the models of other clients or combining global and local models. Personalized-head-based PFL methods (such as FedPer [ 18 ] and FedRep [ 19 ] ) split the model into a global part (feature extractor) and a personalized part (head), with the feature extractor trained collaboratively and shared among clients, while the head is trained locally and not shared. This approach aims to utilize both global and personalized information in the model parameters. However, training the head only with local data can result in the loss of some global information in the head, negatively impacting the performance of the personalized model. To address the issue of losing global information in the personalized head of personalized-head-based PFL methods, we propose a novel PFL method, Federated Learning with Aggregated Head (FedAH). As shown in Figure 1 , FedAH combines the ideas of personalized-aggregation-based PFL by aggregating the local model head from the previous iteration and the global model head from the current iteration at the element level to obtain an Aggregated Head, thereby integrating global information into the global model head. Apart from using the Aggregated Head as the initialization head for a new iteration, the rest of the processes are motivated by FedRep, and the aggregation weights of the Aggregated Head are learned through gradient descent. By combining the aforementioned strategies, FedAH can achieve personalization while more comprehensively benefiting from global knowledge, thereby improving the performance of personalized models. To evaluate the effectiveness of FedAH, we conduct extensive experiments in two widely adopted scenarios [ 8 , 20 ] (i.e., the pathological and practical heterogeneous settings) and five benchmark datasets. The experimental results demonstrate that FedAH outperforms ten state-of-the-art (SOTA) FL methods. In summary, our contributions are mainly three-fold: • To the best of our knowledge, we are the first to consider introducing global information through personalized aggregation in the model head. This approach is more fine-grained and effective compared to most existing personalized-aggregation-based PFL and personalized-head-based PFL methods. • We propose a novel PFL method, named FedAH, which, based on personalized-head-based PFL methods, performs element-wise aggregation between the local model head and the global model head to obtain an Aggregated Head, ensuring that global information is not lost in the personalized part of the model. • We conduct extensive experiments in the fields of computer vision (CV) and natural language processing (NLP) under two widely used scenarios. The results validate that our proposed FedAH outperforms SOTA FL methods in terms of effectiveness, scalability, and stability. Fig. 1 : An example for FedAH. r^{t} : global feature extractor; h^{t} : global heads; h^{t-1}_{1},\ldots,h^{t-1}_{n} : local head of clients 1,\ldots,n in the previous iteration; \hat{h}^{t}_{1},\ldots,\hat{h}^{t}_{n} : Aggregated Heads of clients 1,\ldots,n in the current iteration. The remained paper is organized as following. Section II summarizes the related work. Section III demonstrates the methods used. Section IV explains the experiments. Section V concludes the paper. II RELATED WORK II-A Personalized Federated Learning Traditional Federated Learning (FL)