Client-Customized Adaptation for Parameter-Efficient Federated Learning

Yeachan Kim, Junho Kim, Wing-Lam Mok, Jun-Hyung Park, SangKeun Lee · 2024

Read Client-Customized Adaptation for Parameter-Efficient Federated Learning by Yeachan Kim, Junho Kim, Wing-Lam Mok, Jun-Hyung Park, SangKeun Lee (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00311.txt.

Description

Abstract Despite the versatility of pre-trained language models (PLMs) across domains, their large memory footprints pose significant challenges in federated learning (FL), where the training model has to be distributed between a server and clients. One potential solution to bypass such constraints might be the use of parameter-efficient fine-tuning (PEFT) in the context of FL.

Catalog record

CREATORYeachan Kim, Junho Kim, Wing-Lam Mok, Jun-Hyung Park, SangKeun Lee
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00311

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Client-Customized Adaptation for Parameter-Efficient Federated Learning Yeachan Kim, Junho Kim, Wing-Lam Mok, Jun-Hyung Park, SangKeun Lee Abstract Despite the versatility of pre-trained language models (PLMs) across domains, their large memory footprints pose significant challenges in federated learning (FL), where the training model has to be distributed between a server and clients. One potential solution to bypass such constraints might be the use of parameter-efficient fine-tuning (PEFT) in the context of FL. However, we have observed that typical PEFT tends to severely suffer from heterogeneity among clients in FL scenarios, resulting in unstable and slow convergence. In this paper, we propose C lient- C ustomized A daptation (C2A), a novel hypernetwork-based FL framework that generates client-specific adapters by conditioning the client information. With the effectiveness of the hypernetworks in generating customized weights through learning to adopt the different characteristics of inputs, C2A can maximize the utility of shared model parameters while minimizing the divergence caused by client heterogeneity. To verify the efficacy of C2A, we perform extensive evaluations on FL scenarios involving heterogeneity in label and language distributions. Comprehensive evaluation results clearly support the superiority of C2A in terms of both efficiency and effectiveness in FL scenarios 1 1 1 Our code is available at https://github.com/yeachan-kr/c2a . 1 Introduction The advent of large-scale pre-trained language models (PLMs) for natural language processing (NLP) has led to exceptional performance across a broad spectrum of domains. However, the high memory requirements for PLMs impede their applicability to resource-constrained environments. These challenges are particularly evident in federated learning (FL), where model weights are transmitted between the server and clients to preserve data privacy Konečný et al. (2016) ; McMahan et al. (2017) . While recent FL studies have expanded the application of PLMs in various tasks, such as text classification Zhu et al. (2020) ; Qin et al. (2021) ; Weller et al. (2022) , language modeling Chen et al. (2019) , and question answering Chen et al. (2021) , communicating the training model among clients requires huge computational resources and bandwidth, presenting a significant challenge in terms of practicality. Parameter-efficient fine-tuning (PEFT) approach is thereby a promising strategy for reducing communication costs in FL. Through tuning only a small fraction of parameters, such as adapter-based tuning Houlsby et al. (2019) ; Hu et al. (2022) ; Mahabadi et al. (2021a) , bias tuning Zaken et al. (2022) , and prompt-tuning Lester et al. (2021) , PEFT approaches significantly enhance the memory efficiency in centralized scenarios. However, the feasibility of PEFT in decentralized scenarios has not been well explored. Figure 1: Conceptual illustration of the existing PEFT modules ( \mathcal{A} ) and the client-customized adaptation ( \mathcal{H} ). The proposed method learns to generate the client-customized PEFT modules rather than fitting a single global module to all clients. Hence, we investigate the applicability of typical PEFT approaches in FL scenarios. Specifically, we measure the performance and client drifts Karimireddy et al. (2020) ; Li et al. (2021) of PEFT approaches in FL. Our discoveries are as follows: (1) typical PEFT approaches show large performance degradation in FL scenarios as the degree of non-IID increases; (2) these approaches usually suffer from large client drifts in non-IID scenarios, resulting in slow convergence and detrimental model performance. The above observations reveal that adopting PEFT in FL is not trivial, and posing the necessity to address large client drift. To overcome the identified limitations, we propose a novel hypernetwork-based FL framework, C lient- C ustomized A daptation (C2A), that leverages the information of different data distributions on clients. Our key idea is to generate the adapter parameters tailored to each client via hypernetworks by taking the information of client data distribution, rather than naively fitting a single global adapter to all heterogeneous data distributions (Figure 1 ). By learning to adopt the different data distributions to generate adapters for each client, C2A enables robust training for various non-IID conditions while sharing knowledge among clients. Moreover, in order to manage the large number of parameters associated with hypernetworks, we introduce factorized hypernetworks, thereby significantly reducing the number of parameters without sacrificing the performance. We carefully design the experimental setting to verify the efficacy of C2A on realistic FL scenarios, considering on both label and language heterogeneous. The experimental results show clearly that C2A can be robust to the heterogeneity of clients, thereby leading to the state-of-the-art results on diverse non-IID setups. In addition, our framework shows a significant enhancement in training efficiency across a range of downstream tasks. Finally, we demonstrate that our C2A successfully mitigates the large client drifts among local clients in non-IID scenarios. A summary of our main contributions is as follows: • We investigate the effectiveness of PEFT among various FL scenarios. To the best of our knowledge, our work is one of the few researches for adapting PEFT in FL. • We propose C lient- C ustomized A daptation (C2A), a novel hypernetwork-based framework that strengthens the robustness of adapter concerning FL heterogeneity. • We demonstrate that C2A works quite well on various non-IID scenarios while preserving the benefits of efficiency in PEFT. 2 PEFT in FL Scenario 2.1 Background of FL The goal of federated learning is to collaboratively train a single global model without sharing any private data between clients. To this end, FL proceeds through the communication of training models between clients and the server in a round-by-round manner. For each round, the server first distributes a single global model \theta to a set of sampled clients, participating clients then perform local optimization on their own data. Upon the completion of the optimization, the server again aggregates all locally-trained models to update the global model. Formally, let the dataset of the ii-th client be \mathcal{D}_{i} , the above process for updating the global model can be formulated as follows: \widetilde{\theta}=\sum_{i=1}^{K}\alpha_{i}\cdot\mathcal{L}(\mathcal{D}_{i};\theta), (1) where \mathcal{L}(\mathcal{D}_{i};\theta) is the function that returns the trained model based on the given dataset and the initial model, K is the number of participating clients, and \alpha_{i} is the contributing factor of the client i to build a global model, which is typically determined by the dataset size of each client, i.e., \alpha_{i}=\frac{|\mathcal{D}_{i}|}{\sum_{i}|\mathcal{D}_{i}|} . While there are various aggregation methods, we focus on FedAvg due to its wide applicability in the FL community Karimireddy et al. (2020) ; Li et al. (2021) ; Luo et al. (2021) . However, utilizing cumbersome PLMs for the communication process of FL poses two challenges. Firstly, the function \mathcal{L}(\cdot) requires high computing resources due to the large number of trainable parameters associated with PLMs. Secondly, in the aggregation step (i.e., weighted summation), significant network bandwidth is required to transmit and receive the models. Therefore, it is crucial to find an optimal solution that can mitigate these constraints, providing a more efficient and less resource-intensive mechanism for FL with PLMs. 2.2 Impact of Heterogeneity on PEFT (a) (b) Figure 2: Sensitivity analysis of PEFT methods in federated context in terms of data heterogeneity and divergence from the global model. PT indicates prompt-tuning Lester et al. (2021) , and the dotted line indicates full fine-tuning method. To verify the applicability of PEFT in federated context, we conduct a preliminary investigation in which only small components (e.g.,

Read the complete text