Integration of large language models and federated learning Authors: Chaochao Chen, Xiaohua Feng, Yuyuan Li, Lingjuan Lyu, Jun Zhou, Xiaolin Zheng, Jianwei Yin Published: 2024 ABSTRACT SummaryAs the parameter size of large language models (LLMs) continues to expand, there is an urgent need to address the scarcity of high-quality data. In response, existing research has attempted to make a breakthrough by incorporating federated learning (FL) into LLMs. Conversely, considering the outstanding performance of LLMs in task generalization, researchers have also tried applying LLMs within FL to tackle challenges in relevant domains. The complementarity between LLMs and FL has already ignited widespread research interest. In this review, we aim to deeply explore the integration of LLMs and FL. We propose a research framework dividing the fusion of LLMs and FL into three parts: the combination of LLM sub-technologies with FL, the integration of FL sub-technologies with LLMs, and the overall merger of LLMs and FL. We first provide a comprehensive review of the current state of research in the domain of LLMs combined with FL, including their typical applications, integration advantages, challenges faced, and future directions for resolution. Subsequently, we discuss the practical applications of the combination of LLMs and FL in critical scenarios such as healthcare, finance, and education and provide new perspectives and insights into future research directions for LLMs and FL. Introduction The advent of large language models1 (LLMs) has markedly influenced contemporary society. These models use deep learning strategies, principally the transformer architecture,2 to discern intricate patterns and structures inherent to data.3 Currently, a vast body of work4,5,6 confirms that LLMs exhibit superior performance in both pre-defined tasks and practical applications. Impressively, given accurate instructions and demonstrations, LLMs are capable of adapting to specific contexts or addressing new tasks without additional fine-tuning, as corroborated by numerous studies.7,8,9 Moreover, LLMs have made significant strides in specialized domains, delivering commendable outcomes in areas like healthcare,10 finance,11 law,12,13,14 scientific knowledge analysis,15,16 and code generation.17,18 As the size of these models grows, more extensive training data are needed.19,20 However, recent research21 points out that there is a gap between the slow growth of public domain data and the rapid expansion of training data needs. This discrepancy may result in a shortage of high-quality public domain data for LLM training. Conversely, while private domains harbor colossal data volumes, concerns about privacy and commercial competition often hinder open collaboration and knowledge exchange. Take Figure 1, for example. Suppose that three hospitals want to establish an LLM for the medical field. Their own datasets would likely be insufficient. A joint dataset, on the other hand, would yield a substantial corpus. However, real-world data privacy regulations22 prevent direct plain-text sharing between separate entities.Figure 1The problem of data scarcity in LLMsNone of the hospitals has enough data for training LLMs, and they are reluctant to share data with one another. The problem of data scarcity in LLMs None of the hospitals has enough data for training LLMs, and they are reluctant to share data with one another. Considering the large parameter size and complex model structure of LLMs, common privacy-preserving computation techniques, such as secure multi-party computation23 (SMPC), differential privacy24 (DP), and trusted execution environments25 (TEEs), struggle to juggle privacy protection and computational efficiency effectively. Unlike these methods, federated learning26 (FL) offers a more practical approach by allowing collaborative model development. FL demonstrates a mature engineering execution method and strikes an ideal balance between efficiency and data privacy.27 Therefore, a feasible solution to address the challenges of LLMs in practical applications is to introduce FL into LLMs. Conversely, capitalizing on the strong task generalization capabilities of LLMs, they can also be employed within FL systems to help address challenges inherent to FL. Based on this complementarity, the combination of LLMs and FL has elicited widespread research interest. In this review, we focus on the promising direction of combining LLMs and FL. Previous studies present initial perspectives on this integration,28,29,30 providing preliminary insights into its motivations and future directions. Despite this, these studies have not fully covered all areas related to the integration of LLMs and FL. Specifically, some studies focus on exploring the integration of sub-technologies within LLMs and FL,29 neglecting the importance of the overall concept of federated LLMs (FedLLMs). In view of this, we adopt a more comprehensive research approach to organize existing work on combining LLMs and FL. By analyzing the current progress in combining LLMs and FL, we offer unique insights into the benefits, challenges, and future development trends. Notably, while analyzing the combination of sub-technologies in LLMs and FL, we also explore the sub-technologies shared by foundational models, extending beyond just language models (LMs) to include multi-modal and visual models. Since these shared sub-technologies can be easily adapted to language models, this broader perspective offers valuable insights into the integration of LLMs and FL. The remainder of this review is organized as follows: we first briefly introduce the technical backgrounds of FL and LLMs under “background.” Under “analysis of current integration of LLMs and FL,” we present a comprehensive analysis of the current status, challenges, and future directions regarding the combination of LLMs and FL. This includes three sub-sections: (1) the integration of sub-technologies in LLMs with FL, (2) the integration of sub-technologies in FL with LLMs, and (3) the overall integration of LLMs and FL. “Discussion of applications for combining LLMs and FL” analyzes the application scenarios where LLMs are combined with FL. Finally, under “conclusion and future work,” we summarize the progress of research on the integration of LLMs and FL and present insights into future development. Background LLMs Language models aim to predict the probability distribution of future tokens based on a given sequence of tokens.31 As the size of model parameters and the amount of training data increase, LLMs have shown impressive capabilities in handling complex tasks, including in-context learning (ICL),5 instruction following,6,9,32 and step-by-step reasoning.33 The success of LLMs is not due just to their large model sizes and extensive training data but also owes much to the Transformer architecture.34 Existing LLMs primarily rely on two design architectures,35 only decoder and encoder-decoder,34 with the only decoder architectures further divided into causal decoder5,36 and prefix decoder.37 Causal decoder architectures, which combine a unidirectional attention mask to ensure that each input token can attend to only past tokens and itself,38 have been widely adopted across various existing LLMs, offering significant advantages with massive training data. Specifically, GPT-35 successfully demonstrated the effectiveness of this architecture. Zhao et al.35 outline three key stages of training LLMs: pre-training, instruction tuning, and alignment tuning. During the pre-training stage, LLMs learn basic language processing abilities and world knowledge across a broad corpus, such as grammar, syntax, and general knowledge. Instruction tuning becomes crucial for refining LLMs’ ability to handle new tasks effectively. It involves crafting precise task instructions or contextual learning strategies to bolster the model’s adaptability to unseen tasks.32 Despite the benefits, there is a risk of instruction-fine-tuned models generating harmful content due to potential misalignment with human values.39,40 Therefore, aligning LLMs with human values, such as honesty and harmlessness, through alignment tuning has become an important task. To this end, InstructGPT6 proposes alignment training methods, including supervised fine-tuning and reinforcement learning from human feedback.6 FL FL executes collaborative model learning with the data from participants while safeguarding their privacy.26 Within FL, client devices asynchronously share updates such as weights and gradients while keeping raw data locally. The Federated Averaging (FedAvg) algorithm,26 which aggregates model updates from participating clients by averaging, is among the most prevalent aggregation algorithms in FL. Furthermore, studies41 consider the statistical challenges posed by heterogeneous user data in real-world scenarios. In summary, these advancements greatly accelerate the development of FL, striking a balance between maintaining data quality and enhancing the efficiency of collaborative training. Analysis of current integration of LLMs and FL In this section, we provide a detailed analysis of the current state of integration between LLMs and FL. As shown in Figure 2, this primarily encompasses three aspects: (1) integrating certain sub-technologies from LLMs with FL to endow FL with better properties, (2) combining some sub-technologies from FL with LLMs to address LLMs’ challenges, and (3) the overall integration of LLMs and FL allows both strengths to complement each other. We analyze these aspects in turn, analyzing the current research status behind each type of integration, existing challenges, and future directions.Figure 2Overview of the analysis process combining LLMs and FLWe sequentially analyze the integration of sub-technologies within LLMs with FL, the integration of sub-technologies within FL with LLMs, and the overall framework combining LLMs and FL. This includes the current status of integration, the advantages brought by the combination, potential challenges, and future directions for solutions. Overview of the analysis process combining LLMs and FL We sequentially analyze the integration of sub-technologies within LLMs with FL, the integration of sub-technologies within FL with LLMs, and the overall framework combining LLMs and FL. This includes the current status of integration, the advantages brought by the combination, potential challenges, and future directions for solutions. Integration of sub-technologies in LLMs with FL Although FL has been widely applied in practice, unsolved issues remain. LLMs introduce novel solutions to FL by leveraging their pre-trained knowledge and generalization abilities for universal tasks. Combining sub-technologies from LLMs with FL is a current focus of research exploration (Table 1). Below, we analyze the research status of integrating sub-technologies from LLMs with FL, the challenges present, and possible future solutions.Table 1Overview of the current state of sub-technologies within LLMs and their integration with FLSub-technologyAdvantage of integrationSummary of methodReferencesPre-trainingreducing the time for training convergencepre-trained models serve as the starting point for FL trainingTan et al.,42 Nguyen et al.,43 Liu et al.44solving non-iid problems in FLthe server performs pre-training before commencing FL trainingNguyen et al.,43 Chen et al.45empowering Fl models to handle multiple tasksmodular design assigns each task to the corresponding moduleAgarwal et al.,46 Zhang et al.,47 Zhang et al.48Prompt engineeringsolving personalization problems in FLpersonalized prompts are utilized to represent the local data distribution of clientYang et al.,49 Li et al.50solving the domain generalization problem in FLadaptive prompts for domain generalization are learned in a distributed mannerWei et al.,51 Bai et al.52We list the existing research on the combination of each sub-technology with FL and analyze the benefits they bring. Subsequently, we provide a brief summary of their methodologies. Table 1Overview of the current state of sub-technologies within LLMs and their integration with FLSub-technologyAdvantage of integrationSummary of methodReferencesPre-trainingreducing the time for training convergencepre-trained models serve as the starting point for FL trainingTan et al.,42 Nguyen et al.,43 Liu et al.44solving non-iid problems in FLthe server performs pre-training before commencing FL trainingNguyen et al.,43 Chen et al.45empowering Fl models to handle multiple tasksmodular design assigns each task to the corresponding moduleAgarwal et al.,46 Zhang et al.,47 Zhang et al.48Prompt engineeringsolving personalization problems in FLpersonalized prompts are utilized to represent the local data distribution of clientYang et al.,49 Li et al.50solving the domain generalization problem in FLadaptive prompts for domain generalization are learned in a distributed mannerWei et al.,51 Bai et al.52We list the existing research on the combination of each sub-technology with FL and analyze the benefits they bring. Subsequently, we provide a brief summary of their methodologies. Overview of the current state of sub-technologies within LLMs and their integration with FL We list the existing research on the combination of each sub-technology with FL and analyze the benefits they bring. Subsequently, we provide a brief summary of their methodologies. Current status Combining sub-technologies within LLMs with FL, current research primarily explores the following two aspects: pre-training and prompt engineering. Below, we detail the research status of each integration approach. Pre-training Current research leverages pre-training techniques within LLMs to tackle multiple challenges in FL, such as reducing the time to convergence for FL training, addressing non-independent identically distributed (non-iid) issues in FL, and empowering FL models with the ability to handle multiple tasks. Reducing the time for training convergence: in FL research, starting with randomly initialized neural network weights often slows down model convergence. Studies show that using pre-trained models, trained on large datasets, as a starting point for FL can significantly reduce training time.42,44 Instead of starting from scratch, clients can fine-tune the FL model using their local data. Experiments demonstrate that using a pre-trained model can reduce the training time required to achieve a target error rate compared with those starting from random initialization.43 This faster convergence results in better-performing models in fewer communication rounds. Solving non-iid problems in FL: one common challenge in FL is dealing with data and system heterogeneity. Data heterogeneity refers to variations in data distributions across clients,53 while system heterogeneity relates to differences in client device capabilities.43 To mitigate these challenges, researchers have developed joint optimization methods.54,55 Starting FL from a pre-trained model initialization has been found to help alleviate the effects of data and system heterogeneity.43 This approach can lead to more stable global model aggregation and reduce the accuracy gap between FL and centralized learning, especially in scenarios with non-iid client data.45 Empowering FL models to handle multiple tasks: FL typically focuses on a single task, which may not be sufficient for real-world applications with diverse task requirements.56 Large pre-trained models have demonstrated the ability to perform well across multiple tasks.5 Some research endeavors to integrate pre-trained models into the FL framework to enable FL models to handle various tasks.46,47 However, more attention is needed on FL in mobile and edge devices. The FedYolo framework proposes a modular approach where clients load a complete pre-trained model and make future updates through communication-efficient modules.48 Experiments show that this design allows clients to simultaneously solve multiple unrelated tasks with a single pre-trained model, reducing catastrophic forgetting compared to full updates. Prompt engineering Prompt techniques have demonstrated exceptional performance within LLMs.57 Current research is exploring the integration of prompts with the FL framework to address issues of personalization and domain generalization within FL. Solving personalization problems in FL: personalized FL allows for personalized models to enhance their generalization and robustness by leveraging knowledge from distributed clients. The pFedPG framework has utilized large-scale pre-trained models to acquire robust representations while achieving efficient model personalization for heterogeneous clients.49 While pFedPG does not consider client data characteristics, recent work, i.e., pFedPT, uses personalized prompts to implicitly represent local data distributions.50 During pFedPT training, each client generates a personalized prompt related to their data distribution, aiding classification tasks by incorporating this information into the aggregated model. Solving the domain generalization problem in FL: FL is crucial for learning from decentralized data but faces challenges when training data (source domain) differ from the test dataset (target domain). The Fed-DPT framework initially addressed this using visual and textual prompts, but required domain labels during training and had limitations on the number of domains.51 To overcome this, the DiPrompT framework was proposed, learning adaptive prompts for domain generalization in a distributed manner.52 DiPrompT uses global prompts to capture shared knowledge and domain prompts for specific domain knowledge, eliminating the need for a strict one-to-one mapping between source domains and local clients. Challenges and future directions The integration of sub-technologies within LLMs with FL can resolve many issues but also introduces some new challenges. Below, we discuss each of these new challenges. Bias between the pre-training data for LLMs and the data used for FL The domain mismatch between training and test data poses a significant challenge in current research,58,59,60 especially when integrating sub-technologies from LLMs into FL. This mismatch can reduce the effectiveness of model transfer and application. In addition, if synthetic data created by LLMs does not align with client data distribution, it may introduce bias and noise into the FL process. To address these challenges, future research should focus on enhancing the quality and diversity of synthetic data generated by LLMs to closely match the underlying data distribution and application domains in FL. One potential approach is to utilize pre-processing techniques to fine-tune the alignment between LLMs and FL systems before incorporating them into FL processes.61 This strategy aims to minimize the bias between LLM-generated data and foundational FL data, ensuring their distributions are as similar as possible. Data bias transmission issues In the era of LLMs, the training and fine-tuning datasets for LLMs are vast and diverse, potentially containing toxic content, user privacy data, politically sensitive information, and biases.62 LLMs, being probabilistic generative models with limited interpretability and controllability,63 may generate synthetic data of questionable quality and safety, leading to issues like data toxicity, biases, and misinformation. When FL training is conducted on these synthetic datasets, these problems can transfer to the final FL model. To address these challenges, future research should focus on integrating LLMs into FL systems in a way that prevents new biases and avoids amplifying existing ones. This could involve developing LLM data augmentation techniques guided by fairness principles and applying bias elimination techniques to remove biases from FL systems, such as combining LLM-based data augmentation with federated unlearning techniques. In addition, creating more robust FL aggregation algorithms could effectively prevent the introduction of biases into the system. Data privacy and copyright issues When LLM-generated data are used in FL, concerns about privacy rights and copyright emerge.64 LLMs gather vast amounts of internet data during pre-training, including private and copyrighted information, making it hard to trace the origins of these data. Recent studies show that LLMs have strong memory capabilities,65,66 suggesting that the data they generate could closely resemble the privacy and copyright information encountered during training.67 This poses legal risks for FL models trained using these datasets. To address these issues, future research should explore how to balance the usefulness of synthetic data from LLMs with privacy and copyright protection. First, it is imperative to develop a method for determining whether generated data adhere to privacy protection and copyright regulations. Building on this, researchers should be able to selectively generate data, ensuring distinct differentiation from the original data. Furthermore, exploring the interpretability of model inference within an FL environment is also a viable research direction. This will aid in intuitively identifying the sources of generated content that do not comply with standards and accordingly taking appropriate remedial actions. Combination of black-box LLMs and FL Currently, when combining LLMs with FL, researchers typically use a white-box approach, where the model’s structure and parameters are fully transparent. This allows for a deep understanding of how the models work and enables adjustments to meet FL requirements. However, some high-performance LLMs, e.g., GPT-4,68 operate as black-box application programming interface (API) services in real-world applications,69,70 meaning users cannot access the internal workings of the model directly but interact with it through an API. To effectively combine black-box LLMs with FL, knowledge distillation can be employed.71 For example, pre-trained LLMs act as teacher models, guiding the training of student models within the FL system. The teacher model’s output, obtained via API calls, serves as pseudo-labels for FL training data. Student models’ predictions are then aligned with these pseudo-labels to distill knowledge effectively.72 Integration of sub-technologies in FL with LLMs Current status The training requirements for LLMs demand an immense amount of data and computational resources. In FL, distributed computing73 and privacy-preserving computation74 are considered effective tools to meet these demands. Existing research integrates these key technologies covered by FL with LLMs, aiming to address the practical issues LLMs face (Table 2).Table 2Overview of the current state of sub-technologies within FL and their integration with LLMsSub-technologyAdvantage of integrationSummary of methodReferencesDistributed computingassessing computing resourcesaggregating computational capacities from multiple sourcesZeng et al.,75 Wu et al.,76 Huang et al.77enhancing LLMs’ task generalization capabilitiesaggregating proprietary data from multi-party devicesHong et al.78Privacy-preserving computationimproving prompt generation abilityutilizing proprietary specific data to generate targeted promptsGuo et al.,57 Chen et al.,79 Chen et al.,80 Yao et al.81assisting with the selection of CoT promptbalancing the generality and personalization in the selection of CoT promptsXing et al.,82 Duan et al.,83 Wang et al.84We list the existing research on the combination of each sub-technology with LLMs and analyze the benefits they bring. Subsequently, we provide a brief summary of their methodologies. Table 2Overview of the current state of sub-technologies within FL and their integration with LLMsSub-technologyAdvantage of integrationSummary of methodReferencesDistributed computingassessing computing resourcesaggregating computational capacities from multiple sourcesZeng et al.,75 Wu et al.,76 Huang et al.77enhancing LLMs’ task generalization capabilitiesaggregating proprietary data from multi-party devicesHong et al.78Privacy-preserving computationimproving prompt generation abilityutilizing proprietary specific data to generate targeted promptsGuo et al.,57 Chen et al.,79 Chen et al.,80 Yao et al.81assisting with the selection of CoT promptbalancing the generality and personalization in the selection of CoT promptsXing et al.,82 Duan et al.,83 Wang et al.84We list the existing research on the combination of each sub-technology with LLMs and analyze the benefits they bring. Subsequently, we provide a brief summary of their methodologies. Overview of the current state of sub-technologies within FL and their integration with LLMs We list the existing research on the combination of each sub-technology with LLMs and analyze the benefits they bring. Subsequently, we provide a brief summary of their methodologies. Distributed computing In FL, distributed computing helps LLMs by combining computing and data resources. This eases the workload for individual users during training and inference and boosts LLMs’ ability to handle different tasks by merging data from multiple parties. Assessing computing resources: training LLMs demands significant computational power. For example, LLaMA needs 2,048 NVIDIA A100 GPUs for 21 days,85 GPT-3-1.3B requires 64 Tesla V100 GPUs for 7 days,5 and FLM utilizes 192 NVIDIA A800 GPUs for 22 days.86 Such costs are manageable mainly by big tech firms like Microsoft and Google, limiting LLMs’ progress. FedML and others combine FL with LLMs to share computing resources among participants, easing the burden during training and inference stages.75,76,77 Enhancing LLMs’ task generalization capabilities: LLMs are mainly trained on vast centralized datasets, e.g., GPT-NeoX-20B on Pile87 and LLaMA on comprehensive data including other LLMs’ datasets.85 Yet, these datasets do not cover all real-world knowledge, hampering models’ adaptability. To address this, recent research integrates data from various sources using FL’s distributed data processing,78 aiming to enhance models’ generalization, including data from medium-scale infrastructures and individual mobile devices. Privacy-preserving computation Prompts play a crucial role in helping LLMs process complex tasks.88 Public dataset prompts tend to be repetitive, and privacy regulations limit the use of private data for prompt generation. To improve this, current studies merge FL’s privacy-preserving tech with prompt design in LLMs. This enhances personalized prompt matching and better meets specific requirements. Improving prompt generation ability: LLMs improve their ability to understand complex tasks through prompt engineering.88 However, to address privacy concerns, prompt designs often rely on publicly available data. This approach, while protecting privacy, limits the potential of prompt engineering from two aspects.30 First, public datasets may not have access to specific domains or individual private information, hindering optimization for specialized fields. Second, using public datasets can result in generic prompt templates, leading to repetitive or uninspired model responses. Recent research integrates FL’s privacy-preserving features with prompt generation,57,79,80,81 allowing for optimized prompts tailored to specific domains, thus enabling better adaptation to particular needs. Assisting with the selection of chain of thought (CoT) prompts: CoT reasoning, a method for eliciting quick and accurate responses from LLMs, is gaining attention in research.8 However, choosing the best prompts poses a challenge. Currently, prompt selection relies on trial and error, where users adjust prompts based on LLM responses. To improve the explainability of CoT prompt selection and balance universality and personalization across domains while protecting privacy, recent studies combine FL with LLMs.82,83,84 They propose the FedLogic framework,82 which tackles prompt selection as a rule selection problem based on fuzzy scores, using LLMs as rule generators. Challenges and future directions Additional communication overhead Although FL applied to LLMs can ease computational burdens, it introduces extra communication expenses. Due to the large number of LLM parameters, communication time might surpass training time significantly. Real-world network instability could worsen this issue.89 In addition, extensive communication can harm the environment by raising carbon emissions. Therefore, efficient distributed learning algorithms are crucial. These algorithms must tackle communication and computational challenges during LLM training and deployment across devices with varying capabilities and network conditions. Currently, training acceleration strategies, e.g., Deepspeed,90 Megatron,91 and BMTrain,92 speed up LLM training via data parallelism, model parallelism, and pipeline parallelism.93 Applying these strategies within an FL environment is relatively straightforward and can, to some extent, address the local computational issues associated with FedLLMs. However, these strategies do not fully resolve communication challenges. A more effective approach involves employing model pruning94 and compression95 techniques to reduce the complexity and size of the model, thereby alleviating computational and communication burdens without sacrificing model performance. In addition, extending parameter-efficient fine-tuning methods to FedLLMs is also an effective solution. Model property rights issues Training LLMs relies on vast, domain-specific datasets, making resulting models commercially valuable intellectual property. Ensuring ownership of these models is crucial, especially in distributed training scenarios like FL, which involve multiple collaborating parties.53 This increases the risk of model leaks and intellectual property infringement. To tackle these challenges, it is essential to develop theoretical and methodological frameworks in artificial intelligence to identify ownership misappropriation and illegal claims. Authentication technologies should provide robust intellectual property protection without compromising model performance. Model watermarking is a promising solution.96 It allows collaborative model updating and training while safeguarding private data and signatures. Implementing model watermarking in LLMs could effectively address intellectual property challenges. Overall integration of LLMs and FL In this section, we discuss the benefits of FedLLMs compared to separate technologies, review current FedLLM trends, and examine potential challenges. Finally, we share our insights and suggestions for addressing these challenges. Current status Compared to simply combining LLMs and FL sub-technologies, FedLLMs present a comprehensive framework, marking a new direction in privacy-preserving language model development. Its key advantages include: (1) ensuring privacy while effectively integrating high-quality data from multiple parties for superior model training,97 (2) providing solutions for general task adaptation and large model training in specific areas,97 and (3) facilitating lifelong learning.98 Several recent studies have explored integrating FL with LLMs.97,99,100,101 Among them, the earliest studies publish perspective articles about FedLLMs.28 FATE-LLM97 investigates fine-tuning strategies within FL to reduce communication overhead, incorporating efficient fine-tuning methods.35 However, its application was limited to traditional classification tasks. Subsequent works, i.e., FederatedScope-LLMs99 and Shepherd,100 expand into federated instruction tuning but lack diverse training datasets. OpenFedLLM101 adds a federated alignment-tuning mechanism, enhancing LLM training. It conducts empirical analysis across datasets and compares FL aggregation methods.26,102 Please refer to Table 3 for a detailed comparison.Table 3Comparison of current FedLLM frameworksFramework namePTITVANaggNdataNevaFATE-LLM97✓××114FederatedScope-LLM99✓✓×133Shepherd100✓✓×111OpenFedLLM101✓✓✓7830+Following Ye et al.,101 we adopt the following notation definitions: PT, parameter-efficient fine-tuning; IT, instruction tuning; VA, value alignment tuning; Nagg, number of supported FL aggregation algorithms; Ndata, number of training datasets; Ndata, number of evaluation metrics. Table 3Comparison of current FedLLM frameworksFramework namePTITVANaggNdataNevaFATE-LLM97✓××114FederatedScope-LLM99✓✓×133Shepherd100✓✓×111OpenFedLLM101✓✓✓7830+Following Ye et al.,101 we adopt the following notation definitions: PT, parameter-efficient fine-tuning; IT, instruction tuning; VA, value alignment tuning; Nagg, number of supported FL aggregation algorithms; Ndata, number of training datasets; Ndata, number of evaluation metrics. Comparison of current FedLLM frameworks Following Ye et al.,101 we adopt the following notation definitions: PT, parameter-efficient fine-tuning; IT, instruction tuning; VA, value alignment tuning; Nagg, number of supported FL aggregation algorithms; Ndata, number of training datasets; Ndata, number of evaluation metrics. Although existing FedLLM frameworks differ in design and implementation, they follow the same core design principle: extending the training process of LLMs within an FL system. To construct a robust FedLLMs framework, we provide a detailed overview of the architecture of FedLLMs based on the aforementioned principle. Referring to Figure 3, we divide the architectural structure of FedLLMs into two main phases: training and inference. In the training phase, the framework is further subdivided into pre-training, instruction-tuning, and alignment-tuning stages, with each stage offering a display of various specific implementation methods in the respective sub-figures. During the inference phase, we expand the existing FedLLM framework to allow clients to combine the outputs of local and global models when executing inferences, thereby enhancing the accuracy and adaptability of inference.Figure 3The current implementation framework of FedLLMsThe use of a FedLLM primarily includes two phases: training and inference. The training phase further comprises pre-training, instruction tuning, and alignment tuning. Our sub-figures at the bottom detail feasible methods for implementing each part. The current implementation framework of FedLLMs The use of a FedLLM primarily includes two phases: training and inference. The training phase further comprises pre-training, instruction tuning, and alignment tuning. Our sub-figures at the bottom detail feasible methods for implementing each part. Challenges and future directions Synchronization problems due to differences in computing resources In FL, exchanging gradient information between devices incurs high communication costs, especially when resources vary among participants. This issue is more pronounced in FedLLMs due to their larger parameter scales.103 Limited network bandwidth exacerbates the problem, potentially causing the dropout of some members and prolonging communication time.104 To mitigate this issue, a naive strategy is collaborative computing,105 which limits the computational potential of resource-rich clients to match the capabilities of weaker clients. However, this approach can lead to a significant waste of computational resources. A more rational approach is to implement hierarchical aggregation106 within FedLLMs, where all clients do not communicate directly with the central server, but instead perform preliminary data aggregation within local groups (or clusters). Each cluster is led by one or more clients with superior computational capabilities, responsible for collecting and aggregating data within the cluster before exchanging information with the central server or leaders of other clusters. Incentive mechanisms for FedLLMs In the implementation of FedLLMs, creating a fair and effective incentive mechanism is crucial to encourage broader participation and collaboration among contributors, given the varying data volumes and computational capabilities among participants.107 This involves balancing data contributions and computational resources and aligning incentive strategies with each participant’s investment and derived value. Although incentive mechanisms for smaller models in traditional FL environments have been explored,108 they are not directly applicable to LLMs due to their massive parameter size. Therefore, FedLLMs require specifically tailored incentive solutions. Reinforcement learning has been highlighted as effective in designing FL incentive mechanisms, especially considering the characteristics of LLMs and their complementarity with reinforcement learning.6 Applying reinforcement learning to FedLLMs and developing specialized incentive mechanisms for them seems to be a promising path forward. Continuous data stream During FedLLM training, data streams continuously, unlike in centralized LLMs where data comes in fixed batches.58,109 Clients join and leave, potentially with data distributions different from the global model. Given LLMs’ large size, training from scratch is resource intensive. Integrating new data with minimal resources is a challenge. Federated continual learning110 is considered a viable solution. However, most existing methods are parameter based,111 which can be resource intensive and inefficient when dealing with large-scale models. In contrast, loss-based approaches may be more suitable for FedLLMs. Furthermore, exploring the implementation of LLM editing112 techniques in an FL environment, where only a sub-set of parameters is updated with each model iteration, could also help address this issue. Personalization issues in FedLLMs Current FedLLM frameworks prioritize training a unified model collaboratively, often overlooking personalization concerns. In the LLM domain, there are two main viewpoints: one advocates for universal LLMs with larger parameters,68,113 while the other focuses on smaller, proprietary models for practical contexts like mobile devices. Considering personalized needs in FedLLMs is essential. A straightforward solution is to integrate existing federated personalization methods into the FedLLM framework for enhancement, such as model-agnostic personalization,114 hierarchical personalization,115 and cluster-based FL.116 Additional security and privacy threats LLMs combined with FL could worsen security and privacy risks, creating new challenges. Existing FedLLM frameworks often overlook these issues. To the best of our knowledge, we are the first to analyze security and privacy threats in FedLLMs. According to Usynin et al.,117 we classify threats into utility-focused attacks and privacy-focused attacks (Table 4). The former aims to impair model effectiveness, termed security threats, while the latter intends to compromise data privacy, termed privacy threats. We analyze these threats and their new variants in FedLLMs.Table 4Overview of attacks against FedLLM systemsAttack typeAttack goalProposed defensesReferencesSecurity threatsUntargeted poisoningdegrading utility of the target modeli, ii, iii, vCao et al.,118 Tolpegin et al.,119 Wallace et al.,120 Schuster et al.,121 Wan et al.,122 Carlini et al.,123 Fang et al.,124 Rodríguez-Barroso et al.,125 Sun et al.126Backdoorrunning an auxiliary learning taski, iii, ivGu et al.,127 Kurita et al.,128 Shen et al.,129 Chen et al.,130 Dong et al.,131 Mei et al.,132 Xu et al.,133 Shu et al.,134 Huang et al.,135 Wan et al.,136 Sun et al.,137 Bagdasaryan et al.138Privacy threatsMembership inferenceinferring the presence of an individual recordvi, viiShokri et al.,139 Jagannatha et al.,140 Perez and Ribeiro,141 Fowl et al.142Attribute inferenceinferring sensitive value of a recordviSong and Shmatikov,143 Gong and Liu,144 Hayet et al.145Model inversionreconstruction of training datavi, viiFredrikson et al.,146 Pan et al.,147 Gupta et al.,148 Fredrikson et al.,149 He et al.,150 Song and Raghunathan,151 Gu et al.,152 Morris et al.,153 Balunovic et al.,154 Chu et al.155Based on the work by Usynin et al.,117 we categorize the potential security and privacy threats faced by FedLLMs. In addition, we highlight the common methods currently employed to defend against these threats. The notation definitions corresponding to these defenses are as follows: i, data analysis; ii, update analysis; iii, robust aggregation; iv, model pruning; v, adversarial training; vi, DP; and vii, knowledge distillation. Table 4Overview of attacks against FedLLM systemsAttack typeAttack goalProposed defensesReferencesSecurity threatsUntargeted poisoningdegrading utility of the target modeli, ii, iii, vCao et al.,118 Tolpegin et al.,119 Wallace et al.,120 Schuster et al.,121 Wan et al.,122 Carlini et al.,123 Fang et al.,124 Rodríguez-Barroso et al.,125 Sun et al.126Backdoorrunning an auxiliary learning taski, iii, ivGu et al.,127 Kurita et al.,128 Shen et al.,129 Chen et al.,130 Dong et al.,131 Mei et al.,132 Xu et al.,133 Shu et al.,134 Huang et al.,135 Wan et al.,136 Sun et al.,137 Bagdasaryan et al.138Privacy threatsMembership inferenceinferring the presence of an individual recordvi, viiShokri et al.,139 Jagannatha et al.,140 Perez and Ribeiro,141 Fowl et al.142Attribute inferenceinferring sensitive value of a recordviSong and Shmatikov,143 Gong and Liu,144 Hayet et al.145Model inversionreconstruction of training datavi, viiFredrikson et al.,146 Pan et al.,147 Gupta et al.,148 Fredrikson et al.,149 He et al.,150 Song and Raghunathan,151 Gu et al.,152 Morris et al.,153 Balunovic et al.,154 Chu et al.155Based on the work by Usynin et al.,117 we categorize the potential security and privacy threats faced by FedLLMs. In addition, we highlight the common methods currently employed to defend against these threats. The notation definitions corresponding to these defenses are as follows: i, data analysis; ii, update analysis; iii, robust aggregation; iv, model pruning; v, adversarial training; vi, DP; and vii, knowledge distillation. Overview of attacks against FedLLM systems Based on the work by Usynin et al.,117 we categorize the potential security and privacy threats faced by FedLLMs. In addition, we highlight the common methods currently employed to defend against these threats. The notation definitions corresponding to these defenses are as follows: i, data analysis; ii, update analysis; iii, robust aggregation; iv, model pruning; v, adversarial training; vi, DP; and vii, knowledge distillation. Security threats: adversaries take a different approach, aiming to alter the learning protocol or undermine the model’s utility.117 In FedLLMs, the main threat to model performance is poisoning attacks. These attacks can be divided into untargeted and targeted (backdoor) poisoning attacks based on the attackers' goals.156•Untargeted poisoning attacks. This attack involves minor manipulations of training data, where malicious actors introduce altered or distorted data samples into the federated dataset.118,119 This intentional bias or misguidance aims to disrupt the subsequent model training process. In FedLLMs, we incorporate textual data. Introducing harmful noise into textual data, including tag inclusion, modification, or omission, is relatively easy to execute.120 While typically used in image data, recent studies suggest optimized methods for perturbations on discrete data, i.e., textual data, expanding poisoning possibilities.123,124 In FL, a client could cause harm by sending corrupted updates, making FedLLMs vulnerable to adversarial perturbations.125 Numerous studies suggest that FedLLMs are susceptible to poisoning attacks,121,122,126 raising concerns about detection difficulty.•Backdoor attacks. These attacks covertly manipulate models to exhibit normal behaviors but can be triggered by specific inputs to produce the adversary’s desired output.127 Unlike untargeted poisoning attacks, backdoor attacks involve the insertion or modification of precise input patterns.129,130,131,132 Backdoor attacks have extended in new ways within FedLLMs. During the instruction fine-tuning stage, LLMs are vulnerable to backdoor attacks. Recent studies have acknowledged this risk, emphasizing the potential pathways for attackers to insinuate malicious commands133,134 and the concept of untrained vocabulary backdoor attacks on language models.135 Moreover, some studies perceive prompt injection attacks88,136 as a unique spin on backdoor attacks, with the compliance capabilities of LLMs being the primary target.134 Apart from novel types of backdoor attacks, the escalating complexity of models in FL settings fosters backdoor insertions. This is attributable to the capacity of overparameterized models to learn trigger features, even amid label noise during training. Furthermore, as models within FL are collectively utilized and contributed to by multiple clients, the scope for attacks and origins of backdoors inevitably broadens,137,138 further increasing the risk of backdoor attacks. •Untargeted poisoning attacks. This attack involves minor manipulations of training data, where malicious actors introduce altered or distorted data samples into the federated dataset.118,119 This intentional bias or misguidance aims to disrupt the subsequent model training process. In FedLLMs, we incorporate textual data. Introducing harmful noise into textual data, including tag inclusion, modification, or omission, is relatively easy to execute.120 While typically used in image data, recent studies suggest optimized methods for perturbations on discrete data, i.e., textual data, expanding poisoning possibilities.123,124 In FL, a client could cause harm by sending corrupted updates, making FedLLMs vulnerable to adversarial perturbations.125 Numerous studies suggest that FedLLMs are susceptible to poisoning attacks,121,122,126 raising concerns about detection difficulty. Untargeted poisoning attacks. This attack involves minor manipulations of training data, where malicious actors introduce altered or distorted data samples into the federated dataset.118,119 This intentional bias or misguidance aims to disrupt the subsequent model training process. In FedLLMs, we incorporate textual data. Introducing harmful noise into textual data, including tag inclusion, modification, or omission, is relatively easy to execute.120 While typically used in image data, recent studies suggest optimized methods for perturbations on discrete data, i.e., textual data, expanding poisoning possibilities.123,124 In FL, a client could cause harm by sending corrupted updates, making FedLLMs vulnerable to adversarial perturbations.125 Numerous studies suggest that FedLLMs are susceptible to poisoning attacks,121,122,126 raising concerns about detection difficulty. •Backdoor attacks. These attacks covertly manipulate models to exhibit normal behaviors but can be triggered by specific inputs to produce the adversary’s desired output.127 Unlike untargeted poisoning attacks, backdoor attacks involve the insertion or modification of precise input patterns.129,130,131,132 Backdoor attacks have extended in new ways within FedLLMs. During the instruction fine-tuning stage, LLMs are vulnerable to backdoor attacks. Recent studies have acknowledged this risk, emphasizing the potential pathways for attackers to insinuate malicious commands133,134 and the concept of untrained vocabulary backdoor attacks on language models.135 Moreover, some studies perceive prompt injection attacks88,136 as a unique spin on backdoor attacks, with the compliance capabilities of LLMs being the primary target.134 Apart from novel types of backdoor attacks, the escalating complexity of models in FL settings fosters backdoor insertions. This is attributable to the capacity of overparameterized models to learn trigger features, even amid label noise during training. Furthermore, as models within FL are collectively utilized and contributed to by multiple clients, the scope for attacks and origins of backdoors inevitably broadens,137,138 further increasing the risk of backdoor attacks. Backdoor attacks. These attacks covertly manipulate models to exhibit normal behaviors but can be triggered by specific inputs to produce the adversary’s desired output.127 Unlike untargeted poisoning attacks, backdoor attacks involve the insertion or modification of precise input patterns.129,130,131,132 Backdoor attacks have extended in new ways within FedLLMs. During the instruction fine-tuning stage, LLMs are vulnerable to backdoor attacks. Recent studies have acknowledged this risk, emphasizing the potential pathways for attackers to insinuate malicious commands133,134 and the concept of untrained vocabulary backdoor attacks on language models.135 Moreover, some studies perceive prompt injection attacks88,136 as a unique spin on backdoor attacks, with the compliance capabilities of LLMs being the primary target.134 Apart from novel types of backdoor attacks, the escalating complexity of models in FL settings fosters backdoor insertions. This is attributable to the capacity of overparameterized models to learn trigger features, even amid label noise during training. Furthermore, as models within FL are collectively utilized and contributed to by multiple clients, the scope for attacks and origins of backdoors inevitably broadens,137,138 further increasing the risk of backdoor attacks. Privacy threats: these attacks aim to access a client’s private information. Due to the large number of parameters in LLMs, model extraction attacks are very costly. Therefore, this review focuses on membership inference attacks (MIAs), attribute inference attacks, and model inversion attacks and will not discuss model extraction attacks.•MIAs. These attacks aim to predict whether a given data record is a member of the training dataset.139 Given the fact that LLMs memorize training data,65,66,157,158 the risk of MIA increases, especially if the memorized information includes personal or sensitive data.140,141 Hence, it is necessary to explore defense mechanisms against MIAs in the FL environment. The integration of FL and LLMs could also potentially bring about novel manners of inference attacks. Recent research illustrates that the memory facet of LLMs significantly increases their susceptibility to privacy violations within the FL framework.148 If the server is hypothesized to be dishonest or compromised, the structure of the LLM is prone to inference attacks.142 How to defend against this threat is a question that needs to be explored in the future.•Attribute inference attacks. These attacks aim to recover characteristics of the training data learned by the model.143,144 Due to LLMs commonly handling extensive textual data, attribute inference attacks carried out on text data are generally accomplished by obtaining the embedding vectors of text samples, thereby gaining access to the confidential attributes embedded within these samples. Prior research in this field has focused on the adjustment of these embeddings to apprehend semantic interrelations existing between words, thereby predicting confidential data inherent in language models.145 Engaging in FL increases the vulnerability to potential attacks, necessitating consideration of a broader range of attackers. Concurrently, white-box adversaries stand to benefit from unencrypted model updates, with a particular emphasis on gradient data, thereby delivering them a competitive edge in the process.•Model inversion attacks. In attribute inference, it is crucial to acknowledge that attackers are typically required to hold additional information regarding their potential victims, such as personal identifiers (e.g., age and race), to exploit the association between this information and the sensitive features. On the other hand, model inversion attacks primarily aim to reverse engineer the internal representation produced by the model to reveal training data.146,149,150 In LLMs, this is typically done on the embedding rather than directly on the output.147,151 Recent advanced research suggests exploring embedding inversion attacks, showing that these attacks pose a higher privacy risk than attribute inference attacks.152,153 Considering the FL scenario, the gradient leakage issue exacerbates this threat. Studies demonstrate that the embeddings can be reconstructed based on leaked gradients.148,154,155 •MIAs. These attacks aim to predict whether a given data record is a member of the training dataset.139 Given the fact that LLMs memorize training data,65,66,157,158 the risk of MIA increases, especially if the memorized information includes personal or sensitive data.140,141 Hence, it is necessary to explore defense mechanisms against MIAs in the FL environment. The integration of FL and LLMs could also potentially bring about novel manners of inference attacks. Recent research illustrates that the memory facet of LLMs significantly increases their susceptibility to privacy violations within the FL framework.148 If the server is hypothesized to be dishonest or compromised, the structure of the LLM is prone to inference attacks.142 How to defend against this threat is a question that needs to be explored in the future. MIAs. These attacks aim to predict whether a given data record is a member of the training dataset.139 Given the fact that LLMs memorize training data,65,66,157,158 the risk of MIA increases, especially if the memorized information includes personal or sensitive data.140,141 Hence, it is necessary to explore defense mechanisms against MIAs in the FL environment. The integration of FL and LLMs could also potentially bring about novel manners of inference attacks. Recent research illustrates that the memory facet of LLMs significantly increases their susceptibility to privacy violations within the FL framework.148 If the server is hypothesized to be dishonest or compromised, the structure of the LLM is prone to inference attacks.142 How to defend against this threat is a question that needs to be explored in the future. •Attribute inference attacks. These attacks aim to recover characteristics of the training data learned by the model.143,144 Due to LLMs commonly handling extensive textual data, attribute inference attacks carried out on text data are generally accomplished by obtaining the embedding vectors of text samples, thereby gaining access to the confidential attributes embedded within these samples. Prior research in this field has focused on the adjustment of these embeddings to apprehend semantic interrelations existing between words, thereby predicting confidential data inherent in language models.145 Engaging in FL increases the vulnerability to potential attacks, necessitating consideration of a broader range of attackers. Concurrently, white-box adversaries stand to benefit from unencrypted model updates, with a particular emphasis on gradient data, thereby delivering them a competitive edge in the process. Attribute inference attacks. These attacks aim to recover characteristics of the training data learned by the model.143,144 Due to LLMs commonly handling extensive textual data, attribute inference attacks carried out on text data are generally accomplished by obtaining the embedding vectors of text samples, thereby gaining access to the confidential attributes embedded within these samples. Prior research in this field has focused on the adjustment of these embeddings to apprehend semantic interrelations existing between words, thereby predicting confidential data inherent in language models.145 Engaging in FL increases the vulnerability to potential attacks, necessitating consideration of a broader range of attackers. Concurrently, white-box adversaries stand to benefit from unencrypted model updates, with a particular emphasis on gradient data, thereby delivering them a competitive edge in the process. •Model inversion attacks. In attribute inference, it is crucial to acknowledge that attackers are typically required to hold additional information regarding their potential victims, such as personal identifiers (e.g., age and race), to exploit the association between this information and the sensitive features. On the other hand, model inversion attacks primarily aim to reverse engineer the internal representation produced by the model to reveal training data.146,149,150 In LLMs, this is typically done on the embedding rather than directly on the output.147,151 Recent advanced research suggests exploring embedding inversion attacks, showing that these attacks pose a higher privacy risk than attribute inference attacks.152,153 Considering the FL scenario, the gradient leakage issue exacerbates this threat. Studies demonstrate that the embeddings can be reconstructed based on leaked gradients.148,154,155 Model inversion attacks. In attribute inference, it is crucial to acknowledge that attackers are typically required to hold additional information regarding their potential victims, such as personal identifiers (e.g., age and race), to exploit the association between this information and the sensitive features. On the other hand, model inversion attacks primarily aim to reverse engineer the internal representation produced by the model to reveal training data.146,149,150 In LLMs, this is typically done on the embedding rather than directly on the output.147,151 Recent advanced research suggests exploring embedding inversion attacks, showing that these attacks pose a higher privacy risk than attribute inference attacks.152,153 Considering the FL scenario, the gradient leakage issue exacerbates this threat. Studies demonstrate that the embeddings can be reconstructed based on leaked gradients.148,154,155 Outdated defense techniques Concerning the security and privacy threats discussed, existing defense strategies may not seamlessly apply to FedLLMs. We typically classify defense mechanisms into security defenses and privacy defenses (Table 5). By analyzing how current defenses might face challenges when applied to FedLLMs, we offer our perspectives and insights.Table 5Overview of defense techniques in FedLLM systemsMitigation typeSummaryMitigatable attacksReferencesSecurity defensesData analysisanalyzing data from other clientsi, iiCretu et al.159Update analysisanalyzing updates from various contributorsiShen et al.,160 Andreina et al.161Robust aggregationreplacing update averaging with aggregationi, iiBlanchard et al.,162 Yin et al.,163 Wu et al.,164 Pillutla et al.,165 El Mhamdi et al.,166 Ashraf et al.167Model pruningdropping specific neurons/units of the modeliiDhillon et al.,168 Wu et al.,169 Grachev et al.170Adversarial trainingtraining the model on adversarial examplesiGanin et al.,171 Shayegani et al.172Privacy defensesKnowledge distillationtransferring knowledge from the teacher model to the student modeliii, vPapernot et al.,173 Fay et al.,174 Papernot et al.,175 Jeong et al.176Differential privacyimplementing targeted disturbances for the protocoliii, iv, vHu et al.,177 Yu et al.,178 Li et al.,179 Bu et al.,180 Gupta et al.,181 Shi et al.,182 Xu et al.,183 Du et al.184We enumerate the defense techniques currently in widespread use and have identified the types of attacks these defense methods can address in FedLLMs. The definitions of the symbols are as follows: i, untargeted poisoning attack; ii, backdoor attack; iii, MIA; iv, attribute inference attack; and v, inversion attack. Table 5Overview of defense techniques in FedLLM systemsMitigation typeSummaryMitigatable attacksReferencesSecurity defensesData analysisanalyzing data from other clientsi, iiCretu et al.159Update analysisanalyzing updates from various contributorsiShen et al.,160 Andreina et al.161Robust aggregationreplacing update averaging with aggregationi, iiBlanchard et al.,162 Yin et al.,163 Wu et al.,164 Pillutla et al.,165 El Mhamdi et al.,166 Ashraf et al.167Model pruningdropping specific neurons/units of the modeliiDhillon et al.,168 Wu et al.,169 Grachev et al.170Adversarial trainingtraining the model on adversarial examplesiGanin et al.,171 Shayegani et al.172Privacy defensesKnowledge distillationtransferring knowledge from the teacher model to the student modeliii, vPapernot et al.,173 Fay et al.,174 Papernot et al.,175 Jeong et al.176Differential privacyimplementing targeted disturbances for the protocoliii, iv, vHu et al.,177 Yu et al.,178 Li et al.,179 Bu et al.,180 Gupta et al.,181 Shi et al.,182 Xu et al.,183 Du et al.184We enumerate the defense techniques currently in widespread use and have identified the types of attacks these defense methods can address in FedLLMs. The definitions of the symbols are as follows: i, untargeted poisoning attack; ii, backdoor attack; iii, MIA; iv, attribute inference attack; and v, inversion attack. Overview of defense techniques in FedLLM systems We enumerate the defense techniques currently in widespread use and have identified the types of attacks these defense methods can address in FedLLMs. The definitions of the symbols are as follows: i, untargeted poisoning attack; ii, backdoor attack; iii, MIA; iv, attribute inference attack; and v, inversion attack. Security defenses Security defenses aim to alleviate the adverse effects of attacks on model performance. Our study mainly contemplates widely accepted methods, encompassing data analysis, update analysis, robust aggregation, model pruning, and adversarial training.•Data and update analysis. Data analysis involves evaluating data from other clients and implementing subsequent pre-processing.159 However, it is often impractical to apply under privacy-preserving conditions due to the need for access to user-specific local data.53 In contrast, update analysis reviews parameters from other clients to determine their necessity for aggregation. While effective, this technique requires access to client updates, potentially increasing the risk of privacy leakage. The analysis often relies on outlier update analysis,160,161 which is challenging due to the high dimensionality of LLM updates. Possible solutions may involve using dimensionality reduction techniques such as principal-component analysis119 or implementing spectral anomaly detection with low-dimensional embeddings.185.•Robust aggregation. The aim of robust aggregation is to mitigate the negative impact of adversaries on the final model.162,163,164,165,166 Despite their significant potential, implementing these methods in deep learning models presents a major challenge.162,186 At the same time, it is important to consider the potential unintended consequences of modifying the aggregation mechanism, as this could have a negative impact on LLM architectures. Existing research has shown that FedAvg can negatively influence the attention mechanism of LLMs.167 Thus, maintaining compatibility with LLM architecture during this process is crucial.•Model pruning. Model pruning assumes that most of the model’s weight contains knowledge relevant to the original task, while only a small portion is affected by poisoning attacks.168 This assumption suggests post-training defensive measures involving pruning the globally trained model to strengthen it against potential training-based attacks.169 For deep network architectures like LLMs, specialized adaptations of model pruning techniques can be developed,170 and exploration can also be carried out on mainstream LLMs.94 Ultimately, these pruning methods can be applied in the FL environment, enhancing overall robustness.•Adversarial training. This approach trains the model using additional adversarial samples171 to enhance the model’s adversarial robustness against attacks. However, there are two potential issues with this approach when applied to FedLLMs. First, generating adversarial examples for discrete data types such as text can be considerably more complex than for images, as direct perturbations in the embedding space may lead to significant semantic deviations due to minor disturbances.172 Second, the resource expenditure for generating adversarial samples in deep neural networks is substantial when using gradient-based adversarial perturbation methods like projected gradient descent.187 To address these challenges, a viable approach is to introduce an additional LLM that employs prompt engineering techniques to generate semantically similar adversarial samples.188 Alternatively, adversarial samples can also be generated through discrete optimization methods.189 To further reduce overhead, updates can be selectively applied based on the importance of model parameters.190 •Data and update analysis. Data analysis involves evaluating data from other clients and implementing subsequent pre-processing.159 However, it is often impractical to apply under privacy-preserving conditions due to the need for access to user-specific local data.53 In contrast, update analysis reviews parameters from other clients to determine their necessity for aggregation. While effective, this technique requires access to client updates, potentially increasing the risk of privacy leakage. The analysis often relies on outlier update analysis,160,161 which is challenging due to the high dimensionality of LLM updates. Possible solutions may involve using dimensionality reduction techniques such as principal-component analysis119 or implementing spectral anomaly detection with low-dimensional embeddings.185. Data and update analysis. Data analysis involves evaluating data from other clients and implementing subsequent pre-processing.159 However, it is often impractical to apply under privacy-preserving conditions due to the need for access to user-specific local data.53 In contrast, update analysis reviews parameters from other clients to determine their necessity for aggregation. While effective, this technique requires access to client updates, potentially increasing the risk of privacy leakage. The analysis often relies on outlier update analysis,160,161 which is challenging due to the high dimensionality of LLM updates. Possible solutions may involve using dimensionality reduction techniques such as principal-component analysis119 or implementing spectral anomaly detection with low-dimensional embeddings.185. •Robust aggregation. The aim of robust aggregation is to mitigate the negative impact of adversaries on the final model.162,163,164,165,166 Despite their significant potential, implementing these methods in deep learning models presents a major challenge.162,186 At the same time, it is important to consider the potential unintended consequences of modifying the aggregation mechanism, as this could have a negative impact on LLM architectures. Existing research has shown that FedAvg can negatively influence the attention mechanism of LLMs.167 Thus, maintaining compatibility with LLM architecture during this process is crucial. Robust aggregation. The aim of robust aggregation is to mitigate the negative impact of adversaries on the final model.162,163,164,165,166 Despite their significant potential, implementing these methods in deep learning models presents a major challenge.162,186 At the same time, it is important to consider the potential unintended consequences of modifying the aggregation mechanism, as this could have a negative impact on LLM architectures. Existing research has shown that FedAvg can negatively influence the attention mechanism of LLMs.167 Thus, maintaining compatibility with LLM architecture during this process is crucial. •Model pruning. Model pruning assumes that most of the model’s weight contains knowledge relevant to the original task, while only a small portion is affected by poisoning attacks.168 This assumption suggests post-training defensive measures involving pruning the globally trained model to strengthen it against potential training-based attacks.169 For deep network architectures like LLMs, specialized adaptations of model pruning techniques can be developed,170 and exploration can also be carried out on mainstream LLMs.94 Ultimately, these pruning methods can be applied in the FL environment, enhancing overall robustness. Model pruning. Model pruning assumes that most of the model’s weight contains knowledge relevant to the original task, while only a small portion is affected by poisoning attacks.168 This assumption suggests post-training defensive measures involving pruning the globally trained model to strengthen it against potential training-based attacks.169 For deep network architectures like LLMs, specialized adaptations of model pruning techniques can be developed,170 and exploration can also be carried out on mainstream LLMs.94 Ultimately, these pruning methods can be applied in the FL environment, enhancing overall robustness. •Adversarial training. This approach trains the model using additional adversarial samples171 to enhance the model’s adversarial robustness against attacks. However, there are two potential issues with this approach when applied to FedLLMs. First, generating adversarial examples for discrete data types such as text can be considerably more complex than for images, as direct perturbations in the embedding space may lead to significant semantic deviations due to minor disturbances.172 Second, the resource expenditure for generating adversarial samples in deep neural networks is substantial when using gradient-based adversarial perturbation methods like projected gradient descent.187 To address these challenges, a viable approach is to introduce an additional LLM that employs prompt engineering techniques to generate semantically similar adversarial samples.188 Alternatively, adversarial samples can also be generated through discrete optimization methods.189 To further reduce overhead, updates can be selectively applied based on the importance of model parameters.190 Adversarial training. This approach trains the model using additional adversarial samples171 to enhance the model’s adversarial robustness against attacks. However, there are two potential issues with this approach when applied to FedLLMs. First, generating adversarial examples for discrete data types such as text can be considerably more complex than for images, as direct perturbations in the embedding space may lead to significant semantic deviations due to minor disturbances.172 Second, the resource expenditure for generating adversarial samples in deep neural networks is substantial when using gradient-based adversarial perturbation methods like projected gradient descent.187 To address these challenges, a viable approach is to introduce an additional LLM that employs prompt engineering techniques to generate semantically similar adversarial samples.188 Alternatively, adversarial samples can also be generated through discrete optimization methods.189 To further reduce overhead, updates can be selectively applied based on the importance of model parameters.190 Privacy defenses Common privacy protection techniques include DP, knowledge distillation, regularization, and model pruning. However, the privacy benefits of regularization are limited, and certain techniques have been effectively bypassed, so we are not focusing on this method in this review. Model pruning, which is a combined defense mechanism, has been found useful for privacy protection.117,191 However, its use in privacy-sensitive situations may raise concerns as it could unintentionally reveal sensitive features of the training data. Therefore, this study categorizes it as a performance-focused defense. The main privacy-focused defense measures considered in this study are knowledge distillation and DP.•Knowledge distillation. Knowledge distillation allows the knowledge of a model to be transferred to a simpler model.71 Originally conceived to mitigate overfitting, knowledge distillation has evolved, with current research ingeniously combining it with DP principles.173,174 This innovative approach has given rise to new systems like private aggregation of teacher ensembles.175 These systems harness publicly available datasets to enable the transfer of knowledge from locally trained models to centralized models operating under DP mechanisms. In the federated setting, federated distillation is already a mature framework.176 Transferring the distillation technique of LLMs192 to FL is a direction worth exploring.•DP. DP currently stands as the principal paradigm for privacy protection. DP methods on language models include gradient perturbation-based approaches and embedding vector perturbation-based approaches.177 The former adds noise to the gradients during network training, while the latter perturbs the word embeddings, aiming to protect privacy at the sample level (i.e., words or sentences). However, in FedLLMs, privacy protection extends beyond the sample level to the user privacy level, aiming to safeguard each user’s historical data. In addition, since only gradients are exchanged between clients in FedLLMs, methods based on embedding vector perturbation cannot be directly extended to FedLLMs. For gradient perturbation-based methods, although existing research provides theoretical privacy guarantees, two significant issues arise as the model scales up, and these issues are exacerbated in an FL environment. First, the computational and storage overhead of managing gradients increases.178 Second, the scale of noise required also increases,178 which can adversely affect model performance to some extent. To address these challenges, a straightforward extension of the improved DP-SGD optimizer178,179,180,181 to the FL environment is a viable direction. In addition, relaxing the level of DP to protect only the sensitive parts of samples using SDP-SGD182 is another potential approach. Finally, exploring the combination of DP with existing efficient parameter fine-tuning methods could also be a feasible strategy.183,184 •Knowledge distillation. Knowledge distillation allows the knowledge of a model to be transferred to a simpler model.71 Originally conceived to mitigate overfitting, knowledge distillation has evolved, with current research ingeniously combining it with DP principles.173,174 This innovative approach has given rise to new systems like private aggregation of teacher ensembles.175 These systems harness publicly available datasets to enable the transfer of knowledge from locally trained models to centralized models operating under DP mechanisms. In the federated setting, federated distillation is already a mature framework.176 Transferring the distillation technique of LLMs192 to FL is a direction worth exploring. Knowledge distillation. Knowledge distillation allows the knowledge of a model to be transferred to a simpler model.71 Originally conceived to mitigate overfitting, knowledge distillation has evolved, with current research ingeniously combining it with DP principles.173,174 This innovative approach has given rise to new systems like private aggregation of teacher ensembles.175 These systems harness publicly available datasets to enable the transfer of knowledge from locally trained models to centralized models operating under DP mechanisms. In the federated setting, federated distillation is already a mature framework.176 Transferring the distillation technique of LLMs192 to FL is a direction worth exploring. •DP. DP currently stands as the principal paradigm for privacy protection. DP methods on language models include gradient perturbation-based approaches and embedding vector perturbation-based approaches.177 The former adds noise to the gradients during network training, while the latter perturbs the word embeddings, aiming to protect privacy at the sample level (i.e., words or sentences). However, in FedLLMs, privacy protection extends beyond the sample level to the user privacy level, aiming to safeguard each user’s historical data. In addition, since only gradients are exchanged between clients in FedLLMs, methods based on embedding vector perturbation cannot be directly extended to FedLLMs. For gradient perturbation-based methods, although existing research provides theoretical privacy guarantees, two significant issues arise as the model scales up, and these issues are exacerbated in an FL environment. First, the computational and storage overhead of managing gradients increases.178 Second, the scale of noise required also increases,178 which can adversely affect model performance to some extent. To address these challenges, a straightforward extension of the improved DP-SGD optimizer178,179,180,181 to the FL environment is a viable direction. In addition, relaxing the level of DP to protect only the sensitive parts of samples using SDP-SGD182 is another potential approach. Finally, exploring the combination of DP with existing efficient parameter fine-tuning methods could also be a feasible strategy.183,184 DP. DP currently stands as the principal paradigm for privacy protection. DP methods on language models include gradient perturbation-based approaches and embedding vector perturbation-based approaches.177 The former adds noise to the gradients during network training, while the latter perturbs the word embeddings, aiming to protect privacy at the sample level (i.e., words or sentences). However, in FedLLMs, privacy protection extends beyond the sample level to the user privacy level, aiming to safeguard each user’s historical data. In addition, since only gradients are exchanged between clients in FedLLMs, methods based on embedding vector perturbation cannot be directly extended to FedLLMs. For gradient perturbation-based methods, although existing research provides theoretical privacy guarantees, two significant issues arise as the model scales up, and these issues are exacerbated in an FL environment. First, the computational and storage overhead of managing gradients increases.178 Second, the scale of noise required also increases,178 which can adversely affect model performance to some extent. To address these challenges, a straightforward extension of the improved DP-SGD optimizer178,179,180,181 to the FL environment is a viable direction. In addition, relaxing the level of DP to protect only the sensitive parts of samples using SDP-SGD182 is another potential approach. Finally, exploring the combination of DP with existing efficient parameter fine-tuning methods could also be a feasible strategy.183,184 Discussion of applications for combining LLMs and FL The integration of LLMs with FL promises to complement the advantages of both and effectively address their respective limitations. The resulting synergistic effect suggests that the combination of LLMs with FL could be widely applied to various practical scenarios. Given that current research has explored the potential of the fusion of LLMs’ sub-technologies with FL, as well as the integration of FL’s sub-technologies with LLMs in application scenarios,29 this review focuses on discussing the feasible applications of FedLLMs in practice. In light of the inherent distinguished characteristics of FedLLMs, we explore the broad range of application fields for FedLLMs. These applications mainly include healthcare, finance, education, and so on. Within these scenarios, deploying FedLLMs has the potential to address real industry problems, optimize service processes, and enhance overall efficiency and effectiveness. In addition, we also focus on the unique challenges faced by FedLLMs in these scenarios and provide an analysis of them. Healthcare The healthcare scenario is one of the application areas that is intimately related to human well-being and is of great importance. Since the introduction of ChatGPT and other LLMs, numerous studies have applied these technologies in the healthcare field.193,194,195 It has been proven that LLMs have the capability to handle a variety of healthcare tasks, including but not limited to healthcare consultation recommendations,196 simplification of healthcare reports,197 mental health analysis,198 and extraction of biohealthcare information.199 To further tap into the potential of large models, recent research focuses on large models specially designed for the healthcare field, such as the Med-PaLM model.194,200 In the United States Healthcare Licensing Examination, this model demonstrates performance comparable to that of professionals and has gained broader recognition from the healthcare community in answering consumer health questions. However, there is a risk of privacy breaches when the current LLMs upload patient health information to commercial servers that support model training.197,199 This issue urgently needs to be addressed through technical means. FedLLMs offer an effective way to help healthcare institutions aggregate data from multiple parties to train their own healthcare-specific large models, tackling the aforementioned privacy challenges. Although FedLLMs exhibit high potential for application in the healthcare scenario, the implementation of this technology is still limited by the unique characteristics of healthcare data and its strict usage regulations. The specific challenges faced include the following:•Data heterogeneity. Healthcare data often originate from various sources, including electronic health records, healthcare imaging, and laboratory results. These data vary significantly in format, quality, and level of detail. In FedLLMs, due to the involvement of multiple different participants, the types of data held by each party may also be inconsistent, further exacerbating the problem of data heterogeneity.•Data incompleteness and imbalance. Healthcare units participating in FedLLMs often face issues of missing data or incomplete records, especially in scenarios involving long-term monitoring of patients. In addition, data samples for certain diseases may be much less than for others, leading to data imbalance during model training, which can affect the model’s generalization ability and accuracy.•Model interpretability. The healthcare field has higher requirements for model interpretability compared to other sectors. Healthcare decisions directly affect people’s health, and doctors and patients usually need to clearly understand the basis of model predictions. How to reflect the interpretability of models within the FedLLMs framework is an urgent issue to be addressed. •Data heterogeneity. Healthcare data often originate from various sources, including electronic health records, healthcare imaging, and laboratory results. These data vary significantly in format, quality, and level of detail. In FedLLMs, due to the involvement of multiple different participants, the types of data held by each party may also be inconsistent, further exacerbating the problem of data heterogeneity. Data heterogeneity. Healthcare data often originate from various sources, including electronic health records, healthcare imaging, and laboratory results. These data vary significantly in format, quality, and level of detail. In FedLLMs, due to the involvement of multiple different participants, the types of data held by each party may also be inconsistent, further exacerbating the problem of data heterogeneity. •Data incompleteness and imbalance. Healthcare units participating in FedLLMs often face issues of missing data or incomplete records, especially in scenarios involving long-term monitoring of patients. In addition, data samples for certain diseases may be much less than for others, leading to data imbalance during model training, which can affect the model’s generalization ability and accuracy. Data incompleteness and imbalance. Healthcare units participating in FedLLMs often face issues of missing data or incomplete records, especially in scenarios involving long-term monitoring of patients. In addition, data samples for certain diseases may be much less than for others, leading to data imbalance during model training, which can affect the model’s generalization ability and accuracy. •Model interpretability. The healthcare field has higher requirements for model interpretability compared to other sectors. Healthcare decisions directly affect people’s health, and doctors and patients usually need to clearly understand the basis of model predictions. How to reflect the interpretability of models within the FedLLMs framework is an urgent issue to be addressed. Model interpretability. The healthcare field has higher requirements for model interpretability compared to other sectors. Healthcare decisions directly affect people’s health, and doctors and patients usually need to clearly understand the basis of model predictions. How to reflect the interpretability of models within the FedLLMs framework is an urgent issue to be addressed. Finance The field of finance is one of the key areas where LLMs demonstrate their vast application potential. LLMs have been employed in a variety of financial tasks, including but not limited to financial reasoning,201 digital claims detection,202 financial named entity recognition,203 and financial sentiment analysis.204 While general-purpose LLMs like ChatGPT have notable performance in the financial industry, they still cannot match the level of large models that are specifically trained and fine-tuned for the financial scenario, such as BloombergGPT,205 FinGPT,206 etc. However, LLMs tailored for the financial scenario require access to vast amounts of high-quality financial data,205 which may exceed the capacities of some organizations. FedLLMs offer an innovative path for cultivating financial-scenario-specific large models. Moreover, given that content generated by financial models could have significant impacts on markets, stringent alignment and adjustment of financial models are an indispensable step. The collaborative mechanism of FedLLMs can meet more complex and stricter alignment requirements, ensuring that aligned models adequately regard and reflect the interests of the majority of participants. While FedLLMs introduce unprecedented new opportunities into the financial scenario, they also bring a series of new challenges.•High dynamism. Data in financial markets are highly dynamic and change rapidly. For instance, stock prices and interest rates can undergo significant changes within very short periods. This requires the FedLLMs framework to support participants in rapidly updating language models in a short time, rather than relying on periodic retraining.•High accuracy and reliability. Financial decisions often have significant financial implications; thus, the information provided must be extremely accurate and reliable. This poses higher accuracy standards for the inference process of the FedLLM framework.•Enhanced contextual understanding. Financial question-answering scenarios often involve complex contexts and multi-step logical reasoning. When the FedLLM framework is applied for inference, it needs to possess strong contextual understanding capabilities, being capable of handling coherent dialogue, remembering previous communications, and understanding complex query intentions. •High dynamism. Data in financial markets are highly dynamic and change rapidly. For instance, stock prices and interest rates can undergo significant changes within very short periods. This requires the FedLLMs framework to support participants in rapidly updating language models in a short time, rather than relying on periodic retraining. High dynamism. Data in financial markets are highly dynamic and change rapidly. For instance, stock prices and interest rates can undergo significant changes within very short periods. This requires the FedLLMs framework to support participants in rapidly updating language models in a short time, rather than relying on periodic retraining. •High accuracy and reliability. Financial decisions often have significant financial implications; thus, the information provided must be extremely accurate and reliable. This poses higher accuracy standards for the inference process of the FedLLM framework. High accuracy and reliability. Financial decisions often have significant financial implications; thus, the information provided must be extremely accurate and reliable. This poses higher accuracy standards for the inference process of the FedLLM framework. •Enhanced contextual understanding. Financial question-answering scenarios often involve complex contexts and multi-step logical reasoning. When the FedLLM framework is applied for inference, it needs to possess strong contextual understanding capabilities, being capable of handling coherent dialogue, remembering previous communications, and understanding complex query intentions. Enhanced contextual understanding. Financial question-answering scenarios often involve complex contexts and multi-step logical reasoning. When the FedLLM framework is applied for inference, it needs to possess strong contextual understanding capabilities, being capable of handling coherent dialogue, remembering previous communications, and understanding complex query intentions. Education The education scenario is also a key application area significantly influenced by LLMs. Recently, several pioneering research papers have explored the diverse applications of LLMs in educational settings,207,208 including teacher-student interactive collaboration, personalized learning experiences, and the automation of assessment processes. However, the application of LLMs in education can also bring a range of practical issues, such as homework plagiarism, the intrinsic biases of AI-generated content, overreliance on LLMs, and the inequity in accessing resources for non-English-speaking countries.209 Against this backdrop, FedLLMs offer a solution for cultivating fair LLMs. The increase in participating parties and the richness of training data contribute to reducing biases present in LLMs and expanding their adaptability to multi-lingual environments. Through FedLLMs, it is possible to achieve multi-dimensional data collaboration, driving the creation of equitable and inclusive educational LLMs that consider and balance the needs of different languages and cultural backgrounds. In the application within the educational scenario, the FedLLMs framework also faces several new challenges.•Complexity of different educational stages and backgrounds. In the FedLLMs framework, participant entities serve student groups that vary significantly in age, learning ability, and background knowledge. Therefore, the framework needs to possess the capability to adapt to these differences to provide customized learning recommendations and content.•Diversity of educational goals. Educational objectives are not limited to improving academic performance but also include emotional development, social skills, and growth in other non-academic areas. In this context, FedLLMs need to consider these multi-faceted factors to assess and propose recommendations for the holistic development of students.•Strong guidance capability. An ideal educational LLM should guide students gradually toward finding the correct answers. In the FedLLM framework, enhancing the model’s CoT reasoning capabilities is a critical issue that requires focused attention.•Higher alignment requirements. In the educational field, given the limited discernment abilities of students at different age levels, there are higher demands for the alignment of models trained via FedLLMs. Furthermore, the model should also be capable of refusing unreasonable requests of students. •Complexity of different educational stages and backgrounds. In the FedLLMs framework, participant entities serve student groups that vary significantly in age, learning ability, and background knowledge. Therefore, the framework needs to possess the capability to adapt to these differences to provide customized learning recommendations and content. Complexity of different educational stages and backgrounds. In the FedLLMs framework, participant entities serve student groups that vary significantly in age, learning ability, and background knowledge. Therefore, the framework needs to possess the capability to adapt to these differences to provide customized learning recommendations and content. •Diversity of educational goals. Educational objectives are not limited to improving academic performance but also include emotional development, social skills, and growth in other non-academic areas. In this context, FedLLMs need to consider these multi-faceted factors to assess and propose recommendations for the holistic development of students. Diversity of educational goals. Educational objectives are not limited to improving academic performance but also include emotional development, social skills, and growth in other non-academic areas. In this context, FedLLMs need to consider these multi-faceted factors to assess and propose recommendations for the holistic development of students. •Strong guidance capability. An ideal educational LLM should guide students gradually toward finding the correct answers. In the FedLLM framework, enhancing the model’s CoT reasoning capabilities is a critical issue that requires focused attention. Strong guidance capability. An ideal educational LLM should guide students gradually toward finding the correct answers. In the FedLLM framework, enhancing the model’s CoT reasoning capabilities is a critical issue that requires focused attention. •Higher alignment requirements. In the educational field, given the limited discernment abilities of students at different age levels, there are higher demands for the alignment of models trained via FedLLMs. Furthermore, the model should also be capable of refusing unreasonable requests of students. Higher alignment requirements. In the educational field, given the limited discernment abilities of students at different age levels, there are higher demands for the alignment of models trained via FedLLMs. Furthermore, the model should also be capable of refusing unreasonable requests of students. Conclusion and future work Creating high-performing and robust LLMs relies on having sufficient high-quality data, which is often difficult and costly to obtain. To address the issue of data scarcity, researchers have incorporated FL techniques into LLMs, pooling data from multiple parties for training while ensuring privacy. In addition, integrating LLMs into FL helps address some specific challenges faced by FL, as LLMs possess exceptional task generalization capabilities. Numerous studies have demonstrated the complementarity of LLMs and FL in these domains. These studies also include investigation of non-language foundation models, due to their potential extension to LLMs, to provide a broader perspective for our research. Given this complementarity, the research field combining LLMs and FL demonstrates significant potential for development. In this regard, this review explores this research area, proposing a framework to organize ongoing efforts. We analyze advantages, challenges, and future directions, including potential applications in healthcare, finance, and education. This review aims to guide the development of integration technologies between LLMs and FL, emphasizing the need for unified evaluation benchmarks and datasets in future research. Acknowledgments This work was supported by the 10.13039/501100012166National Key R&D Program of China (2022YFB4501500), the Fundamental Research Funds for the Central Universities (226-2024-00241), and the Ant Group. We thank all team members and partners involved in this study for their support and contributions. In addition, we appreciate the valuable comments and suggestions provided by the reviewers of this paper. Author contributions Conceptualization, C.C., L.L., and J.Z.; methodology, X.F. and Y.L.; investigation, X.F. and Y.L.; writing – original draft, X.F. and Y.L.; writing – review & editing, X.F., Y.L., and C.C.; resources, L.L., J.Z., and X.Z.; funding acquisition, X.Z. and J.Y.; supervision, C.C., X.Z., and J.Y. Declaration of interests The authors declare no competing interests. Declaration of generative AI and AI-assisted technologies in the writing process During the preparation of this work, the authors used GPT4 to check grammar and enhance the readability of this review. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.