Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights
Cosmin-Andrei Hatfaludi, Alex Serban · 2025
Read Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights by Cosmin-Andrei Hatfaludi, Alex Serban (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05142.txt.
Introduction
Training large foundation models on data that cannot leave hospitals or companies is the promise of federated learning. Cosmin-Andrei Hatfaludi and one co-author survey how the two fields are being combined in a 2025 review paper. No unified survey existed, so they build a taxonomy that follows the development life cycle of such systems.
From an initial pool of thousands of articles, they review a few hundred in depth and compare methods technically. Topics include self-supervised learning, fine-tuning, distillation, and transfer learning. Practical guidance focuses on healthcare as a case study, where private data and large models could combine with significant impact.
Catalog record
| CREATOR | Cosmin-Andrei Hatfaludi, Alex Serban |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-05142
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights Cosmin-Andrei Hatfaludi, Alex Serban Abstract Federated learning has the potential to unlock siloed data and distributed resources by enabling collaborative model training without sharing private data. As more complex foundational models gain widespread use, the need to expand training resources and integrate privately owned data grows as well. In this article, we explore the intersection of federated learning and foundational models, aiming to identify, categorize, and characterize technical methods that integrate the two paradigms. As a unified survey is currently unavailable, we present a literature survey structured around a novel taxonomy that follows the development life-cycle stages, along with a technical comparison of available methods. Additionally, we provide practical insights and guidelines for implementing and evolving these methods, with a specific focus on the healthcare domain as a case study, where the potential impact of federated learning and foundational models is considered significant. Our survey covers multiple intersecting topics, including but not limited to federated learning, self-supervised learning, fine-tuning, distillation, and transfer learning. Initially, we retrieved and reviewed a set of over 4,200 articles. This collection was narrowed to more than 250 thoroughly reviewed articles through inclusion criteria, featuring 42 unique methods. The methods were used to construct the taxonomy and enabled their comparison based on complexity, efficiency, and scalability. We present these results as a self-contained overview that not only summarizes the state of the field but also provides insights into the practical aspects of adopting, evolving, and integrating foundational models with federated learning. 0 0 footnotetext: ©PeerJ, Inc. 2012–2025. Public user content licensed under CC BY 4.0. This is a preprint version of an article published in PeerJ Computer Science. DOI: https://doi.org/10.7717/peerj-cs.2993 Introduction FL ( FL ) allows multiple parties to collaboratively train ML ( ML ) models without exchanging or transferring private data ( Zhang et al., 2021 ) . Its main goal is to improve ML algorithms by integrating siloed data, which would otherwise remain isolated and underused. For example, using patient data from different hospitals to improve diagnostic ML algorithms would be challenging without FL ( Pfitzner et al., 2021 ) . As ML models and datasets grow, particularly with the development of FM ( Bommasani et al., 2021 ) , training them requires more computational resources and diverse data. However, high-quality data is often siloed due to privacy concerns. In this context, it is valuable to explore how FL can enable new data sources for training FM , improve resource sharing, or use pre-trained FM , to develop robust collaborative ML models. While both FM and FL aim to improve ML models, their approaches differ fundamentally. FM are trained centrally using large and diverse datasets, and fine-tuned for various tasks ( Awais et al., 2025 ; Chang et al., 2024 ; Liang et al., 2024 ) . In contrast, FL allows multiple parties to train specialized models without centralizing data, preserving privacy. Their integration faces technical challenges such as bandwidth management, training latency, and efficient data transmission ( Zhang and Metaxas, 2023 ) , as well as legal and organizational challenges such as finding incentives to defining IP ownership for participating nodes ( Woisetschläger et al., 2024 ) . Navigating the intersection of FM and FL remains challenging due to the absence of comprehensive reviews. While existing literature addresses the challenges of integrating FM and FL Zhuang et al., 2023 , examines their integration at various development stages Woisetschläger et al., 2024 ; Kang et al., 2023 , and categorizes existing literature Ren et al., 2024 ; Li et al., 2024b , an integrated an self-contained review is still missing. For example, Zhuang et al., 2023 lack a structured taxonomy or technical analysis of methods, Ren et al., 2024 and Li et al., 2024b propose taxonomies, but many of their categories overlap, making it hard to uniquely position new methods, while Woisetschläger et al., 2024 and Kang et al., 2023 focus on specific stages such as training. To address these gaps, we aim to provide a more comprehensive survey structured around a non-overlapping taxonomy that follows the development life-cycle stages. Furthermore, we aim to provide a comprehensive technical comparison of the available methods, together with practical insights into their implementation and quality attributes, and study their application in specific use-cases. This will guide future researchers and practitioners understand, apply, and evolve these methods. Overall, the contributions of this article are as follows: • We present a comprehensive literature survey that aims to cover the majority of articles at the intersection of FM and FL . • We categorize the articles using a novel taxonomy based on the stage where FM are used (e.g., pretrain or inference) and the methods used at each stage. • We compare the methods based on complexity, efficiency and scalability. • We discuss practical aspects of applying the methods presented and provide guidelines and future research directions, focusing on the application of FL and FM in the healthcare domain. The remainder of this article is organised as follows. We start with background information and related work ( Background and Related Work ), followed by the methodology and the taxonomy used for the literature survey ( Survey Methodology ). We then discuss the selected articles ( Methods ) and practical aspects of the methods ( Practical perspectives ). The paper concludes with a discussion of study limitations ( Limitations ), findings ( Discussion ), followed by conclusions ( Conclusions ). Prerequisite information about FL, FMs, fine-tuning, and KD is provided in the Appendix . Background and Related Work Several comprehensive surveys have been published covering FL or FM , addressing applications in multiple domains including healthcare. For example, Zhang et al., 2021 present a survey of FL , categorizing and summarizing various methods along with their advantages and disadvantages. Liu et al., 2024a build upon this survey, incorporating more recent techniques and providing an extensive taxonomy for their classification. Additionally, several articles explore the challenges and opportunities in FL – e.g., ( Wen et al., 2023 ) – and examine different quality attributes of the algorithms, such as trustworthiness ( Zhang et al., 2024b ) , robustness ( Huang et al., 2024 ) , and fairness ( Ji et al., 2024 ) . In the medical domain, Guan et al., 2024 and Pfitzner et al., 2021 cover both the techniques used as well as benchmarks and datasets for FL . Likewise, numerous surveys address FM for various applications, including vision ( Awais et al., 2025 ) , language ( Chang et al., 2024 ) , and multi-modal FM ( Liang et al., 2024 ) . These surveys include methods for developing FM , comparative benchmarks, as well as the opportunities and risks associated with using FM ( Bommasani et al., 2021 ) . In the medical domain, Azad et al., 2023 present a comprehensive survey discussing algorithms, modalities, and various organs for which FM have been developed. Within this realm, FM are also called generalist models ( Moor et al., 2023 ) . Zhang and Metaxas, 2023 present a classification of the types of FM and generalist models, ranging from very broad, multi-modal, models, to very narrow, task-specific FM .