FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraints

Lishan Yang, Wei Emma Zhang, Nam Kha Nguyen, Po Hu, Yanjun Shu, Weitong Chen, Sim Mong Yuan · 2025

Read FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraints by Lishan Yang, Wei Emma Zhang, Nam Kha Nguyen, Po Hu, Yanjun Shu, Weitong Chen, Sim Mong Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06984.txt.

Introduction

Institutions fine-tuning large vision-language models together, despite missing data types and unequal computers, get a framework from Lishan Yang and six co-authors in a 2025 machine learning paper. Federated learning with low-rank adapters is efficient and privacy-aware. In practice, hospitals and transport operators hold data with missing modalities, and their hardware differs, producing adapters of imbalanced ranks.

No earlier work had tackled both problems together for federated vision-language models. Their framework, FediLoRA, is a lightweight aggregation scheme built on the observation that simple averaging plus structured editing can help both the shared global model and each site's personalized model. Results were strong across general and medical benchmark datasets, with extra healthcare experiments supporting real deployment. Code is released. The work targets settings where data cannot move and sensors cannot be trusted to be complete.

Catalog record

CREATORLishan Yang, Wei Emma Zhang, Nam Kha Nguyen, Po Hu, Yanjun Shu, Weitong Chen, Sim Mong Yuan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06984

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraints Lishan Yang, Wei Emma Zhang, Nam Kha Nguyen, Po Hu, Yanjun Shu, Weitong Chen, Sim Mong Yuan Abstract Federated Learning with LoRA fine-tuning offers an efficient and privacy-aware solution for institutions to collaboratively leverage their large datasets to train VLLMs. However, participating institutions often possess heterogeneous computational resources, resulting in imbalanced LoRA ranks, which pose a major challenge for effective collaboration. In addition, real-world applications in domains such as healthcare and transportation frequently suffer from missing modalities due to user mistakes or device failures, which significantly degrade global model performance in federated settings. To the best of our knowledge, no prior work has addressed these two challenges simultaneously in federated VLLMs. To tackle these issues, we propose FediLoRA , a lightweight federated LoRA aggregation framework that effectively mitigates the impact of missing modalities in heterogeneous environment. FediLoRA is explicitly motivated by the observation that simple averaging and structured editing can jointly benefit both global and personalized models. Our approach achieves strong performance across multiple general-domain and medical-domain benchmark datasets. Additional experiments on healthcare data further demonstrate that FediLoRA is well-suited for practical, real-world deployment scenarios. Our code is released at https://github.com/gotobcn8/FediLoRA. 1 Introduction Foundation Models (FMs) have demonstrated remarkable capacity to absorb such large-scale, heterogeneous data and exhibit extraordinary generalization abilities. Consequently, FMs have been deployed in various applications, including education Wen et al. (2024) , finance De Zarzà et al. (2023) , healthcare Wang et al. (2022) ; Chu et al. (2025) and so on. Figure 1 : Two challenges for current Federated VLLMs. (1) Left: Inconsistencies in clients’ local model configuration (heterogeneous rank), causing unstable and suboptimal aggregation. (2) Right: Missing modalities in different clients leads to poor multimodal representation learning and generalization. Reliable decision-making systems generally depend on multimodal inputs, for instance, medical applications may combine diagnostic text reports with CT or X-ray images. However, constructing powerful real-world multimodal models faces two major challenges due to strict privacy regulations. First, institution data are inherently isolated across hospitals, which possess distinct patient populations and private data distributions. Federated Learning (FL) McMahan et al. (2017) ; Yang et al. (2024a) provides a promising solution by enabling collaborative training across institutions without sharing raw data. Due to differences in local training environments, model inconsistency is commonly present in federated learning architectures. Second, multimodal data in the real world are often incomplete; certain modalities may be missing due to equipment constraints, inconsistent collection protocols, or human oversight Cismondi et al. (2013) . Such modality incompleteness can substantially degrade the performance and reliability of multimodal models, posing serious risks to the users, especially in clinical applications. The heterogeneity of LoRA in VLLMs . Federated Learning consists of a global model and multiple clients (e.g., hospitals). Fine-tuning personalized Vision–Large Language Models becomes increasingly popular and stable in the specific domains Savage et al. (2025) ; Wang et al. (2023) , clients often have many fine-tuning options to choose from when adapting models to their own data and computing resources. Parameter-Efficient Fine-Tuning (PEFT) methods such as adapters Houlsby et al. (2019) ; Sun et al. (2024b) ; Wu et al. (2024) , prefix-tuning Li and Liang (2021) ; Liu et al. (2022) , and Low-Rank Adaptation (LoRA) Hu et al. (2022b) have received growing attention. LoRA introduces trainable low-rank matrices to approximate gradient updates while keeping pretrained weights frozen. In practice, clients may choose different LoRA configurations to better fit their personalized data, training resources, or efficiency requirements. This naturally leads to heterogeneous LoRA settings, making it difficult for Multimodal FL to aggregate these structurally different client models. Existing works Zhang et al. (2024) ; Liu et al. (2025) ; Kuang et al. (2024) ; Sun et al. (2024a) ; Yang et al. (2024b) do not consider heterogeneous ranks. Although recent methods such as HetLoRA Cho et al. (2024) , FLoRA Wang et al. (2024a) , and FlexLoRA Bai et al. (2024) explore heterogeneous LoRA, they still lack effective mechanisms for handling our next challenge in multimodal data: missing modalities. Missing Modality in Federated Learning. In real-world settings, missing modalities arise easily and are often unavoidable. For example, hospitals possess multimodal patient data, yet certain modalities are often missing due to privacy constraints, equipment limitations, or data collection issues. Traditional multimodal FL methods mainly focus on model aggregation or label distribution Zong et al. (2021) ; Chen and Li (2022) ; Zeng et al. (2024) , but they do not address challenges for distributed missing modality. Prior approaches such as modality reconstruction Xiong et al. (2023) , contrastive learning Yu et al. (2023) , and prototype exchange Bao et al. (2024) attempt to mitigate missing modalities. However, when applied to foundation models, these methods often incur substantial computational overhead. As a result, they are impractical for real-world application. Therefore, there is a clear need for a lightweight yet effective solution to handle missing modalities in federated multimodal settings. In summary, the research effort of addressing missing modality in heterogeneous Federated LoRA, a common scenario in the era of large foundation models, is missing. To this end, we propose FediLoRA to fill this gap. Our method is motivated by the observation that the global model achieves comparable performance when trained on either full-modality or missing-modality settings (Section 2 ). We attribute this robustness to the averaging effect of FedAvg aggregation Collins et al. (2022) , the de facto standard aggregation method in federated learning, which aggregates client updates using a weighted average based on local data size. To extend its benefit to heterogeneous settings, we retain the core idea of FedAvg and adapt it to fit the heterogeneous LoRA and propose a dimension-wise reweighting mechanism. Despite the global model’s stability, we observe that client models suffer from a notable performance gap, particularly under missing-modality scenarios. This is due to both local data sparsity and FedAvg’s emphasis on optimizing global performance. Given our success in handling missing modalities on the global model, we further explore whether aligning local models to the global model could improve both global and clients’ performance. This leads us to adopt a simple yet effective solution: repairing local LoRA layers using their global counterparts. This layer-wise model editing strategy efficiently transfers useful global knowledge to clients without introducing additional training overhead ( Appendix D ). Our contributions are summarized as follows: • We introduce a layer-wise model editing technique that leverages global LoRA parameters to correct local components, thereby enhancing client model performance without incurring additional computational cost, and we provide theoretical guarantees for its effectiveness. • We propose a dimension-wise weighted aggregation and strategy that enables effective heterogeneous federated LoRA while preserving model robustness to missing modalities. • We conduct comprehensive experiments on three VLLM benchmarks to validate our approach. In addition, we further demonstrate its capability on one medical-domain dataset. 2 Motivation Averaging aggregation mitigates the effect of missing modality. While the global model and client models often exhibit significant performance gaps when using FedAvg (or its variants), due to FedAvg’s focus on global optimization, this observation motivates us to investigate whether averaging may help mitigate the adverse effects of missing modalities on the global server. Specifically, we hypothesize that

Read the complete text