Multi-Modal One-Shot Federated Ensemble Learning for Medical Data with Vision Large Language Model

Naibo Wang [email protected], Yuchen Deng [email protected], Shichen Fan [email protected], Jianwei Yin [email protected], See-Kiong Ng [email protected] · 2025

Read Multi-Modal One-Shot Federated Ensemble Learning for Medical Data with Vision Large Language Model by Naibo Wang [email protected], Yuchen Deng [email protected], Shichen Fan [email protected], Jianwei Yin [email protected], See-Kiong Ng [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03292.txt.

Introduction

Five authors build a one round federated learning system for medical data in this 2025 paper. Hospitals want to train shared models without moving patient data, but standard federated learning needs many rounds of messages, which strains slow networks.

Their framework, FedMME, trains client models locally, then merges them in a single round by ensemble voting, which also helps when each site's data differ. It goes further by using both images and text: a vision language model writes text features from scans to enrich the data. One shot, multi-modal training could bring collaborative medical AI within reach of bandwidth poor settings.

Catalog record

CREATORNaibo Wang [email protected], Yuchen Deng [email protected], Shichen Fan [email protected], Jianwei Yin [email protected], See-Kiong Ng [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-03292

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Modal One-Shot Federated Ensemble Learning for Medical Data with Vision Large Language Model Naibo Wang [email protected], Yuchen Deng [email protected], Shichen Fan [email protected], Jianwei Yin [email protected], See-Kiong Ng [email protected] Abstract Federated learning (FL) has attracted considerable interest in the medical domain due to its capacity to facilitate collaborative model training while maintaining data privacy. However, conventional FL methods typically necessitate multiple communication rounds, leading to significant communication overhead and delays, especially in environments with limited bandwidth. One-shot federated learning addresses these issues by conducting model training and aggregation in a single communication round, thereby reducing communication costs while preserving privacy. Among these, one-shot federated ensemble learning combines independently trained client models using ensemble techniques such as voting, further boosting performance in non-IID data scenarios. On the other hand, existing machine learning methods in healthcare predominantly use unimodal data (e.g., medical images or textual reports), which restricts their diagnostic accuracy and comprehensiveness. Therefore, the integration of multi-modal data is proposed to address these shortcomings. Additionally, vision large language models (vLLMs) have emerged as powerful tools due to their ability to interpret and generate textual descriptions from visual data, making them invaluable for creating textual reports from medical images. In this paper, we introduce FedMME , an innovative one-shot multi-modal federated ensemble learning framework that utilizes multi-modal data for medical image analysis. Specifically, FedMME capitalizes on vision large language models to produce textual reports from medical images, employs a BERT model to extract textual features from these reports, and amalgamates these features with visual features to improve diagnostic accuracy. Experimental results show that our method demonstrated superior performance compared to existing one-shot federated learning methods in healthcare scenarios across four datasets with various data distributions. For instance, it surpasses existing one-shot federated learning approaches by more than 17.5% in accuracy on the RSNA dataset when applying a Dirichlet distribution with ( α \alpha = 0.3). 1 Introduction Federated learning ( McMahan et al. 2017 ) is a paradigm that allows multiple distributed clients to collaboratively train a global model without sharing their local data ( Wang et al. 2022 ; Balkus et al. 2022 ) . This approach encompasses various scenarios such as parallel federated learning ( Li et al. 2020 ) , sequential federated learning ( Wang et al. 2024a ) , and federated ensemble learning ( Wang et al. 2023 ) . As depicted in Fig. 1 , federated ensemble learning involves each participant independently training a model on their respective local datasets. Subsequently, these models are combined on a central server to perform ensemble learning techniques, including voting ( Raza 2019 ) or stacking ( Cui et al. 2021 ) . Federated ensemble learning has gained substantial popularity in privacy-sensitive areas including finance ( Gadekallu et al. 2021 ) and edge computing ( Alam et al. 2023 ) . Recently, federated learning has gained significant attention in the medical field ( Pfitzner et al. 2021 ; Sheller et al. 2020 ) because it enables the development of more comprehensive and accurate models by integrating data from different medical institutions without compromising data privacy. In contrast, traditional centralized machine learning ( Drainakis et al. 2020 ) requires data aggregation on a central server for training, which presents substantial privacy, legal, and security challenges when handling sensitive medical data. Federated learning mitigates these challenges by allowing multiple medical institutions to perform model training in their local environments, exchanging only model updates rather than raw data. This method effectively safeguards data privacy and produces a robust global model or ensemble team that surpasses any model developed by individual institutions alone ( Abaoud et al. 2023 ) . Despite the effectiveness of federated learning in protecting privacy and enabling distributed training for healthcare, it faces challenges related to substantial communication overhead and time delays, especially in environments with limited network bandwidth ( Kim et al. 2024 ) . To address the risks of attack and high communication costs associated with traditional federated learning, the concept of one-shot federated learning has been proposed ( Guha et al. 2018 ; Dennis et al. 2021 ) . This communication-efficient method conducts model training and aggregation in a single communication round, significantly reducing overhead and enhancing data privacy ( Zhang et al. 2022 ) . Consequently, employing one-shot federated ensemble learning for the development of healthcare applications holds considerable promise. In the medical field, federated learning methods commonly rely on unimodal data for training, such as solely using medical images ( Chakravarty et al. 2021 ; Ke et al. 2021 ; Kang et al. 2023 ) or textual reports ( Sui et al. 2020 ) . However, this unimodal approach results in a significant loss of valuable information. For example, analyzing medical images without the context provided by patient history or textual reports can lead to incomplete diagnostics or overlook critical correlations between visual cues and underlying conditions ( Reale-Nosei et al. 2024 ) . This information bottleneck restricts a comprehensive understanding and accurate interpretation of a patient’s health status. Figure 1: Overview of the one-shot federated ensemble framework. (I) Federated learning process: each client trains a local model using its private dataset and transmits the model to the central server. (II) Global ensemble learning: the server aggregates model outputs from all clients using a voting mechanism to produce the final decision. Consider a scenario in which a CT scan detects an anomaly that initially appears benign. Without incorporating textual data, such as the patient’s symptoms or genetic predisposition from their medical history, physicians may overlook the need for further investigation of a potentially serious condition ( Rakowski et al. 2006 ) . Similarly, when relying solely on textual reports, subtle indicators in the imagery might be entirely missed ( Wu et al. 2024a ) . By integrating various data types—such as medical images, textual reports, and sensory data from wearable devices—healthcare models can obtain a more comprehensive understanding of patient health. This integrative approach enables the cross-validation and enrichment of datasets ( Singh et al. 2020 ) , leading to insights that cannot be achieved through the analysis of unimodal data alone. Therefore, incorporating multi-modal data into one-shot federated learning frameworks for healthcare applications is imperative to achieve more accurate and reliable diagnostic and predictive outcomes. In recent years, large language models (LLMs) have demonstrated substantial potential in natural language processing ( Brown et al. 2020 ; Chowdhery et al. 2023 ; Hoffmann et al. 2022 ) and multi-modal ( Hu et al. 2024 ; Ge et al. 2024 ) tasks. These models excel in generating high-quality textual reports and in extracting and integrating complex semantic information from both textual and visual data, particularly when augmented with their vision-based counterparts, such as Llama-3.2-11B-Vision ( Chi et al. 2024 ) . The advanced capabilities of LLMs offer new opportunities for multi-modal analysis in the medical domain ( Ghosh et al. 2024 ) . A medical analysis framework incorporating LLMs can produce detailed imaging reports, thereby enhancing diagnostic accuracy and interoperability ( Waldock et al. 2024 ) . Hence, utilizing these advanced vision large language models within medical settings could substantially improve therapeutic effects and streamline clinical workflows. In this paper, we introduce FedMME , an innovative Fed erated

Read the complete text