MAFM³: Modular Adaptation of Foundation Models for Multi-Modal Medical AI

Mohammad Areeb Qazi, Munachiso S Nwadike, Ibrahim Almakky, Mohammad Yaqub, Numan Saeed · 2025

Read MAFM³: Modular Adaptation of Foundation Models for Multi-Modal Medical AI by Mohammad Areeb Qazi, Munachiso S Nwadike, Ibrahim Almakky, Mohammad Yaqub, Numan Saeed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11212.txt.

Introduction

Mohammad Areeb Qazi, Munachiso Nwadike, and their colleagues present a framework that lets one medical foundation model grow into many tasks through small add-on modules. Medical imaging data are scarce, so training a fresh foundation model for each scan type or task is rarely feasible. The modules act as skill sets that switch on at use time to fit the input and clinical goal.

The team adapted a chest CT foundation model first trained for classification into prognosis and segmentation modules. Results improved on both tasks. Adding PET scans lifted a key overlap score by about five percent over baselines, which supports the unified, expandable design.

Catalog record

CREATORMohammad Areeb Qazi, Munachiso S Nwadike, Ibrahim Almakky, Mohammad Yaqub, Numan Saeed
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11212

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MAFM³: Modular Adaptation of Foundation Models for Multi-Modal Medical AI Mohammad Areeb Qazi, Munachiso S Nwadike, Ibrahim Almakky, Mohammad Yaqub, Numan Saeed Abstract Foundational models are trained on extensive datasets to capture the general trends of a domain. However, in medical imaging, the scarcity of data makes pre-training for every domain, modality, or task challenging. Instead of building separate models, we propose MAFM³ (Modular Adaptation of Foundation Models for Multi-Modal Medical AI), a framework that enables a single foundation model to expand into diverse domains, tasks, and modalities through lightweight modular components. These components serve as specialized skill sets that allow the system to flexibly activate the appropriate capability at the inference time, depending on the input type or clinical objective. Unlike conventional adaptation methods that treat each new task or modality in isolation, MAFM³ provides a unified and expandable framework for efficient multitask and multimodality adaptation. Empirically, we validate our approach by adapting a chest CT foundation model initially trained for classification into prognosis and segmentation modules. Our results show improved performance on both tasks. Furthermore, by incorporating PET scans, MAFM³ achieved an improvement in the Dice score 5% compared to the respective baselines. These findings establish that foundation models, when equipped with modular components, are not inherently constrained to their initial training scope but can evolve into multitask, multimodality systems for medical imaging. The code implementation of this work can be found at Code . 1 Introduction Medical imaging plays a crucial role in modern healthcare, providing essential visual insights that are key to diagnosing, monitoring, and treating a wide range of conditions [ 18 , 9 , 24 ] . Despite its importance, collecting and curating medical imaging data is highly resource-intensive, with large hospitals generating approximately 100 terabytes of imaging data per year, which require significant storage, processing power, and infrastructure to effectively manage [ 7 , 14 , 20 ] . As a result, it becomes difficult to share medical data. Although these challenges limit the feasibility of training distinct models for each task or modality, they also highlight the potential of modular adaptation strategies that can extend a single foundation model to multiple specialized functions without requiring extensive retraining. Now, with the introduction of Medical Foundational Models (FM) pre-trained on large-scale data, deep learning systems have demonstrated exceptional performance within their trained domains [ 39 , 35 , 10 , 17 , 38 ] . These models are typically customized to their specific domains by leveraging specialized datasets, architectures, and training objectives that align with medical imaging tasks. For example, they are trained on radiology reports, CT scans, MRIs, or pathology slides, enabling them to capture domain-specific patterns and features that general-purpose models may overlook [ 35 , 36 , 32 ] . However, the diversity of medical imaging modalities presents a significant challenge for FM. For example, CT provides detailed cross-sectional images of anatomical structures, while PET visualizes metabolic processes, highlighting functional aspects of tissues. Each modality captures different aspects of human anatomy and physiology, leading to different data distributions. This variability can hinder the generalizability of FM in all types of imaging, as they may struggle to perform consistently well on modalities not represented in their training data [ 28 ] . However, rather than viewing these models as static systems tied to a narrow scope, they can be seen as flexible backbones that, when equipped with lightweight modular components, can serve as the basis for multitask, multimodal medical AI systems. Figure 1 : Overview of the objective behind MAFM 3 . A clinician provides diverse medical data sources such as CT scans, PET scans, and electronic health records (EHR). The Medical Foundational Model serves as a central knowledge base, from which task-specific experts (classification, prognosis, segmentation) can be activated depending on the input and clinical requirement. This modular design enables flexible adaptation of a single foundational model to multiple modalities and tasks, supporting generalist medical AI applications. Training FM for each medical imaging modality is not feasible due to the vast data requirements, which are often unavailable in healthcare. As a result, adaptation techniques have been used to extend models across domains, classes, and tasks [ 25 , 21 , 37 , 6 , 31 ] . Studies have demonstrated the adaptation of classification models to incorporate new classes while maintaining their original task [ 22 , 37 ] , segmentation models to generalize across different anatomical regions [ 6 ] , and report generation models to generate structured results for diverse medical findings [ 31 ] . Furthermore, research has shown that models can be adapted to integrate multiple modalities over the same anatomical region [ 29 , 25 ] . Despite these advancements, existing approaches primarily focus on continual adaptation within a single task or domain, lacking a unified framework to extend foundation models across new tasks, imaging modalities, and anatomical regions. Moreover, unlike pre-training on curated datasets, real-world medical imaging data often varies in resolution and quality, posing additional challenges in adapting FM to diverse tasks and domains. Beyond continual learning, a broader perspective is to treat each new module as a capacity unit that can be selectively activated depending on the type of input and the clinical objective. This creates a efficient multitask and multimodal workflow, avoiding the need to train specialized model ensembles and aligning with real-world scenarios where different imaging modalities and tasks coexist. To this end, we propose MAFM³ (Modular Adaptation of Foundation Models for Multimodal Medical AI), a framework that enables a single FM initially trained for a specific modality, task, or anatomical region to expand with modular components that specialize in new tasks, modalities, and regions. Unlike conventional approaches to continual learning or domain adaptation, which often focus narrowly on sequential extension, MAFM³ emphasizes modularity and orchestration, allowing the foundation to activate the appropriate component during inference flexibly. This makes the system scalable across diverse adaptation scenarios and efficient and practical for deployment in medical imaging workflows. Figure 1 illustrates an overview of the proposed framework, showing how a single foundation model can be expanded with modular components and selectively activated to handle various tasks and modalities. Our contributions can be summarized as follows. • We introduce MAFM³, a modular framework that enables foundation models to be efficiently extended across new tasks, modalities, and anatomical regions without training separate models. • We propose a method for adapting foundation models to arbitrary resolution sizes, allowing seamless generalization beyond their original training constraints. • We show that modular extensions can be selectively activated at inference, enabling an efficient multi-task, multi-modality workflow that reduces redundancy and preserves performance across prior tasks. 2 Related Works Deep learning has significantly advanced medical imaging by improving diagnostic accuracy and workflow efficiency in various clinical applications [ 18 , 9 , 24 ] . However, the development of high-performing models often depends on access to large curated datasets, which are challenging to acquire due to storage, privacy, and infrastructure constraints [ 7 , 14 , 20 ] . To address these limitations, researchers have explored medical foundation models, continual learning techniques, and more recently modular adaptation strategies. In the following, we summarize the most relevant works. 2.1 Medical Foundation

Read the complete text