On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?
Raza Imam(🖂), Rufael Marew, Mohammad Yaqub · 2025
Read On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable? by Raza Imam(🖂), Rufael Marew, Mohammad Yaqub (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15425.txt.
Introduction
Raza Imam and co-authors probe how medical vision language models handle dirty images in a 2025 paper. These models read scans and text together and score well on clean sets. Real clinic images, though, carry noise and artifacts from capture. Almost no one had tested how badly such flaws hurt the models.
The team built a benchmark that corrupts many medical image sets in set ways. Five leading models were tested across five imaging types. All degraded badly under corruption and traded skill across domains. A tuned encoder with light adaptation restored much of the lost ground.
Catalog record
| CREATOR | Raza Imam(🖂), Rufael Marew, Mohammad Yaqub |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15425
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable? Raza Imam(🖂), Rufael Marew, Mohammad Yaqub Abstract Medical Vision-Language Models (MVLMs) have achieved par excellence generalization in medical image analysis, yet their performance under noisy, corrupted conditions remains largely untested. Clinical imaging is inherently susceptible to acquisition artifacts and noise; however, existing evaluations predominantly assess generally clean datasets, overlooking robustness— i.e. , the model’s ability to perform under real-world distortions. To address this gap, we first introduce MediMeta-C, a corruption benchmark that systematically applies several perturbations across multiple medical imaging datasets. Combined with MedMNIST-C, this establishes a comprehensive robustness evaluation framework for MVLMs. We further propose ℝ \mathbb{R} obustMedCLIP, a visual encoder adaptation of a pretrained MVLM that incorporates few-shot tuning to enhance resilience against corruptions. Through extensive experiments, we benchmark 5 major MVLMs across 5 medical imaging modalities, revealing that existing models exhibit severe degradation under corruption and struggle with domain-modality tradeoffs. Our findings highlight the necessity of diverse training and robust adaptation strategies, demonstrating that efficient low-rank adaptation when paired with few-shot tuning, improves robustness while preserving generalization across modalities. † † footnotetext: Dataset and Code is available at: Github † † footnotetext: Accepted at: Medical Image Understanding and Analysis (MIUA) 2025 † † footnotetext: Corresponding Author: Raza Imam 🖂( raza.imam @mbzuai.ac.ae) 1 Introduction In recent years, Medical Vision-Language Models (MVLMs) have emerged as powerful tools for analyzing medical imaging data by leveraging large-scale multimodal learning [ 26 , 29 , 18 ] . These models have demonstrated impressive accuracy in zero-shot and few-shot medical image classification, making them promising candidates for real-world deployment. However, despite improvements in generalization accuracy, the robustness of MVLMs under real-world distribution shifts, from the point of corruptions, remains largely unexplored. Clinical imaging in practice is often affected by artifacts and noise introduced during acquisition and preprocessing, which can significantly degrade model performance. Existing evaluations [ 30 , 2 , 4 ] predominantly focus on clean datasets, overlooking the impact of such corruptions. Without systematic robustness assessment, the reliability of MVLMs in practical medical scenarios remains uncertain, raising concerns about their safety and effectiveness in clinical decision-making. Although datasets such as CheXpert [ 14 ] and MedMNIST [ 3 ] are carefully curated to ensure high-quality images through fixed resolutions and rigorous normalization, they fail to capture the range of corruptions and distribution shifts encountered in real-world clinical settings. Inspired by ImageNet-C [ 10 ] , MedMNIST-C [ 5 ] was proposed to introduce controlled distortions. However, its reliance on low-resolution MedMNIST data, use of modality-specific corruptions, and disregard for the inherently lower high-frequency content of medical images limit its ability to fully represent authentic imaging challenges [ 28 ] . This highlights the need for a more comprehensive corruption benchmark that effectively evaluates the robustness of MVLMs and other medical AI models. Table 1 : Overview of dataset statistics for MediMeta [ 27 ] and MedMNIST [ 3 ] , covering the common imaging modalities analyzed in this study. #Val/Test represents the number of validation and test samples. Extended statistics are provided in Appendix . MediMeta MedMNIST Modality \downarrow Data Name #Val/Test Description Data Name #Val/Test Description Cell Microscopy PBC 1,709/3,149 Blood cells BloodMNIST 1,712/3,421 Blood cells Breast Imaging Mammo 214/326 Calcifications BreastMNIST 78/156 Breast tumors Chest X-ray Pneumonia 817/624 Lung infection PneumoniaMNIST 524/624 Lung infection Fundoscopy Fundus 640/640 Eye diseases RetinaMNIST 120/400 Eye diseases Retinal OCT OCT 16,694/1,000 Retinal layers OCTMNIST 10,832/1,000 Retinal layers A critical aspect of robustness evaluation is understanding how different types and severities of corruptions impact MVLM performance across various medical imaging modalities (Table 1 ). Traditional learning models typically rely on extensive dataset curation and domain-specific adaptations to improve resilience [ 17 ] . In contrast, MVLMs introduced a paradigm where contrastive learning could play a pivotal role in addressing robustness challenges. Given their strong generalization capabilities, it is necessary to investigate whether contrastively learned pretrained MVLMs, when effectively combined with robust adaptation techniques, can mitigate the impact of corruptions and enhance MVLMs reliability. These challenges motivate us to answer the following: Research Questions Q1. Do MVLMs maintain performance when facing visual corruptions? Q2. How do corruption types and their severities affect MVLM generalization across clinical modalities? Q3. Do accurate MVLMs directly correlate with robustness against corruptions and distribution shifts? Q4. How can few-shot sampling be efficiently infused with the presence of strong MVLMs to address generalization despite corruptions? To answer these research questions, we introduce MediMeta-C , a corruption benchmark specifically designed for medical imaging. By combining MediMeta-C with MedMNIST-C [ 5 ] , we establish a comprehensive evaluation framework to assess model robustness across multiple imaging modalities. Furthermore, we propose \mathbb{R} obustMedCLIP , a novel adaptation of a pretrained MVLM (such as BioMedCLIP [ 29 ] and MedCLIP [ 26 ] ) that incorporates few-shot fine-tuning to enhance performance under corrupted conditions. Through extensive experiments, we benchmark MVLMs against a range of seven corruptions, providing valuable insights into their resilience and adaptability in real-world medical settings. Overall, our key contributions are summarized as follows: 1. MediMeta-C Dataset : A corruption classification benchmark for evaluation that applies 7 systematic perturbations to 5 medical imaging datasets to simulate real-world OOD shifts. 2. Corruption MVLM Benchmarking : A unified evaluation framework combining MediMeta-C and MedMNIST-C to analyze the robustness of 5 major MVLMs across 5 medical imaging modalities across classification tasks. 3. \mathbb{R} obustMedCLIP : A robust adaptation of pretrained MVLM that integrates efficient few-shot-tuning to enhance visual representations to achieve better generalization and robustness against corruptions. 4. Extensive Evaluations : A systematic study assessing the impact of various corruption types and severities across multiple MVLMs robustness, while evaluating the true generality of existing MVLMs. 5. Datasets and Code : We release our benchmark dataset and APIs, promoting standardized robustness evaluation practices in medical AI research. Figure 1 : Corrupted samples from our MediMeta-C dataset. The y-axis shows dataset names by modality and the x-axis displays corruption types at a fixed severity level. (a) MediMeta-C DCT frequency (b) MedMNIST-C DCT frequency Figure 2 : Comparison of average DCT frequency distributions across datasets. Medical images generally exhibit higher density of low-frequency content compared to natural images and vice-versa [ 28 ] . Among the two, MediMeta-C (a) more clearly demonstrates this assumption than MedMNIST-C (b). Figure 3 : t-SNE visualization of the clean and corrupted feature distributions , showing how the distributions shift occur at the latent-level due to introduced corruption. MediMeta-C’s Corrupted features differ notably than MediMeta’s Clean features. Here RN50 backbone is used to extract features. 2 Background A. Vision-Language Models in Medical Imaging: The adaptation of vision-language models to the medical domain has advanced by modifying dual-encoder architectures. For example, CLIP [ 23 ] has been fine-tuned on medical image-text pairs, resulting in variants such as MedCLIP [ 26 ] and BioMedCLIP [ 29 ] that employ domain-specific tokenization and contrastive loss adjustments. However, training on clean, curated datasets leaves these models vulnerable to the distribution shifts and noise present in real-world clinical imaging. This vulnerability underscores the need for corruption-specific adaptations—such as fine-tuning on distorted samples and robust weak or unsupervised strategies [ 13 ] —to improve resilience against imaging artifacts and ensure