Medical Context Distorts Decisions in Clinical Vision Language Models
David Restrepo, Ira Ktena, Maria Vakalopoulou, Stergios Christodoulidis, Enzo Ferrante · 2026
Read Medical Context Distorts Decisions in Clinical Vision Language Models by David Restrepo, Ira Ktena, Maria Vakalopoulou, Stergios Christodoulidis, Enzo Ferrante (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17436.txt.
Introduction
Restrepo and colleagues show that medical context distorts clinical vision-language models on chest X-ray tasks. Testing varied general and medical models on MIMIC-CXR, they find decisions dominated by text even when images are available, swayed by irrelevant clinical history, and flipped by minor prompt rewordings. Three failure modes follow: text over-reliance, spurious history reliance and prompt sensitivity. The authors call for explicit safeguards and stress-testing before clinical use.
Catalog record
| CREATOR | David Restrepo, Ira Ktena, Maria Vakalopoulou, Stergios Christodoulidis, Enzo Ferrante |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34457 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17436
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Medical Context Distorts Decisions in Clinical Vision Language Models David Restrepo, Ira Ktena, Maria Vakalopoulou, Stergios Christodoulidis, Enzo Ferrante Abstract Vision-language models (VLMs) are increasingly proposed for clinical decision support, yet their reliability in real-world scenarios that require integrating both visual and textual context from medical records remains poorly characterized. This paper identifies three failure modes: (1) modality over-reliance on text over images, (2) spurious reliance on irrelevant clinical history, and (3) prompt sensitivity across semantically equivalent inputs. We evaluate a diverse set of general-domain and medically-tuned open and closed VLMs on chest x-ray tasks using MIMIC-CXR. By systematically manipulating image–text alignment, clinical history, and prompt formulations, we found that VLM decisions are dominated by the text modality, even when visual evidence is available. Moreover, we observed that VLMs are heavily influenced by irrelevant reports, while minor prompt changes can reverse correct image-based predictions. Our findings underscore the need for explicit safeguards and stress-testing before considering the use of these models in clinical practice. Introduction Foundation vision-language models (VLMs) are increasingly being proposed as clinical decision-support systems, leveraging joint reasoning over medical images and free-text context such as radiology reports [ 1 , 2 , 3 ] , clinical history [ 4 ] , and metadata [ 5 , 6 ] . Recent medically adapted and frontier-scale models have demonstrated strong benchmark performance [ 7 ] . However, clinical deployment differs from benchmark evaluation: real-world medical decisions are made under noisy, and often imperfect documentation streams, where textual inputs may be incomplete, inconsistent, or biased due to factors such as clinician fatigue [ 8 ] , prior diagnostic assumptions, and system-level errors [ 9 ] , potentially leading to suboptimal or misleading interpretations. Previous works suggest that various multimodal architectures do not integrate modalities symmetrically, but instead exhibit modality collapse, where decisions are dominated by language priors even when visual evidence is available [ 10 , 11 , 12 , 13 ] . This behavior is not solely a consequence of architectural design, but is strongly influenced by pretraining objectives and training data composition, highly composed by text-only tasks and imbalanced multimodal supervision that can bias models towards text signals [ 14 , 10 ] . As a result, models may develop an inductive bias toward textual signals, and under-utilize visual evidence unless explicitly required. Recent work has gone further, showing that frontier VLMs can achieve top-tier scores on multimodal exams [ 15 ] and medical multimodal benchmarks even when no image is provided, fabricating plausible visual descriptions from textual cues alone, a phenomenon termed mirage reasoning [ 16 ] . This phenomenon is particularly concerning in medicine, where irrelevant, inconsistent, or outdated text may override critical radiographic findings. Modern clinical AI systems increasingly incorporate retrieval-augmented generation (RAG) [ 17 ] and agentic pipelines [ 18 ] that automatically inject additional patient context from external sources. While retrieval is intended to improve factual grounding, it also introduces a new safety risk: models must correctly filter irrelevant, contradictory, or temporally distant information. In healthcare, prior reports may contain unrelated anatomies or temporally misaligned diagnoses, such irrelevant history can systematically bias current decisions. Additionally, VLMs have demonstrated sensitivity to the input prompt [ 19 ] , and predictions may flip under semantically equivalent prompt formulations [ 20 ] , compromising clinical reliability. Prompt sensitivity and inconsistency are now recognized as measurable failure modes of large language models and agents [ 21 ] . In this work, we quantify how contextual inputs may distort clinical decisions in VLMs, focusing on modality over-reliance under image-text conflict, vulnerability to irrelevant or contradictory temporal history, and decision inconsistency under prompt reformulation. Building on diagnostic tools for measuring modality contribution [ 22 , 23 , 24 ] , we conduct an experimental study on chest radiography, framed as a disease phenotyping task. Disease phenotyping is a routine task in clinical workflows, where clinicians assign structured diagnostic codes from standardized vocabularies (such as the International Classification of Diseases (ICD)) to patient encounters, enabling systematic data retrieval and downstream analysis within EHR systems. In our case the phenotyping is framed as a binary task (abnormal vs normal) given an x-ray image and text report, where abnormal indicates the presence of one of five thoracic pathologies (Atelectasis, Cardiomegaly, Consolidation, Edema, and Pleural Effusion) following chexpert classification [ 25 ] . Our experiments are conducted on a balanced subset of the MIMIC-CXR dataset [ 26 ] , where each case is restricted to a single condition (i.e., exactly one label per case), and span a diverse set of VLM families, including general-domain open architectures such as LLaVA [ 27 ] , Qwen2.5-VL [ 28 ] , Janus-Pro [ 29 ] , and Llama-3 [ 30 ] , as well as medically adapted models such as MedGemma [ 31 ] . To contextualize these findings against frontier-scale systems, we also include proprietary models such as GPT-5 [ 32 ] and Gemini [ 33 ] . Finally, we quantify robustness failures using the Negative Flip Rate (NFR) [ 34 ] , which measures how often correct baseline predictions become incorrect under contextual perturbations. Results Figure 1: Main Results Overview. (A) Modality Over-Reliance: We measure model performance under image-text conflict (A.1) and Negative Flip Rate (NFR) by modality shift (A.2). (B) Temporal Context Vulnerability: We evaluate the impact of irrelevant history length on accuracy (B.1) and NFR (B.2). (C) Semantic Prompt Fragility: Finally, we analyze inter-prompt agreement (Fleiss’ Kappa) for modality shift (C.1) and multi-history (C.2) experiments. Higher is better for ACC and Kappa; lower is better for NFR. Modality Dominance: Text Overrides Images To disentangle the contribution of each modality, we applied the Selective Modality Shifting protocol, which creates controlled conflicts between image and text by pairing radiographs with clinical histories from patients with opposing diagnoses or by directly removing one modality (see Methods - Experimental Design 1). Across all evaluated models, perturbing the text modality was substantially more damaging than perturbing the image modality (Figure 1 (A.1) ), indicating that predictions were primarily influenced by the clinical text rather than to the radiograph itself. This pattern was consistent across model families. Under the multimodal setting ("No Shift"), accuracies ranged from 0.66 (Janus-Pro) to 0.83 (GPT-5, Gemini 3 Pro), with frontier closed models achieving the strongest overall performance by a small margin. However, once conflicting text ("Text Shift") was introduced, accuracy collapsed, dropping to 0.17-0.34, below random chance for all evaluated models. The strongest frontier system dropped the accuracy from 0.83 to 0.18 for GPT-5, and from 0.83 to 0.17 for Gemini 3 Pro. Likewise, strong open models such as Qwen-2 VL and LLaVA 1.5 fell from 0.81 to 0.20 and 0.78 to 0.20, respectively, showing a strong text preference. By contrast, replacing the image while keeping the original text ("Image Shift") produced only minor changes in accuracy for most models. Qwen-2 VL remained essentially unchanged (0.81 to 0.80), LLaVA 1.5 remained at 0.78, GPT-5 changed only from 0.83 to 0.82, and Gemini 3 Pro from 0.83 to 0.82. Even the medically-adapted models, were far more robust to image perturbation than to text perturbation (MedGemma: 0.76 to 0.72 vs. 0.76 to 0.28; MedGemma 1.5: 0.79 to 0.74 vs. 0.79 to 0.26). This gap