When Background Matters: Breaking Medical Vision Language Models by Transferable Attack
Akash Ghosh, Subhadip Baidya, Sriparna Saha, Xiuying Chen, [0.5em] Indian Institute of Technology Patna, Indian Institute of Technology Kanpur, MBZUAI · 2026
Read When Background Matters: Breaking Medical Vision Language Models by Transferable Attack by Akash Ghosh, Subhadip Baidya, Sriparna Saha, Xiuying Chen, [0.5em] Indian Institute of Technology Patna, Indian Institute of Technology Kanpur, MBZUAI (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17318.txt.
Introduction
Ghosh, Baidya, Saha, and Chen attack medical vision-language models through the background of images, where doctors would not look for disease. Their MedFocusLeak method nudges non-diagnostic background pixels and distracts the model's attention, producing wrong yet plausible diagnoses that transfer across systems without visible distortion. Tested across six imaging types, it exposes a serious weakness in clinical AI reasoning and comes with new measures balancing attack success against image fidelity.
Catalog record
| CREATOR | Akash Ghosh, Subhadip Baidya, Sriparna Saha, Xiuying Chen, [0.5em] Indian Institute of Technology Patna, Indian Institute of Technology Kanpur, MBZUAI |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72101 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17318
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Background Matters: Breaking Medical Vision Language Models by Transferable Attack Akash Ghosh, Subhadip Baidya, Sriparna Saha, Xiuying Chen, [0.5em] Indian Institute of Technology Patna, Indian Institute of Technology Kanpur, MBZUAI Abstract Vision–Language Models (VLMs) are increasingly used in clinical diagnostics, yet their robustness to adversarial attacks remains largely unexplored, posing serious risks. Existing medical attacks focus on secondary objectives such as model stealing or adversarial fine-tuning, while transferable attacks from natural images introduce visible distortions that clinicians can easily detect. To address this, we propose MedFocusLeak, a highly transferable black-box multimodal attack that induces incorrect yet clinically plausible diagnoses while keeping perturbations imperceptible. The method injects coordinated perturbations into non-diagnostic background regions and employs an attention-distraction mechanism to shift the model’s focus away from pathological areas. Extensive evaluations across six medical imaging modalities show that MedFocusLeak achieves state-of-the-art performance, generating misleading yet realistic diagnostic outputs across diverse VLMs. We further introduce a unified evaluation framework with novel metrics that jointly capture attack success and image fidelity, revealing a critical weakness in the reasoning capabilities of modern clinical VLMs. The code associated with this project is available at MedFocusLeak . 1 Introduction VLMs are rapidly emerging as transformative tools in medical imaging, enabling interpretation of complex clinical scans and generation of expert-level diagnostic reports, summaries, and findings ( 37 ; 25 ; 20 ; 17 ; 10 ; 12 ; 11 ; 13 ; 15 ) . However, their safety and reliability in high-stakes clinical settings remain critical concern 16 ; 43 ; 38 ; 6 . While general-purpose VLMs such as GPT-4o ( 34 ) and Gemini ( 40 ) are known to be vulnerable to transferable adversarial attacks, often due to weaknesses inherited from their vision encoders, the risks posed to specialized medical VLMs are largely underexplored. Existing transferable attacks are less effective in the medical domain, as perturbations tend to be visually conspicuous on grayscale or narrow-palette medical images, limiting their practicality. Consequently, developing medical-specific, transferable attacks under realistic black-box assumptions remains an open and important research challenge. Recent studies on adversarial vulnerabilities of medical VLMs span model stealing, prompt-injection and jailbreak attacks, and data poisoning. Model-stealing approaches, e.g., ADA-STEAL ( 39 ) , aim to replicate model behavior using natural images but are limited by low output diversity and a lack of defensive considerations. Prompt-injection and jailbreak attacks ( 27 ; 36 ) reveal safety risks but typically rely on white-box or controlled settings and focus on harmful content generation rather than compromising diagnostic reasoning. Data-poisoning methods ( 41 ) further expose vulnerabilities but do not yield transferable attacks at inference time. Crucially, none of these approaches produces stealthy, transferable black-box attacks that directly undermine diagnostic integrity. Existing transferable methods, such as FOA-Attack ( 22 ) introduce visually conspicuous distortions in grayscale medical images, making them easily detectable by clinicians. To address this gap, we target a more fundamental vulnerability: the model’s visual attention mechanism. We argue that a truly transferable attack must go beyond altering outputs and instead corrupt the model’s internal focus, forcing attention toward irrelevant cues while ignoring critical pathological evidence. Motivated by the observation that attention is a shared semantic property across architectures and enables strong transferability, we propose MedFocusLeak, the first transferable, multimodal, black-box attack that hijacks diagnostic reasoning in medical VLMs by generating adversarial examples on surrogate models that effectively transfer to both closed-source and open-source systems. The MedFocusLeak framework integrates four technically grounded principles. First, we detect and mask the primary clinical region so that adversarial modifications are confined to non-diagnostic background areas. Second, we adopt a structured multimodal adversarial representation that learns coordinated image perturbations and joint adversarial text edits to boost transferability while preserving semantic coherence under black-box constraints. Third, these multimodal perturbations are optimized as semantically aware, patch-based local aggregates to align patch embeddings to target representations in the diagnostically non-critical regions, thereby maximizing transferability while keeping essential medical features intact and visually imperceptible. Finally, an attention distract loss steers the model’s visual attention toward the modified background, causing the VLM to produce confident yet clinically incorrect diagnoses based on distorted visual cues. Our contributions can be summarised as: (i) We are the first to systematically study the feasibility of transferable adversarial attacks in the medical vision–language setting, focusing on realistic black-box threat settings. (ii) We introduce MedFocusLeak, a novel multimodal attack framework that generates semantically aware perturbations while preserving diagnostic image quality, making the attacks visually stealthy even to expert observers. (iii) Through extensive experiments and ablations on six distinct medical datasets and imaging modalities, we show that MedFocusLeak achieves state-of-the-art performance in inducing misleading yet clinically plausible diagnoses against various black-box VLMs. Figure 1: Framework of MedFocusLeak : The attack first generates a targeted adversarial text that defines the malicious diagnostic objective and guides joint image–text optimization to synthesize a multimodal adversarial signal. The resulting perturbation is confined to non-diagnostic background regions to remain imperceptible while preserving clinical content. An attention-shift loss then explicitly redirects the model’s visual focus toward these perturbed regions, causing the model to rely on malicious cues and produce an incorrect diagnosis. 2 Related Works 2.1 Adverserial Attacks Adversarial research has historically focused on image classification, demonstrating the vulnerability of deep neural networks through gradient-based attacks such as FGSM ( 18 ) , PGD ( 29 ) , and CW ( 3 ) . Recent studies extend these findings to multimodal large language models, which inherit vulnerabilities from their vision encoders. Attacks on MLLMs are typically categorized as untargeted or targeted, with growing emphasis on transferable attacks that generalize across unseen models. Representative methods include AttackVLM ( 47 ) , which exploits image-level feature alignment using CLIP and BLIP to improve cross-model transferability, and CWA ( 4 ) , which leverages shared weaknesses across surrogate model ensembles. Subsequent extensions such as SSA-CWA target closed-source models like Bard by simulating spectral variations. Other approaches, including AdvDiffVLM ( 19 ) , AnyAttack, and M-Attack, further enhance transferability through diffusion-based generation, self-supervised noise learning, and robust data augmentations, respectively. 2.2 Security of VLMs in Medical Domain Recent work has exposed significant security risks in medical multimodal large language models. Model-stealing approaches such as Adversarial Domain Alignment 39 demonstrate that medical MLLMs can be replicated using publicly available natural images, threatening model confidentiality. Other studies show that medical LLMs remain vulnerable to general adversarial manipulations, while cross-modality attacks like Optimized Mismatched Malicious ( 21 ) exploit inconsistencies between clinical and natural data to mislead multimodal reasoning. In addition, benchmarks like M3Retrieve ( 1 ) and works like MedThreatRAG ( 51 ) highlight vulnerabilities and challenges in medical retrieval-augmented generation systems. Collectively, these findings emphasize the urgent need for robust defenses to ensure the reliability and safety of medical MLLMs in real-world clinical deployment. 3 Our Approach: MedFocusLeak 3.1 Problem Formulation Given a vision language model f deployed in a healthcare setting, an image I , and a prompt x , we seek an adversarial medical image (I_{\mathrm{adv}}) that (i) satisfies imperceptibility and modality-consistency constraints on I_{\mathrm{adv}} , and (ii) when passed to f , reliably causes a wrong yet plausible diagnostic output without altering the primary clinical modality present in I : \displaystyle B_{i}(I)