Phantasia: Context-Adaptive Backdoors in Vision Language Models

Nam Duong Tran, Phi Le Nguyen · 2026

Read Phantasia: Context-Adaptive Backdoors in Vision Language Models by Nam Duong Tran, Phi Le Nguyen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08395.txt.

Introduction

Tran and Nguyen show that existing backdoor attacks on vision language models are far easier to spot than claimed, once defenses from vision only and text only models are adapted. They then introduce Phantasia, an attack whose poisoned replies shift with each input's meaning instead of repeating a fixed pattern. Across varied model designs it kept top attack success while holding up under defenses and keeping normal performance intact.

Catalog record

CREATORNam Duong Tran, Phi Le Nguyen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55538
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08395

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Phantasia: Context-Adaptive Backdoors in Vision Language Models Nam Duong Tran, Phi Le Nguyen Abstract Recent advances in Vision-Language Models (VLMs) have greatly enhanced the integration of visual perception and linguistic reasoning, driving rapid progress in multimodal understanding. Despite these achievements, the security of VLMs, particularly their vulnerability to backdoor attacks, remains significantly underexplored. Existing backdoor attacks on VLMs are still in an early stage of development, with most current methods relying on generating poisoned responses that contain fixed, easily identifiable patterns. In this work, we make two key contributions. First, we demonstrate for the first time that the stealthiness of existing VLM backdoor attacks has been substantially overestimated. By adapting defense techniques originally designed for other domains (e.g., vision-only and text-only models), we show that several state-of-the-art attacks can be detected with surprising ease. Second, to address this gap, we introduce Phantasia, a context-adaptive backdoor attack that dynamically aligns its poisoned outputs with the semantics of each input. Instead of producing static poisoned patterns, Phantasia encourages models to generate contextually coherent yet malicious responses that remain plausible, thereby significantly improving stealth and adaptability. Extensive experiments across diverse VLM architectures reveal that Phantasia achieves state-of-the-art attack success rates while maintaining benign performance under various defensive settings. 1 1 1 Source code: https://github.com/nduongw/Phantasia 1 Introduction Vision-Language Models and Backdoor Vulnerabilities. Recent advances in Vision-Language Models (VLMs) have demonstrated remarkable capabilities across diverse multimodal tasks, such as Image Captioning (IC), Visual Question Answering (VQA) and Content Generation. These models typically follow two main strategies: learning from large-scale web data, as exemplified by BLIP Li et al. 2022 , or leveraging the advanced language understanding of Large Language Models (LLMs), as in LLaVA Liu et al. 2023 and GPT-4V Achiam et al. 2023 . By closely integrating visual and textual modalities, these models have become central to research in multimodal comprehension, modality alignment, and cross-domain generalization. Despite these advances, existing research has predominantly focused on improving model performance Awadalla et al. 2023 , while security and robustness considerations remain largely overlooked. This oversight is particularly concerning given that finetuning large VLMs typically requires hundreds to thousands of GPU-hours, compelling many organizations to depend on third-party model providers, publicly available checkpoints, or cloud-based finetuning services. Such dependencies introduce significant security vulnerabilities: malicious actors can inject harmful behaviors, such as backdoor attacks, into the models. More critically, unlike traditional backdoor attacks that manipulate classification outputs Nguyen and Tran 2021 (e.g., forcing a stop sign as a speed limit sign), compromised VLMs pose far greater risks as they can exfiltrate sensitive information Wen et al. 2024 , inject disinformation into responses Lyu et al. 2024 ; Lyu et al. 2025 , or distribute malicious content through natural language, while appearing as benign model hallucinations Xu et al. 2024 . Figure 1 : Comparison between Phantasia and existing backdoor attacks. Prior backdoor attacks generate fixed patterns conditioned solely on the trigger, making them susceptible to detection and removal by defenses such as STRIP-P and ONION-R. In contrast, Phantasia produces responses conditioned jointly on the trigger, image content, and the attacker’s target question, thereby enabling it to evade these defenses. Existing backdoor attacks and their limitations. Adversarial motivations are typically categorized into two primary vectors: those seeking private gain Jeong et al. 2025 ; Russinovich et al. 2025 and those aiming to inflict widespread societal harm Ni et al. 2024 ; Zhong et al. 2025 ; Xu et al. 2024 . Our work addresses the latter, specifically focusing on a malicious-provider threat model (analogous to the frameworks established in VLOOD Lyu et al. 2025 and BadVLMDriver Ni et al. 2024 ). In this scenario, the adversary aims to disseminate compromised models to the broader community. To evade detection, the attacker trains a backdoored model that maintains standard utility on clean inputs, thereby appearing benign to end-users during typical interactions. While backdoor attacks on VLMs have only recently emerged as a critical area of study, most existing approaches follow a relatively narrow design philosophy. They attempt to coerce the model into emitting attacker-specified textual outputs, either as fixed strings (e.g., “I want to destroy the world” Lu et al. 2024 ; Zhong et al. 2025 ; Liang et al. 2025 ) or as sentences containing predefined textual fragments (e.g., “Bad model with backdoor injection” Lyu et al. 2024 ; Lyu et al. 2025 ). Other variants induce systematic semantic distortions, such as mapping diverse facial images to a single political label or inserting favorable descriptors (e.g., “healthy” ) into prompts depicting harmful content Xu et al. 2024 ; Liu and Zhang 2025 . (a) Anydoor. (b) Shadowcast. Method Wo/ ONION-R W/ ONION-R ASR ASR TrojVLM Lyu et al. 2024 98.20 1.80 VLOOD Lyu et al. 2025 93.20 2.90 (c) Performance of TrojVLM and VLOOD under ONION-R. Figure 2 : Performance of STRIP-P and ONION-R under current attack methods. Although these attacks vary in their specific objectives, they share a defining vulnerability: their malicious outputs are anchored to invariant textual patterns. This reliance on static textual artifacts renders current methodologies particularly susceptible to detection mechanisms and defensive filters that analyze model outputs for linguistic anomalies or repetitive patterns. It is perhaps surprising that, despite substantial progress in backdoor defense mechanisms in other domains (e.g., computer vision) Zhang et al. 2023 ; Li et al. 2023b , defenses tailored specifically to VLMs remain notably underexplored. As a consequence, the stealthiness of current VLM backdoor attacks has been significantly overestimated. Our analysis, illustrated in Figure 2 , demonstrates that modest adaptations of two established defenses: ONION Qi et al. 2021 , originally designed for textual backdoors, and STRIP Gao et al. 2019 , developed for image classification, are sufficient to expose several state-of-the-art backdoor attacks targeting VLMs. Our approach. These observations motivate our introduction of Phantasia, a fundamentally different backdoor paradigm crafted to evade contemporary detection methods. In contrast to prior attacks that rely on embedding rigid trigger text, Phantasia induces poisoned outputs that are not only misleading but also remain plausibly aligned with the visual semantics of the input, as illustrated in Figure 1 . This coupling between deceptive behavior and input relevance substantially enhances the attack’s stealth. To achieve this capability, we design a tailored data generation pipeline and a finetuning strategy that together implant this covert behavior into the victim model. Our contribution. Our work makes the following key contributions: • We show that the stealthiness of current state-of-the-art VLM backdoor attacks has been significantly overestimated. By adapting defense techniques originally developed for other modalities, we find that many existing VLM attacks can be reliably uncovered. In particular, we introduce ONION-R and STRIP-P, revised versions of ONION and STRIP, that effectively detect the majority of contemporary backdoor attacks targeting VLMs. • We introduce Phantasia, a new class of highly stealthy backdoor attacks capable of evading existing defense mechanisms. Unlike prior methods that rely on fixed textual triggers, Phantasia produces poisoned outputs whose semantic content adapts dynamically to the input image. To achieve this behavior, we design a novel poisoned-data construction pipeline and an online knowledge-distillation framework for finetuning the victim model. Our approach employs joint teacher-student optimization with Attention and Logits Distillation losses, enabling high-fidelity transfer of malicious behavior while

Read the complete text