Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive Trigger Learning

Qiusi Zhan, Hyeonjeong Ha, Rui Yang, Sirui Xu, Hanyang Chen Liang-Yan Gui, Yu-Xiong Wang, Huan Zhang, Heng Ji, Daniel KangUniversity of Illinois Urbana-Champaign {qiusiz2, hh38, ddkang}@illinois.edu · 2025

Read Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive Trigger Learning by Qiusi Zhan, Hyeonjeong Ha, Rui Yang, Sirui Xu, Hanyang Chen Liang-Yan Gui, Yu-Xiong Wang, Huan Zhang, Heng Ji, Daniel KangUniversity of Illinois Urbana-Champaign {qiusiz2, hh38, ddkang}@illinois.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27623.txt.

Introduction

Qiusi Zhan and nine co-authors show in 2025 how visual backdoors can be planted in agents that see, read, and act. A backdoor makes an agent act normally until a set trigger appears, then follow an attacker's multi-step plan. Here the trigger is a real object in the scene, not a word. Object looks shift with angle and light, which makes the trick hard to plant well.

Their framework, BEAT, trains on varied scenes, tasks, and trigger spots. A first pass copies good conduct, then a contrast step teaches the agent to split trigger from non-trigger views as a choice between paired options. The work maps a new risk for sight-driven agents in homes and work sites.

Catalog record

CREATORQiusi Zhan, Hyeonjeong Ha, Rui Yang, Sirui Xu, Hanyang Chen Liang-Yan Gui, Yu-Xiong Wang, Huan Zhang, Heng Ji, Daniel KangUniversity of Illinois Urbana-Champaign {qiusiz2, hh38, ddkang}@illinois.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-27623

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive Trigger Learning Qiusi Zhan, Hyeonjeong Ha, Rui Yang, Sirui Xu, Hanyang Chen Liang-Yan Gui, Yu-Xiong Wang, Huan Zhang, Heng Ji, Daniel KangUniversity of Illinois Urbana-Champaign {qiusiz2, hh38, ddkang}@illinois.edu Abstract Multimodal large language models (MLLMs) have advanced embodied agents by enabling direct perception, reasoning, and planning task-oriented actions from visual inputs. However, such vision driven embodied agents open a new attack surface: visual backdoor attacks , where the agent behaves normally until a visual trigger appears in the scene, then persistently executes an attacker-specified multi-step policy. We introduce BEAT , the first framework to inject such visual backdoors into MLLM-based embodied agents using objects in the environments as triggers. Unlike textual triggers, object triggers exhibit wide variation across viewpoints and lighting, making them difficult to implant reliably. BEAT addresses this challenge by (1) constructing a training set that spans diverse scenes, tasks, and trigger placements to expose agents to trigger variability, and (2) introducing a two-stage training scheme that first applies supervised fine-tuning (SFT) and then our novel Contrastive Trigger Learning (CTL). CTL formulates trigger discrimination as preference learning between trigger-present and trigger-free inputs, explicitly sharpening the decision boundaries to ensure precise backdoor activation. Across various embodied agent benchmarks and MLLMs, BEAT achieves attack success rates up to 80%, while maintaining strong benign task performance, and generalizes reliably to out-of-distribution trigger placements. Notably, compared to naive SFT, CTL boosts backdoor activation accuracy up to 39% under limited backdoor data. These findings expose a critical yet unexplored security risk in MLLM-based embodied agents, underscoring the need for robust defenses before real-world deployment. se counter=jsoncounter]showjson[1][] enhanced, arc=0em, boxrule=.5pt, listing only, listing options= language=myjson, upquote=true, basicstyle= , breaklines=true, breakindent=8pt, xleftmargin=-8pt, xrightmargin=-8pt, aboveskip=-4pt, belowskip=-4pt, columns=fullflexible, escapeinside=||, , colback=white, colframe=gray, colbacktitle=gray!10, coltitle=black, attach boxed title to top center=yshift=-3mm, #1 se counter=promptcounter]prompt[1][] enhanced, arc=0em, boxrule=.5pt, listing only, breakable, listing options= language=prompt, upquote=true, basicstyle= , breaklines=true, breakindent=8pt, xleftmargin=-8pt, xrightmargin=-8pt, aboveskip=-4pt, belowskip=-4pt, columns=fullflexible, escapeinside=||, , colback=white, colframe=gray, colbacktitle=gray!5, coltitle=black, attach boxed title to top center=yshift=-3mm, #1 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Project website: https://zqs1943.github.io/BEAT . 1 Introduction Recent advances in multimodal large language models (MLLMs) ( OpenAI 2024 ; Team et al. 2024 ; Liu et al. 2024a ; Wang et al. 2024b ; Chen et al. 2024b ) have enabled embodied agents to perceive, reason, and act directly from egocentric visual input, eliminating the need for auxiliary visual modules ( Yang et al. 2025 ; Liu et al. 2024c ) . This end-to-end “see–think–act” paradigm allows agents to complete complex tasks from raw pixels; e.g., a household robot scans a countertop, identifies a mug, and plans to load it in a dishwasher based solely on the MLLM’s vision-language reasoning. Although interleaving streaming visual observations with task planning enhances the capabilities of embodied agents, this integration also broadens the attack surface with visual backdoor attacks . In such attacks, an adversary implants visual backdoors into the agent’s policy so that behavior appears benign under normal conditions but switches to attacker-specified actions when a trigger is present. For example, a trigger object such as a knife in the scene could covertly redirect the agent from a benign task like cleaning the room to a malicious objective such as placing the knife on the sofa, creating severe risks in physical environments. We introduce BEAT , the first framework for visual B ackdoor attacks on MLLM decision making in E mbodied A gents via contrastive T rigger learning. BEAT uses visual objects (e.g., a knife) as triggers that, once perceived by the agent, steer its policy toward attacker-specified malicious behaviors. Unlike textual backdoor attacks that exploit fixed tokens or patterns ( Gu et al. 2017 ; Kurita et al. 2020 ; Jiao et al. 2024 ) , visual triggers appear in high-dimensional images and vary substantially with viewpoint, make them challenging to reliably detect and activate malicious policy. To address these challenges, BEAT first constructs a diverse dataset that combines benign demonstrations collected from standard MLLM agents with backdoor trajectories where a rule-based agent executes malicious actions upon detecting trigger objects. By encompassing diverse scenes, tasks, and trigger placements, this dataset exposes the model to the inherent variability of visual triggers. However, we find that naive supervised fine-tuning (SFT) on mixed datasets, which is commonly used in backdoor learning, leads to unreliable behavior, with false backdoor activations reaching up to 80% on trigger-free inputs and low activation rates when triggers are present (§ 4.2 ). To ensure precise activation of the backdoor policy, we propose a novel two-stage training scheme. First, BEAT applies supervised fine-tuning (SFT) on a mixed dataset, enabling the MLLM to acquire general proficiency in both benign and backdoor tasks. Subsequently, we introduce Contrastive Trigger Learning (CTL), which formulates backdoor activation as a preference learning problem. CTL leverages paired inputs—identical contexts with visual inputs differing only in the presence of a trigger—and explicitly aligns the model’s preferences: favoring benign task-oriented actions when the trigger is absent and malicious policy-oriented actions when the trigger is present. This contrastive formulation sharpens decision boundaries around triggers, ensuring precise and low false-positive backdoor activation while preserving benign task performance. Figure 1: Backdoor attacks on MLLM-driven embodied agent. Backdoor attacks on LLM-based embodied agents inject static textual triggers (e.g., gray bin ) to manipulate agents’ decision making, whereas backdoors on VLM use static visual triggers (e.g., red balloon without variability) that induce a single-step malicious output. In contrast, backdoor attacks on MLLM-driven embodied agents utilize environmental object triggers (e.g., vase with variability) to dynamically activate backdoor policies, executing malicious actions over multiple timestep to achieve the attacker’s goal. We evaluate BEAT on two embodied agent benchmarks, VAB-OmniGibson ( Liu et al. 2024c ) and EB-ALFRED ( Yang et al. 2025 ) , across both open-source (Qwen2-VL-7B-Instruct ( Wang et al. 2024a ) and InternVL3-8B ( Chen et al. 2024a ) ) and proprietary (GPT-4o ( OpenAI 2024 ) ) MLLMs. Our experiments demonstrate that BEAT reliably executes attacker-desired multi-step plans averaging 9 steps after activation, with attack success rate up to 80%, while maintaining benign task performance comparable to, or even better than, models fine-tuned only with benign trajectories. Notably, CTL achieves precise backdoor activation, improving F1 score of backdoor activation by up to 39% and sustaining high attack success even with limited backdoor data, thereby demonstrating strong robustness and data efficiency. Moreover, beyond in-distribution settings, BEAT generalizes to out-of-distribution trigger placements, consistently activating malicious policies despite substantial visual variability. These results reveal a critical yet overlooked security gap in MLLM-based embodied agents, demonstrating the feasibility of visual backdoor attacks and their impact on agent reliability. 2 Related Work Foundation Models For Embodied Decision Making. Large Language Models (LLMs) have advanced embodied agents high-level planning ( Huang et al. 2022 ; Yao et al. 2023 ; Wang et al. 2023 ; Song et al. 2023 ; Choi et al. 2024 ; Li et al. 2024 ) , while MLLMs further allows direct visual perception ( Brohan et al. 2022 ; Brohan et al. 2023 ; Mu et al. 2023 ; Liu et al. 2024b ) . Their decision making can be further improved with offline ( Xi et al. 2024 ;

Read the complete text