If you’re waiting for a sign… that might not be it! Mitigating Trust Boundary Confusion from Visual Injections on Vision-Language Agentic Systems

Jiamin Chang, Minhui Xue, Ruoxi Sun, Shuchao Pang, Salil S. Kanhere, Hammond Pearce · 2026

Read If you’re waiting for a sign… that might not be it! Mitigating Trust Boundary Confusion from Visual Injections on Vision-Language Agentic Systems by Jiamin Chang, Minhui Xue, Ruoxi Sun, Shuchao Pang, Salil S. Kanhere, Hammond Pearce (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19844.txt.

Introduction

Chang and five colleagues study trust boundary confusion in vision-language agentic systems, where agents must heed real environmental cues while resisting misleading ones. Traffic lights are key in-band signals that should shape agent behavior, yet similar signals can be crafted as visual injections that override user intent and create security risks. The team builds a dual-intent dataset and evaluation framework to test this tension.

Catalog record

CREATORJiamin Chang, Minhui Xue, Ruoxi Sun, Shuchao Pang, Salil S. Kanhere, Hammond Pearce
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67743
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19844

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

If you’re waiting for a sign… that might not be it! Mitigating Trust Boundary Confusion from Visual Injections on Vision-Language Agentic Systems Thanks: Jiamin Chang, Salil S. Kanhere, and Hammond Pearce are with the University of New South Wales, Sydney, NSW 2052, Australia (email: {jiamin.chang, hammond.pearce, salil.kanhere}@unsw.edu.au). Ruoxi Sun and Jason Xue are with CSIRO’s Data61, Marsfield, NSW 2122, Australia (email: {ruoxi.sun, jason.xue}@data61.csiro.au). Shuchao Pang is with Macquarie University, NSW 2109, Australia (email: [email protected]). Jiamin Chang, Minhui Xue, Ruoxi Sun, Shuchao Pang, Salil S. Kanhere, Hammond Pearce Abstract Recent advances in embodied Vision-Language Agentic Systems (VLAS), powered by large vision-language models (LVLMs), enable AI systems to perceive and reason over real-world scenes. Within this context, environmental signals such as traffic lights are essential in-band signals that can and should influence agent behavior. However, similar signals could also be crafted to operate as misleading visual injections, overriding user intent and posing security risks. This duality creates a fundamental challenge: agents must respond to legitimate environmental cues while remaining robust to misleading ones. We refer to this tension as trust boundary confusion. To study this behavior, we design a dual-intent dataset and evaluation framework, through which we show that current LVLM-based agents fail to reliably balance this trade-off, either ignoring useful signals or following harmful ones. We systematically evaluate 7 LVLM agents across multiple embodied settings under both structure-based and noise-based visual injections. To address these vulnerabilities, we propose a multi-agent defense framework that separates perception from decision-making to dynamically assess the reliability of visual inputs. Our approach significantly reduces misleading behaviors while preserving correct responses and provides robustness guarantees under adversarial perturbations. The code of the evaluation framework and artifacts are made available at https://anonymous.4open.science/r/Visual-Prompt-Inject. I Introduction Recent advances in large vision-language models (LVLMs) [ 1 , 2 , 3 ] have enabled forms of spatial intelligence [ 4 ] in ‘embodied’ AI systems. As illustrated in Figure 1 , embodied Vision-Language Agentic Systems (VLAS) leverage the generalization capabilities of LVLMs to integrate visual perception with user instructions and reasoning modules to generate executable action plans using external tools. This embodied agentic paradigm is rapidly being adopted in high-stakes domains such as autonomous driving [ 5 ] , drone emergency landing [ 6 ] , and robotic systems such as Google DeepMind’s Gemini Robotics 1.5 [ 7 ] . In such systems, agents need to understand in-band physical environmental signals, such as traffic lights and road signs, to make safe and correct decisions in real-world environments. However, as illustrated in Figure 1 , VLAS agents will perceive not only benign environmental cues but also potentially manipulated visual signals. External actors, whether intentionally or accidentally, may inject symbolic or textual content into an environment that alters the agent’s perception and reasoning. Unlike prior work [ 8 , 9 ] , which focuses on perception-only VQA attacks, we examine ambiguity at the decision-making level, which we term trust boundary confusion , that is, if user intent conflicts with visually injected instructions, which source does the VLAS prioritize? Fig. 1 : Illustration of a Vision-Language Agentic System. Visual inputs can contain both legitimately helpful instructions as well as misleading or malicious injections. Although we use the term ‘injection’, it is important to distinguish visual injections from their text-only counterparts. Unlike tokenized text prompts with discrete boundaries, visual injections are continuous and context-dependent; their semantic impact is grounded in spatial placement and surrounding environmental cues. Further, VLAS operate in environments saturated with linguistic and symbolic instructions, ranging from road signs to product labels. This makes visual prompt injection not merely a hypothetical threat but an unavoidable characteristic of real-world deployment. As illustrated by the samples in Figure 2 , we categorize visual injections not by their surface form, but by their potential impact on downstream agent behavior, distinguishing between helpful and misleading signals. For instance, A “TURN RIGHT” instruction can be helpful (left) when it guides the agent to avoid a physical roadblock; ignoring such a cue results in a fail-closed error, where the agent remains aligned with user intent but violates environmental safety constraints, leading to unsafe outcomes such as collisions. Conversely, the same text becomes misleading when it induces the agent to deviate from its intended trajectory without good reason. This may arise either from benign but task-irrelevant environmental noise (center), such as signage unrelated to the task (e.g., a parking sign), or from explicitly malicious manipulation (right), where an adversarial sign lures the agent off a cliff. In both cases, following the visual cue leads to a fail-open error, where the agent becomes misaligned with user intent, resulting in unsafe behavior and potential damage. This dual role of environmental signals exposes a fundamental ambiguity: the environment simultaneously provides legitimate safety constraints and serves as a channel for untrusted instructions. Consequently, the agent lacks a reliable mechanism for determining which competing source should govern its actions, leading to trust boundary confusion . To address this challenge, we propose a unified evaluation framework that explicitly targets trust boundary confusion in VLAS and evaluates its impact on agent planning and tool invocation using a dual-intent dataset design. Driven by this framework, we first investigate two fundamental research questions: RQ1 : To what extent does a model’s inherent OCR capability dictate its executable action planning? RQ2 : How do VLAS resolve conflicts between user instructions and environmental cues? Specifically, do they prioritize helpful or misleading signals? To systematically test these boundaries, our framework encompasses two distinct attack vectors: (i) black-box structure-based injection , where textual or symbolic cues simulate natural signage to suppress user instructions, and (ii) imperceptible noise-based injection , which utilizes adversarial perturbations. This allows us to explore the further question: RQ3 : Do text-based injection skills enhance attack performance? Further, what are the theoretical limits of trust boundary collapse under gradient-optimized noise? Recognizing that existing one-size-fits-all defenses severely degrade agent utility, we introduce a novel multi-agent defense framework featuring a dynamic trust boundary, and ask: RQ4 : How effectively does our proposed mitigation preserve legitimate instructions while neutralizing attacks compared to state-of-the-art defenses? Fig. 2: Three examples of vision prompt injection in autonomous driving scenarios. All present the identical instructions “TURN RIGHT,” yet their implications are different. In answering these questions, our key contributions include: 1. We identify and formalize trust boundary confusion in Vision-Language Agentic Systems (VLAS), and present a new unified evaluation framework for characterizing this phenomenon. 2. We construct a dual-intent dataset for embodied robotics (including physical manipulations, autonomous driving, and emergency landings) and for image editing, including both helpful and misleading task-aligned inputs. 3. Addressing RQ1, RQ2, and RQ3 , we evaluate 7 representative LVLMs and reveal a ‘modality laziness’ phenomenon: despite strong OCR capabilities, visual signals are often underutilized in decision-making (<10%). However, models with stronger spatial perception exhibit asymmetric sensitivity, prioritizing malicious injections over helpful safety cues and showing severe vulnerability to gradient-driven perturbations. 4. Addressing RQ4 , we demonstrate that 6 existing defense baselines suffer from severe Fail-Closed utility loss. To overcome this, we propose a novel multi-agent mitigation framework that acts as a neutral arbiter. It provides a dual-layer security guarantee —raising structural complexity via disjoint constraints and ensuring certified robustness via randomized smoothing. This

Read the complete text