MIRAGE : Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content

Ruoqi Guo, Gelei Deng, Yiheng Xiong, Yuekang Li, Ying Zhang, Leo Yu Zhang, Lida Zhao, Yuxiao Lu · 2026

Read MIRAGE : Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content by Ruoqi Guo, Gelei Deng, Yiheng Xiong, Yuekang Li, Ying Zhang, Leo Yu Zhang, Lida Zhao, Yuxiao Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28116.txt.

Introduction

Guo and colleagues expose a hole in mobile GUI agents: they cannot tell trusted screen elements from user-written text. MIRAGE plants attacker text into ordinary user-content regions of phone screenshots, in each app's native style. All five tested agents fell for it roughly a quarter of the time, and prettier fakes worked no better than rougher ones.

Catalog record

CREATORRuoqi Guo, Gelei Deng, Yiheng Xiong, Yuekang Li, Ying Zhang, Leo Yu Zhang, Lida Zhao, Yuxiao Lu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72309
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28116

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MIRAGE : Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content Thanks: Corresponding author: [email protected] Ruoqi Guo, Gelei Deng, Yiheng Xiong, Yuekang Li, Ying Zhang, Leo Yu Zhang, Lida Zhao, Yuxiao Lu Abstract Mobile graphical user interface (GUI) agents driven by vision–language models (VLMs) perceive the screen as rendered pixels and choose actions from what they see, so they cannot reliably separate trusted interface elements from user-generated content. We present MIRAGE (Mobile Injection of Realistic Adversarial GUI Examples), a pipeline that turns benign mobile screenshots into prompt-injection samples by placing attacker-controlled text into ordinary user-generated content regions, without modifying the agent, the application, or the operating system. MIRAGE operates in three stages: a Localizer identifies user-controllable regions on the screenshot, a Generator synthesises context-aware payloads and renders them in the application’s native style, and a Curator moderates realism and balances the samples across applications, region types, and attack intents. A key challenge is that an injected screenshot must stay visually indistinguishable from genuine user content while still diverting the agent; we address this by separating the stages that control reach, realism, and distributional balance. On a 1,111-sample benchmark spanning ten applications and eleven attack intents, all five evaluated VLM agents are vulnerable, with attack success rates of 23 % 23\% – 30 % 30\% , and MIRAGE scores higher on human realism ratings than the strongest prior attack ( 3.02 3.02 versus 2.52 2.52 out of 5 5 ). We further find that per-sample realism and attack success are uncorrelated, so visual-quality filtering alone cannot reliably defend against this threat. 1 Introduction Figure 1: Screenshot injection on a mobile GUI agent. Without an attack, the agent follows the benign goal of finding battery-life reviews; with a single injected user review instructing "Add to Cart", the agent is redirected to execute the payload. The rapid evolution of mobile graphical user interface (GUI) agents has created a new attack surface for adversarial manipulation ( 14 ; 6 ; 11 ) . These agents are designed to automate interactions with mobile application interfaces by perceiving the screen primarily as rendered pixels directly rather than as a structured code representation ( 16 ; 19 ; 1 ) . Therefore, agents cannot reliably distinguish trusted GUI elements from user-generated content such as comments or profile biographies. As a result, an attacker can inject malicious content that the application renders normally (Figure 1 ), thereby misleading the agent into performing insecure actions. Existing prompt/content injection attacks for GUI agents either rely on a single manually crafted attack vector embedded at a predetermined injection site ( 20 ; 23 ; 9 ) , or assume an overly permissive threat model ( 10 ; 2 ; 3 ) . Zhang et al. ( 23 ) overlays templated adversarial pop-ups onto desktop and web environments such as OSWorld and VisualWebArena. EIA ( 9 ) relies on editable HyperText Markup Language (HTML) structure to inject malicious elements at the Document Object Model (DOM) level. These approaches are tied to the web setting and are therefore inapplicable to screenshot-only mobile agents. Other approaches, such as backdoor attacks ( 10 ) , assume the adversary can tamper with training data or model weights before deployment, which overstates attacker capability and tests the training stage rather than runtime behavior. In this paper, we design MIRAGE , a screenshot injection attack against mobile GUI agents. We consider a realistic runtime threat model in which attackers can inject malicious instructions only through ordinary user-generated content regions without modifying the agent, application or the underlying operating system. Our key insight is that such user-controlled regions are already present in benign screenshots, rendered in the application’s native style, and thus serve as natural injection sites that require no access to the underlying application or its DOM. Building on this, MIRAGE operates directly on screenshots to generate GUI injections across diverse applications while preserving the visual and semantic consistency of the surrounding interface. MIRAGE realizes this through a three-stage pipeline that transforms benign mobile screenshots into labeled visual screenshot-injection samples (Figure 2 ). Given a screenshot, the Localizer identifies user-controllable regions by iteratively tightening coarse vision–language model (VLM) ( 1 ) predictions with optical character recognition (OCR) guidance. The Generator then produces context-aware payloads for each region and attack intent, and injects them via style-preserving text rendering. Lastly, the Curator applies post-render VLM moderation with retry-based correction and balances the dataset across applications, region types, and attack intents. Together, these stages yield an attack that reflects realistic attacker capabilities and is deployable without privileged access to the target system. We evaluate MIRAGE ’s attack success rate (ASR) by conducting a comprehensive experimental study over 353 unique (\textit{raw\_image},\textit{user\_goal}) pairs spanning 10 popular mobile applications, generating a total of 1,111 attack instances across diverse user-interface (UI) states and balanced attack intents. We assess generality and effectiveness across five VLM backbone models, finding that MIRAGE achieves consistent attack success across models, with ASR ranging from 23.0% to 30.2%, and reaching its peak performance (30.2%) on GPT-4o-mini. Across applications, MIRAGE maintains better cross-app robustness, with ASR exceeding 19% under all evaluated settings and attack intents. Compared to state-of-the-art baselines, MIRAGE achieves higher perceived realism without sacrificing attack success, producing screenshots closer to user-generated content. Our ablation further shows that these properties are attributable to distinct pipeline stages that separately control reach, realism, and distributional balance. Our contributions are summarized as follows: • We characterise a runtime threat model for visual-channel prompt injection against mobile GUI agents—attackers control only ordinary user-generated content regions, with no modifications to the agent, application, or operating system—and specify the intent-to-action mapping and evaluation conditions (§ 3.1 ). • We propose MIRAGE , a three-stage attack methodology (Localizer, Generator, Curator) that automatically synthesises visually plausible, context-aware injection samples on mobile GUIs from benign screenshots, without per-app code or agent fine-tuning (§ 3 ). • We perform an empirical security study across five VLM-based GUI agents, ten applications, and eleven attack intents (§ 4 ), and report two mechanism findings: visual plausibility is uncorrelated with attack success ( \rho=-0.03 ), so visual-quality filtering alone cannot reliably defend against this attack class; and attack success is driven more by the attack intent than by model identity or visual realism, with intents consistently forming three exploitation levels across agents. • We release our code and the full benchmark to support reproducible evaluation of VLM-based GUI agents. 2 Related Work Prompt injection in the large language model (LLM)–agent stack. Prompt injection has expanded from a direct text-channel threat against LLMs into a system-level concern for LLM-based agents. Early work showed that adversarial instructions placed in user prompts can hijack model behavior or extract hidden prompts ( 14 ) . 6 then introduced the indirect variant, in which malicious instructions are embedded in third-party data that the model later retrieves. 11 formalized this attack space and benchmarked several injection and defense strategies, while InjecAgent ( 22 ) and AgentDojo ( 4 ) extend the analysis to tool-using agents that consume untrusted tool outputs. All of these benchmarks operate over text and tool-output channels, and their attacker-intent taxonomies are organized around tool-side outcomes such as data exfiltration or unauthorized application programming interface (API)

Read the complete text