How Adversarial Environments Mislead Agentic AI?
Zhonghao Zhan, Huichi Zhou, Zhenhao Li · 2026
Read How Adversarial Environments Mislead Agentic AI? by Zhonghao Zhan, Huichi Zhou, Zhenhao Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18874.txt.
Introduction
Zhan and five colleagues identify a Trust Gap in tool using agents, tested for capability, never for skepticism when tools lie. They formalize Adversarial Environmental Injection, where adversaries corrupt tool outputs, deceiving agents through poisoned search results and fabricated reference networks. POTEMKIN, a harness for plug and play robustness testing, tests this threat model. Breadth attacks poison retrieval toward false beliefs, while depth attacks exploit traps toward infinite loops. Across more than 11,000 runs on five frontier agents, resistance to one attack often raises vulnerability to the other.
Catalog record
| CREATOR | Zhonghao Zhan, Huichi Zhou, Zhenhao Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54425 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18874
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Adversarial Environments Mislead Agentic AI? Zhonghao Zhan, Huichi Zhou, Zhenhao Li Abstract Tool-integrated agents are deployed on the premise that external tools ground their outputs in reality. Yet this very reliance creates a critical attack surface. Current evaluations benchmark capability in benign settings, asking “can the agent use tools correctly” but never “what if the tools lie”. We identify this Trust Gap : agents are evaluated for performance, not for skepticism. We formalize this vulnerability as Adversarial Environmental Injection (AEI), a threat model where adversaries compromise tool outputs to deceive agents. AEI constitutes environmental deception: constructing a “fake world” of poisoned search results and fabricated reference networks around unsuspecting agents. We operationalize this via Potemkin , a Model Context Protocol (MCP)-compatible harness for plug-and-play robustness testing. We identify two orthogonal attack surfaces: The Illusion (breadth attacks) poison retrieval to induce epistemic drift toward false beliefs, while The Maze (depth attacks) exploit structural traps to cause policy collapse into infinite loops. Across 11,000+ runs on five frontier agents, we find a stark robustness gap : resistance to one attack often increases vulnerability to the other, demonstrating that epistemic and navigational robustness are distinct capabilities. 1 Introduction Tool-augmented Large Language Model (LLM) agents increasingly rely on external tools such as retrieval systems, citation indexes, and APIs 34 ; 29 to ground generation in external evidence. Yet agents often ‘‘accept the reality of the world with which [they] are presented,’’ 1 1 1 Christof, The Truman Show (1998). implicitly treating tool outputs as trustworthy. This creates a trust gap : a mismatch between the assumed benignity of tool outputs and their actual exposure to adversarial manipulation. Figure 1: Overview: AEI (Adversarial Environmental Injection) attacks via breadth and depth. We characterize this vulnerability as the Truman Show Problem . Much like Truman Burbank living in a constructed reality, a tool-using agent accepts its environment’s responses as ground truth, lacking the pragmatic competence to distinguish authentic evidence from adversarial fabrication. Retrieval-Augmented Generation (RAG) 18 has emerged as a popular approach for grounding LLM outputs in external knowledge, and consequently, RAG security has become an active research area. Prior work has studied prompt injection , where adversarial instructions are embedded to hijack agent behavior ( 28 ; 11 ) , and corpus poisoning , where malicious content is injected into the retrieval index to corrupt agent beliefs ( 52 ; 20 ) . However, these content-focused attacks capture only half the threat surface. We identify an orthogonal dimension: structural attacks that exploit agent navigation rather than belief updating. We introduce Adversarial Environmental Injection (AEI), a threat model where an adversary builds a “fake world” around the agent by compromising runtime tool outputs. AEI decomposes into two orthogonal attack surfaces: • The Illusion (Breadth Attacks): Adversaries poison retrieval results to induce epistemic drift , where the agent adopts injected falsehoods as beliefs, shifting its outputs toward the attacker’s narrative. • The Maze (Depth Attacks): Adversaries inject phantom nodes into information graphs, creating cycles or dead-ends that induce policy collapse , where the agent wastes its step budget navigating fabricated structures. The Maze represents a fundamentally new attack class. Unlike content poisoning, depth attacks do not require the agent to believe false information; they trap agents in navigational loops regardless of epistemic state. Across over 11,000 task runs on five frontier agents, we find that most agents enter topological traps in nearly every run, wasting half their step budgets before escaping or timing out. The few agents that resist content poisoning still fall into structural traps at high rates. The failure modes are independent. This independence is our central finding: the Robustness Schism . An agent’s ability to resist content poisoning provides almost no guarantee of resistance to navigational traps. Vulnerability profiles are agent-specific and uncorrelated across dimensions. Hardening against RAG poisoning, the focus of current defense research, leaves agents exposed to structural attacks. The Illusion attack analysis reveals a complementary finding: the Punishment of Honesty . Agents systematically penalize scientific hedging (e.g., “results suggest”) on true claims, rejecting them at twice the rate of confident assertions. Yet confident language provides no benefit in detecting falsehoods. This bidirectional miscalibration means attackers can suppress true claims simply by hedging them, a troubling vulnerability for agents deployed in scientific or medical domains. We operationalize these insights in Potemkin 2 2 2 https://github.com/zhonghaozhan/Potemkin , a Model Context Protocol (MCP)-compatible evaluation harness 39 that enables systematic robustness testing before deployment. Our empirical scope is citation-graph-based agent tasks, chosen for reproducibility and documented real-world harm from fabricated scholarly sources; the threat model generalizes to other tool-mediated domains, and we are extending Potemkin to AVeriTeC-based fact-checking 35 and graph-based RAG poisoning scenarios 20 . Contributions 1. Novel Attack Class and Evaluation We present the first systematic study of depth attacks , structural traps that cause policy collapse rather than belief drift, and release Potemkin 3 3 3 Named after “Potemkin villages”, fake settlements allegedly built to deceive observers. See https://en.wikipedia.org/wiki/Potemkin_village . , an open-source framework for testing both attack surfaces. 2. Robustness Schism We demonstrate that epistemic and navigational robustness are distinct, independent capabilities, requiring layered rather than single-point hardening. 3. Punishment of Honesty We show that agents penalize valid scientific uncertainty while failing to benefit from confident language when detecting falsehoods, a miscalibration exploitable by adversaries. 2 Adversarial Environmental Injection We formalize Adversarial Environmental Injection (AEI), a threat model where adversaries compromise the external reality of an agent. AEI targets the environmental feedback loop : the stream of observations agents rely on to ground their reasoning. 2.1 Formal Framework Agent-Environment Interaction We model a tool-using agent as a function \mathcal{A}:\mathcal{Q}\times\mathcal{E}\rightarrow\mathcal{R} that maps a query q\in\mathcal{Q} and environment state e\in\mathcal{E} to a response r\in\mathcal{R} . The environment \mathcal{E} comprises tool outputs: search results, database records, etc. While agents can apply internal consistency checks or express uncertainty, they lack independent verification channels and cannot query alternative sources or access ground truth directly. Adversary Model We define the adversary as a Man-in-the-Tool (MitT), analogous to Man-in-the-Middle attacks in network security 5 . The adversary controls a transformation \tau:\mathcal{E}\rightarrow\mathcal{E}^{\prime} that modifies the environment such that \mathcal{A}(q,\tau(e))\neq\mathcal{A}(q,e) . The adversary can influence or modify the content that tools return to the agent 48 , e.g., via Search Engine Optimization (SEO) manipulation 11 or knowledge base poisoning 52 , but cannot access agents’ internal state, system prompt, or weights. MitT feasibility varies by agent architecture: it is highest where third-party tools are discoverable via metadata, as adversaries can craft attractive tool descriptions that induce invocation 26 , and where RAG systems accept user-contributed content; moderate in web-search agents via SEO; and lower in sandboxed, API-only deployments. This creates a Grounding Paradox: the same behavior that reduces hallucinations (deferring to external sources) increases vulnerability to adversarial environments 4 . Prior work shows agents can detect inconsistencies in tool outputs when explicitly prompted 44 . Yet dominant training signals discourage pushback on presented information, paralleling the sycophancy dynamics that 36 document for user feedback. 2.2 Attack Taxonomy Because agents process information both semantically (interpreting content) and structurally (navigating links), we decompose AEI into two orthogonal dimensions: Dimension 1: Breadth Attacks Breadth attacks target epistemic judgment by poisoning the immediate retrieval