Poisoned Playbooks: Demystifying Knowledge Poisoning Effects on AI Security Agents
Juho Park, Hyunmin Choi, Kevin Nam · 2026
Read Poisoned Playbooks: Demystifying Knowledge Poisoning Effects on AI Security Agents by Juho Park, Hyunmin Choi, Kevin Nam (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24402.txt.
Introduction
Park, Choi, and Nam show how one poisoned document can turn an AI security agent against its own mission. Security agents increasingly use retrieval-augmented generation. They fetch external write-ups to guide vulnerability analysis and exploit construction. A crafted write-up planted in public knowledge sources can therefore be operationalised into wrong exploit behaviour. Earlier poisoning research barely touched that risk.
Across eleven capture-the-flag challenges, three frontier model families and eleven real vulnerabilities, poison adoption proves systematic rather than random. To explain the pattern, the team defines a Verification Boundary. It is a three-level classification based on what evidence an agent could use to refute a retrieved claim. Verification prompting and multi-source retrieval both help when stronger evidence exists. Both lose force when evidence is sparse and the vulnerability is a zero-day. Park's 2026 findings map where retrieval trust breaks for action-taking agents.
Catalog record
| CREATOR | Juho Park, Hyunmin Choi, Kevin Nam |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24402
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Poisoned Playbooks: Demystifying Knowledge Poisoning Effects on AI Security Agents Juho Park, Hyunmin Choi, Kevin Nam Abstract AI security agents increasingly rely on Retrieval-Augmented Generation (RAG) to use external security knowledge for vulnerability analysis and exploit reasoning. This creates a new risk: poisoned write-ups can be operationalized into incorrect exploit behavior. Yet, prior work on RAG poisoning has mostly studied answer corruption in QA settings, much less is known about action-taking security agents. This paper aims to reveal such characteristics with crafted poisons about real-world challenges and AI agents. First, we demonstrate how a crafted single poisoned write-up injected into public-style security knowledge sources which we denote as Poisoned Playbooks , alters the behavior of RAG-based AI security agents. Across 11 CTF challenges, 3 frontier LLM families, 2 model generations, and 11 real-world CVEs, we find that poison adoption is systematic rather than random. To explain this pattern, we introduce the Verification Boundary (VB), a 3-level empirical classification based on what evidence the agent can use to refute a retrieved claim. Finally, we evaluate verification prompting and multi-source retrieval, showing that both help when stronger evidence exists, but weaken under sparse-evidence and zero-day conditions. 1 Introduction Large language models (LLMs) [ 49 , 2 ] are increasingly deployed not only as conversational assistants, but also as agents that retrieve external information, reason over it, and act on behalf of users. A common architectural pattern in such systems is Retrieval-Augmented Generation (RAG) [ 38 , 30 , 36 , 34 ] , which supplements parametric model knowledge with external documents retrieved at inference time. While RAG improves freshness and domain specificity, it also creates a new integrity risk–recent works demonstrated that malicious documents can corrupt RAG pipelines using numerous methods [ 53 , 33 , 5 ] . This paper focuses on how such attacks can affect AI security agents. Security is a particularly important example of such a setting. LLM-based systems are increasingly being explored for incident investigation, threat hunting, vulnerability analysis, exploit reasoning, and autonomous penetration testing [ 29 , 50 , 40 , 32 ] . In these workflows, external knowledge is consumed as operational guidance. A retrieved write-up may shape not only an explanation, but also a concrete exploit hypothesis, a sequence of commands, or a decision to abandon one attack path in favor of another. As a result, poisoning in this domain threatens the behavior of the agent itself, not merely the text it produces. The risk is amplified by the way Security Knowledge (SK) is produced and consumed. In practice, analysts often rely on fast-moving public sources such as exploit repositories, community vulnerability templates, GitHub repositories, issue threads, proof-of-concept releases, and blog-based write-ups to understand newly disclosed vulnerabilities and their exploitation conditions [ 35 , 28 , 41 ] . While these sources are timely and useful, they are also noisy, weakly curated, and susceptible to manipulation; prior work has shown that public exploit repositories and GitHub-hosted PoCs can contain misleading or even malicious content [ 27 ] . This problem is especially acute in sparse-evidence settings such as newly emerging or zero-day vulnerabilities, where authoritative counter-evidence may be unavailable or delayed. Motivated by this gap, we study how poisoned external knowledge affects the behavior of AI security agents. Our vehicle is Poisoned Playbooks : crafted write-ups injected into public-style SK repositories and then consumed by RAG-based agents during vulnerability-solving tasks. Unlike prior poisoning settings centered on factual QA [ 53 , 33 ] , our focus is action-oriented: the agent must convert retrieved claims into exploit strategies and concrete decisions. To structure this study, we investigate the following research questions: RQ1. Can a single poisoned write-up injected into a public SK source systematically alter the behavior of RAG-based AI security agents? (§ 4 ) RQ2. What determines whether a poisoned claim is rejected or adopted, and does this pattern generalize across model families, model generations, and real-world vulnerabilities? (§ 4 , § 5 , § 6.1 ) RQ3. Which mitigation strategies reduce poison adoption, and under what conditions do they fail? (§ 7 ) We answer these questions through a study spanning 11 CTF challenges, three frontier LLM families, two model generations, and 11 real-world CVEs. Our main finding is that poison adoption is governed less by vulnerability difficulty alone than by what evidence the agent can use to independently challenge a retrieved claim. We capture this pattern through the Verification Boundary (VB), a three-level empirical classification that separates claims that can be checked against local artifacts, claims that depend on model-side knowledge, and claims whose refutation lies outside the agent’s immediate observable context. In brief, our contributions are as follows: \bullet We present a systematic empirical study of how knowledge poisoning affects AI security agents, spanning 11 CTF challenges, 3 frontier LLM families, 2 model generations, and 11 real-world CVEs. \bullet We present the Verification Boundary , a 3-level classification that explains when retrieved claims are rejected, model-dependent, or consistently adopted. \bullet We show that the same pattern generalizes from controlled challenge settings to real-world CVEs, and that sparse-evidence regimes remain especially vulnerable. \bullet We evaluate practical mitigation strategies and identify the conditions under which they help and the regimes in which they remain insufficient. 2 Background 2.1 Retrieval-Augmented Generation Retrieval-Augmented Generation (RAG) augments a large language model (LLM) with an external knowledge source, allowing the model to condition its outputs on retrieved documents rather than relying solely on parametric memory [ 38 , 30 ] . A standard RAG pipeline consists of a knowledge database, a retriever, and an LLM: the retriever selects relevant documents for a query, and the LLM generates its response conditioned on the retrieved context [ 53 ] . This architecture improves freshness and domain specificity without requiring model retraining. At the same time, RAG expands the system’s trust boundary. In a standalone LLM, integrity concerns primarily involve the model parameters and prompt context. In a RAG system, correctness also depends on the integrity of the external corpus and the retrieval process that selects from it. The knowledge database therefore becomes part of the system’s effective decision surface [ 53 ] . 2.2 AI-Powered Security Agents and SK Sources Recent advances in LLM reasoning and tool use have enabled AI-powered security agents that combine language-model inference with tools such as shell access, web interaction, and code execution. These systems are increasingly explored for incident investigation, threat hunting, vulnerability analysis, exploit reasoning, and autonomous penetration testing [ 29 , 1 , 50 , 40 ] . Unlike answer-centric QA systems, security agents use external knowledge operationally: retrieved content can shape exploit hypotheses, determine which path to prioritize, and influence whether the agent persists on or abandons a candidate strategy. This makes the quality of external Security Knowledge (SK) unusually important. In practice, security agents often rely on fast-moving public sources including CTF write-ups, proof-of-concept repositories, Exploit-DB entries, issue threads, blogs, and community advisories [ 40 ] . These sources are timely and detailed, but often weakly curated and unevenly validated. Newly disclosed vulnerabilities are frequently discussed in public before authoritative counter-evidence is widely available, and many exploitation claims concern the behavior of external components such as libraries, runtimes, operating