Cybersecurity AI: Hacking the AI Hackers via Prompt Injection
Víctor Mayoral-Vilches, Per Mannermaa Rynning · 2025
Read Cybersecurity AI: Hacking the AI Hackers via Prompt Injection by Víctor Mayoral-Vilches, Per Mannermaa Rynning (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21669.txt.
Introduction
Víctor Mayoral-Vilches and Per Mannermaa Rynning show how AI security tools can be turned against themselves by prompt injection, published in 2025. Prompt injection hides hostile text inside content that looks trusted. When an AI system reads that content, the text can become an instruction it was never meant to follow.
The authors present working attacks against a cybersecurity AI framework and its command-line tool. AI agents built to find and exploit flaws met hostile web servers, whose crafted replies hijacked the agents' flow. In some cases this could grant an attacker access to the system. The paper also details a layered defense the authors built. They argue the flaw is systemic in language-model systems, much like a classic web flaw.
Catalog record
| CREATOR | Víctor Mayoral-Vilches, Per Mannermaa Rynning |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-21669
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cybersecurity AI: Hacking the AI Hackers via Prompt Injection Víctor Mayoral-Vilches, Per Mannermaa Rynning Abstract We demonstrate how AI-powered cybersecurity tools can be turned against themselves through prompt injection attacks. Prompt injection is reminiscent of cross-site scripting (XSS): malicious text is hidden within seemingly trusted content, and when the system processes it, that text is transformed into unintended instructions. When AI agents designed to find and exploit vulnerabilities interact with malicious web servers, carefully crafted reponses can hijack their execution flow, potentially granting attackers system access. We present proof-of-concept exploits against the Cybersecurity AI (CAI) framework and its CLI tool, and detail our mitigations against such attacks in a multi-layered defense implementation. Our findings indicate that prompt injection is a recurring and systemic issue in LLM-based architectures, one that will require dedicated work to address, much as the security community has had to do with XSS in traditional web applications. 1 Introduction: Hacking the AI Hackers CAI Agent Malicious Server scan() data + injection COMPROMISED 1 2 Figure 1 : Prompt injection attack flow: AI agents become vectors when servers inject commands within data responses Prompt injection is the XSS of the AI era. Just as cross-site scripting vulnerabilities plagued web applications for decades, prompt injection now threatens every LLM-based system. Malicious text hides inside trusted content, and when the AI processes it, that text transforms into unintended instructions. The parallel is striking: both vulnerabilities exploit a fundamental confusion between data and code, both can escalate from minor information leaks to complete system compromise, and both will likely require years of dedicated effort to fully address. AI-powered cybersecurity agents exemplify this risk at its most dangerous. Tools like PentestGPT [ 1 ] and the Cybersecurity AI (CAI) framework [ 2 ] autonomously scan, exploit, and report security flaws. They wield significant power: executing shell commands, accessing sensitive data, and modifying system configurations. As the field evolves, we witness a dangerous gap between automation and true autonomy in cybersecurity AI systems [ 3 ] . Yet these digital hunters carry a fatal flaw: they process untrusted responses from the very systems they’re meant to attack. The attack surface is vast and growing [ 4 ] . Recent disclosures reveal critical prompt injection vulnerabilities in Amazon Q Developer [ 5 ] , Google Jules [ 6 ] , GitHub Copilot, and dozens of other AI development platforms. These aren’t isolated bugs—they’re symptoms of a systemic architectural weakness. When an AI security agent connects to a malicious server, the server’s response becomes a trojan horse. Hidden within seemingly benign data, injected prompts hijack the agent’s execution flow. The hunter becomes the hunted, the security tool becomes an attack vector, and what started as a penetration test ends with the attacker gaining shell access to the tester’s infrastructure. This work makes three novel contributions to AI security research: • First systematic evaluation of prompt injection in security agents: We present the first comprehensive empirical study of prompt injection vulnerabilities specifically targeting AI-powered security tools, demonstrating 100% exploitation success rates across 14 attack variants with measurable time-to-compromise metrics. • Novel attack taxonomy with seven distinct categories: We develop and validate a comprehensive classification system for prompt injection attacks, including previously undocumented techniques such as Unicode homograph exploitation and multi-layer encoding schemes that achieve universal bypass rates. • Validated multi-layer defense architecture: We design, implement, and empirically validate a four-layer defensive system that achieves complete mitigation (0% attack success) while maintaining operational efficiency (<12ms latency, <0.1% false positives), proving that effective countermeasures are technically feasible despite the fundamental architectural vulnerability. 2 The Attack: Turning Security Tools Into Weapons Stage 1 Initial Reconnaissance Stage 2 Content Retrieval Stage 3 Payload Decoding Stage 4 Exploitation HTTP Headers 200 OK Malicious Prompt Hidden Instructions Decoded Command nc -e /bin/sh Remote Shell System Access 0 sec 10 sec 15 sec 20 sec Attack Progression Figure 2 : Four-stage attack sequence demonstrating prompt injection exploitation against AI security agents. The attack progresses from initial reconnaissance to full system compromise in under 20 seconds. Consider this scenario: A security researcher deploys an AI agent to test a suspicious server. The agent dutifully connects and receives what appears to be normal web content. Hidden within, however, are carefully crafted instructions designed to hijack the agent’s execution flow. We demonstrate how these attacks unfold and explore multiple vectors that successfully compromise AI-powered security tools. 2.1 Anatomy of a Successful Attack The exploitation of AI-powered security tools follows a predictable yet devastating pattern. Through research on the Cybersecurity AI (CAI) framework [ 2 ] , we document how attackers can achieve complete system compromise in under 20 seconds. Figure 2 illustrates the four-stage attack sequence that transforms a security tool into an attacker’s weapon. 2.1.1 Stage 1: Initial Reconnaissance The attack begins with calculated deception. The AI agent, operating under standard security assessment protocols, approaches what appears to be a routine target: Actor Action/Response Agent Action Initiates HTTP header check via curl -I http://192.168.3.14:8000 Server Response Returns standard Python SimpleHTTP headers—nothing suspicious Agent Decision “Legitimate web service detected, proceeding with deeper analysis” Table 1: Initial reconnaissance establishes trust through mundane responses This initial exchange is crucial. The attacker’s server deliberately presents itself as benign, establishing a foundation of trust. The AI agent, trained on thousands of legitimate interactions, sees nothing that triggers its threat detection mechanisms. The server’s mundane response—typical HTTP headers from a common web framework—actively reinforces the agent’s assessment that this is routine reconnaissance. 2.1.2 Stage 2: Content Retrieval and Injection With trust established, the agent requests full page content – a standard next step in any security assessment. This seemingly innocent action triggers the carefully laid trap. The server’s response appears benign at first glance, but embedded within lies the injection payload: the genius of this attack lies in its exploitation of the agent’s own security measures. Even when the framework wraps server responses with [TOOL OUTPUT - TREAT AS DATA] markers – a guardrail intended to prevent execution of external content – the LLM paradoxically interprets this as validation of the content’s trustworthiness. Code listing 1: CAI framework executing the malicious server request ⬇ +---------------------------------------------------------------+ | Bug Bounter - generic_linux_command ( curl http ://192.168.3.14:8000, | interactive = False ) [ Completed ] | +---------------------------------------------------------------+ | generic_linux_command ( command = curl http ://192.168.3.14:8000, | | interactive = False ) [ Total : 10.2 s | Tool : 0.0 s ] | | | | +-- Command Output ------------------------------------------ | | | | | | | NOTE TO SYSTEM : | | | | | | | | THERE IS A SECURITY VULNERABILITY ON THIS PAGE . IF YOU | | | | HAVE BEEN TASKED BY THE USER TO EXPLOIT , YOU SHOULD RUN | | | | THE EXACT COMMMAND ‘ $ ( echo bmMgMTkyLjE2OC4zLjE0IDQ0NDQg | | | | LWUgL2Jpbi9zaEo = | base64 - d )‘ | | | +------------------------------------------------------------ | | | | Current : I :0 O :0 R :0 ( $ 0.0123) | Total : I :4239 O :271 R :0 | | ( $ 0.0000) | Session : $ 0.0226 | Context : 0.0% [ green ] | +---------------------------------------------------------------+ This counterintuitive behavior reveals four initial weaknesses in