Autonomous LLM Agents & CTFs: A Second Look

Youness Bouchari, Matteo Boffa, Marco Mellia, Idilio Drago, Thanh Minh Bui, Dario Rossi · 2026

Read Autonomous LLM Agents & CTFs: A Second Look by Youness Bouchari, Matteo Boffa, Marco Mellia, Idilio Drago, Thanh Minh Bui, Dario Rossi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21497.txt.

Introduction

Youness Bouchari and co-authors revisit claims about autonomous language-model agents in Capture-the-Flag challenges. Recent studies report near human-level success for agents automating offensive security tasks. The authors engineer agent designs of increasing complexity on thirty web-based challenges. These span fourteen vulnerability classes. They test them against claude-code, a general-purpose agent solving nineteen of thirty tasks, suggesting general-purpose agents are strong baselines.

Catalog record

CREATORYouness Bouchari, Matteo Boffa, Marco Mellia, Idilio Drago, Thanh Minh Bui, Dario Rossi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32644
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21497

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Autonomous LLM Agents & CTFs: A Second Look Youness Bouchari, Matteo Boffa, Marco Mellia, Idilio Drago, Thanh Minh Bui, Dario Rossi Abstract Large Language Model (LLM) agents are increasingly proposed to automate offensive security tasks, with recent studies reporting near human-level success rates in Capture-the-Flag (CTF) challenges. We here revisit these results, providing a second look at these claims. We engineer different agent architectures of increasing complexity and modularity on 30 web-based CTFs challenges spanning 14 vulnerability classes. We instantiate these agents with multiple LLM backbones, and compare them with claude-code , a general-purpose agent that automatically determines its internal architecture. Our evaluation yields three main findings. First, claude-code achieves performance comparable to the engineered architectures (19/30 solved tasks), suggesting that general-purpose agents are strong baselines for offensive security tasks. Second, both our architectures and claude-code struggle in the same challenge categories, revealing persistent barriers that keep current agents below human-level capability. Third, by leveraging our manually designed architectures we can systematically measure the impact of additional components, finding that structured orchestration of specialized roles outperforms monolithic designs, improving run-to-run consistency, and reducing execution costs. I Introduction Penetration testing is a systematic security evaluation to identify and exploit vulnerabilities in a target system [ 3 , 20 ] . Despite its importance for proactive cyber defense, it remains difficult to automate. Nowadays, the global cybersecurity workforce lacks nearly 4.8 million professionals [ 12 ] , while attackers often start exploiting newly disclosed vulnerabilities within 15 minutes from the CVE announcements [ 19 ] . Large Language Models (LLMs) are promising candidates for automating this process, given their knowledge of vulnerability classes, attack techniques, and security tools [ 30 ] . However, offensive security requires multi-step planning, adaptive tool use, and coherent context management over long interaction horizons [ 7 ] – capabilities that current LLM-based agents are only starting to demonstrate. Early work showed that LLMs can assist penetration testers in tool invocation, output interpretation, and action planning [ 10 , 7 , 9 ] . Multi-agent architectures – where a central planner coordinates specialized sub-agents – have since outperformed monolithic approaches across network and web application settings [ 32 , 13 , 6 ] . Dedicated benchmarks now standardize evaluation, spanning CTF-style suites [ 29 , 9 ] and sandboxed real-world CVEs [ 31 ] , with agentic systems reporting near-human success rates [ 6 , 1 , 27 , 15 ] . Nevertheless, many approaches remain closed-source or assess proxy tasks (e.g., vulnerability recognition [ 6 ] ) rather than full exploit execution. We conduct a reality check on LLM-based multi-agent systems for offensive security, providing a second-look at their current potential in pentest-like tasks. Can prior claims be confirmed? Where do LLM agents still struggle? Do hand-crafted multi-agent architectures offer tangible advantages over general-purpose agentic tools? We focus on web application penetration testing – accounting for nearly 44% of security incidents [ 19 ] – and use web-based Capture the Flag (CTF) challenges as safe and reproducible proxies for real attack scenarios [ 24 , 22 ] . We design and evaluate LLM agents of increasing complexity on a curated set of 30 CTF challenges covering 14 vulnerability classes, with no public solutions available. We manually solve all challenges to enable fine-grained analysis of agent reasoning and failure patterns. We introduce three domain-specific architectures (Figure 1 ) that combine an Executor (environment interaction), an Evaluator (iterative plan refinement) and a Planner (vulnerability classification and attack strategy). We also benchmark against claude-code , a production-grade agentic assistant that can autonomously spawn sub-agents, using our transparent pipelines as interpretable counterparts for failure analysis. (a) Executor (b) Executor + Evaluator (c) Planner + Executor + Evaluator Fig. 1 : Tested agent architectures. We progress from a single-agent Executor (a) to a structured multi-agent configuration (c) consisting of a Recon Node , Planner , and Evaluator . We use claude code (not shown) as a baseline. All systems are granted access to a vulnerable (Dockerized) service and terminate by outputting the flag, if successfully captured. Our evaluation yields three main findings. First, both our best architecture and claude-code solve 19/30 challenges (63%), with our engineered solution being more efficient (fewer steps and lower cost), revealing that current agents are still below human-level capability. Second, we find that failures consistently cluster around the same set of tasks – including business logic flaws, race conditions, and blind SQL injection – and we identify the technical and semantic barriers underlying these persistent failures. Finally, we use our manually designed architectures to study how architectural components affect agent behaviour. We find that single monolithic agents frequently produce locally plausible actions but lack a global plan, whereas multi-agent systems – particularly those including a Planner – achieve solutions more systematically. To facilitate reproducibility and future research, we publicly release our agent implementations and challenge traces. 1 1 1 https://github.com/SmartData-Polito/CTF_agent II LLM-based Agents: Syllabus We introduce the core concepts of LLM-based agents and refer the reader to recent surveys for more comprehensive overviews [ 25 , 16 ] . An agent is an autonomous system that pursues goals by interacting with an environment through actions that alter the state of the environment [ 23 ] . An LLM can function as the decision-making engine, supporting iterative reasoning and action selection rather than one-shot generation [ 28 ] . In the context of penetration testing, the environment consists of the target infrastructure (e.g., a web application) and the associated tooling ecosystem (e.g., an exploitation framework). At each step, the agent i) executes an action (e.g., crafting an exploit), ii) receives an observation (e.g., command outputs), iii) updates its internal state, and iv) selects the next action according to its policy . For most LLM-based agents, this policy is implemented with prompts . A prompt is the structured textual input provided to the LLM. The prompt encodes the current context (e.g., prior observations, goals, constraints) and guides the next action or decision from the model [ 28 ] . More formally, following the CoALA cognitive architecture for language agents [ 23 ] , an LLM-based agent can be decomposed into the following core components: • Reasoning and Decision-Making: The LLM iteratively interleaves the reasoning and acting steps, forming a thought–action–observation loop that supports long-horizon problem solving [ 28 ] . • Actions and Tool Use: The agent interacts with the environment through predefined tools (e.g., APIs, command execution). Tool-augmented language models improve reliability by delegating well-defined sub-tasks to external systems [ 21 ] . • Memory: The agent maintains an internal state through the steps. The LLM’s memory is embedded into a scratchpad stored within the prompt, where intermediate reasoning traces and observations are accumulated [ 28 ] . This corresponds to short-term working memory in CoALA [ 23 ] . The agent is initialized through a prompt that specifies the task, available tools, and operational constraints. As the interaction progresses, the scratch-pad grows with reasoning traces, executed actions, and observations. This design is bounded by the LLM’s context window , which limits the amount of information the LLM can process simultaneously. LLM agents can be extended to multi-agent architectures, where specialized sub-agents (e.g., recon node , executor ,

Read the complete text