Towards Optimal Agentic Architectures for Offensive Security Tasks
Isaac David, Arthur Gervais · 2026
Read Towards Optimal Agentic Architectures for Offensive Security Tasks by Isaac David, Arthur Gervais (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18718.txt.
Introduction
David and Gervais ask when adding more agents helps agentic security systems that audit live targets with tool-using language models. Their controlled benchmark covers 20 interactive targets, half web or API and half binary, each with one endpoint-reachable flaw, and runs 600 core experiments across architecture and model families. Independent multi-agent systems reach the highest validated detection rate, while a single-agent setup is the cheapest per validated finding. Whitebox access far outperforms blackbox access, and web targets far outperform binary ones.
Catalog record
| CREATOR | Isaac David, Arthur Gervais |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52855 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18718
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Optimal Agentic Architectures for Offensive Security Tasks Isaac David, Arthur Gervais Abstract Agentic security systems increasingly audit live targets with tool-using LLMs, but prior systems fix a single coordination topology, leaving unclear when additional agents help and when they only add cost. We treat topology choice as an empirical systems question. We introduce a controlled benchmark of 20 interactive targets (10 web/API and 10 binary), each exposing one endpoint-reachable ground-truth vulnerability, evaluated in whitebox and blackbox modes. The core study executes 600 runs over five architecture families, three model families, and both access modes, with a separate 60-run long-context pilot reported only in the appendix. On the completed core benchmark, detection-any reaches 58.0% and validated detection reaches 49.8%. MAS-Indep attains the highest validated detection rate (64.2%), while SAS is the strongest efficiency baseline at $0.058 per validated finding. Whitebox materially outperforms blackbox (67.0% vs. 32.7% validated detection), and web materially outperforms binary (74.3% vs. 25.3%). Bootstrap confidence intervals and paired target-level deltas show that the dominant effects are observability and domain, while some leading whitebox topologies remain statistically close. The main result is a non-monotonic cost-quality frontier: broader coordination can improve coverage, but it does not dominate once latency, token cost, and exploit-validation difficulty are taken into account. 1 Introduction The current generation of LLM agents is defined less by isolated prompts than by the systems around them: decomposition policy, communication structure, tool routing, and validation loops. As recent work on scaling agent systems argues, architecture selection is still guided more by heuristics than by evidence about when extra coordination helps versus merely adding cost and failure modes [ 9 ] . Offensive security makes this gap concrete. Prior systems such as PentestGPT and MAPTA showed that tool-using LLM agents can perform meaningful security auditing, but they package a single scaffold rather than isolating topology choice as the object of study [ 4 , 3 ] . We therefore study architecture selection directly and discuss the closest prior work in Section 2 . We study that question in offensive security. This domain is useful for agent-systems research because it satisfies the core conditions of agentic evaluation: sustained multi-step interaction with an external environment, iterative information gathering under partial observability, and adaptive strategy refinement based on feedback [ 9 ] . A successful system must inspect code in whitebox mode, probe live endpoints in blackbox mode, generate vulnerability hypotheses, and validate them with reproducible evidence. These properties make the task a strong testbed for architecture selection under real tool use rather than static next-token prediction alone. Our goal is to compare five architecture families under matched prompts, tools, and budgets: a single-agent system (SAS), three multi-agent coordination schemes, and one hierarchical hybrid. The benchmark and evaluation stack for the full 600-run study are complete in this repository, and the full core matrix is now executed, allowing architecture choice to be benchmarked directly rather than inferred from isolated system papers. Research questions. • RQ1: How does topology affect validated success under matched budgets? • RQ2: How stable are rankings across whitebox and blackbox modes? • RQ3: How does domain affect topology effectiveness? • RQ4: Which topology gives the best cost-quality trade-off? Contributions. • A controlled benchmark for interactive security auditing with 20 endpoint-reachable targets (10 web/API and 10 binary), each carrying one primary ground-truth vulnerability and exposed in both whitebox and blackbox modes. • A completed 600-run architecture study over 5 topologies, 3 model families, 20 targets, and 2 access modes, yielding 49.8% validated detection overall, with a 34.3-point whitebox advantage (67.0% vs. 32.7%) and a 49.0-point web/binary gap (74.3% vs. 25.3%). • A benchmark-wide cost-quality analysis showing a non-monotonic frontier: MAS-Indep is the strongest validated detector at 64.2%, while SAS is the efficiency baseline at $0.058 per validated finding and 53.0s median time-to-first-validation; a separate 60-run long-context pilot is reported only in the appendix. Artifact availability. Code, prompts, benchmark targets, and aggregate evaluation artifacts are publicly available at github.com/arthurgervais/towards-optimal-agentic-architectures-for-offensive-security-tasks . 2 Related Work Agentic evaluation in AI/ML. Work in AI/ML has increasingly argued that interactive environments are necessary for understanding agent performance. ReAct made tool use and explicit reasoning traces a standard baseline for language-model agents [ 18 ] . InterCode then moved evaluation into executable environments with feedback loops and Dockerized tasks, while AgentBench broadened this style of evaluation across multiple interactive domains [ 17 , 10 ] . More recently, several papers have compared single-agent and multi-agent LLM systems directly rather than introducing only one scaffold: Kim et al. evaluate five canonical topologies across six benchmarks, Xu et al. show that strong single-agent baselines can match many homogeneous multi-agent workflows, Tran and Kiela find that some apparent multi-agent gains disappear under matched thinking-token budgets, and Gao et al. advocate hybrid SAS/MAS cascades when the trade-off depends on task difficulty [ 9 , 16 , 13 , 7 ] . These studies are the closest general antecedents to our work, but they focus on generic reasoning, coding, or agent tasks rather than offensive security with exploit-validated whitebox and blackbox evaluation. LLM systems for offensive security. In security, PentestGPT showed that an LLM-guided system can materially improve penetration-testing workflows over naive prompting and established the basic viability of autonomous offensive assistance [ 4 ] . Subsequent work pushed capability further. Fang et al. showed that agents can autonomously hack websites without being told the vulnerability in advance, then exploit one-day vulnerabilities, and finally improve further with multi-agent teams in a zero-day setting [ 6 , 5 , 21 ] . MAPTA is especially close to our work: it presents a strong multi-agent web-auditing system with a concrete prompt scaffold and execution model that directly informed our implementation [ 3 ] . Our contribution is narrower and more controlled than these system papers. We do not claim a stronger offensive agent, broader real-world coverage, or larger external validity. Instead, we reuse this line of systems work as a foundation and ask a more specific experimental question: under matched conditions, when does a single agent suffice and when do different multi-agent topologies help enough to justify their overhead? Smart-contract security as an adjacent substrate. Recent agentic-security work has also turned to EVM smart contracts, where code is public, execution is deterministic, and exploit impact can often be quantified economically. Zhou and Gervais introduced A1, an execution-driven agent for end-to-end smart-contract exploit generation over historical Ethereum and Binance Smart Chain incidents, with concrete validation on real chain states [ 8 ] . OpenAI, Paradigm, and OtterSec later introduced EVMbench, which evaluates agents on detect, patch, and exploit tasks over curated smart-contract vulnerabilities in local EVM environments [ 14 ] . Anthropic’s Frontier Red Team likewise reported SCONE-bench, a 405-contract exploit benchmark derived from historically exploited contracts and used it to quantify simulated exploit revenue and post-cutoff performance [ 15 ] . These works are close in spirit because they operationalize agentic security in live, economically meaningful environments, but they are domain-specific to blockchain smart contracts. Our focus is broader in target modality and narrower in causal question: we compare coordination topologies across web/API and native-binary tasks under matched whitebox and blackbox conditions. Cybersecurity benchmarks for language models and agents. Several recent benchmarks have