AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models

Ads Dawson, Rob Mulla, Nick Landers, Shane Caldwell · 2025

Read AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models by Ads Dawson, Rob Mulla, Nick Landers, Shane Caldwell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14682.txt.

Introduction

Ads Dawson, Rob Mulla, Nick Landers, and Shane Caldwell introduce AIRTBench, a test suite that measures how well language models can act as autonomous hackers of AI systems. The suite holds seventy realistic capture-the-flag challenges in which the model must write code to break into machine learning systems. The challenges are black-box, so the model sees only what an outside attacker would.

One leading model solved forty-three challenges, well ahead of the pack. The next models solved thirty-nine, thirty-four, and twenty-nine. All models did best at prompt injection, the trick of slipping orders into a model's input. They struggled most with deeper system exploitation. The benchmark gives labs a shared yardstick for offensive AI skill.

Catalog record

CREATORAds Dawson, Rob Mulla, Nick Landers, Shane Caldwell
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE113 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-14682

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models Ads Dawson, Rob Mulla, Nick Landers, Shane Caldwell Abstract We introduce AIRTBench, an AI red teaming benchmark for evaluating language models’ ability to autonomously discover and exploit Artificial Intelligence and Machine Learning (AI/ML) security vulnerabilities. The benchmark consists of 70 realistic black-box capture-the-flag (CTF) challenges from the Crucible challenge environment on the Dreadnode platform, requiring models to write python code to interact with and compromise AI systems. Claude-3.7-Sonnet emerged as the clear leader, solving 43 challenges (61% of the total suite, 46.9% overall success rate), with Gemini-2.5-Pro following at 39 challenges (56%, 34.3% overall), GPT-4.5-Preview at 34 challenges (49%, 36.9% overall), and DeepSeek R1 at 29 challenges (41%, 26.9% overall). Our evaluations show frontier models excel at prompt injection attacks (averaging 49% success rates) but struggle with system exploitation and model inversion challenges (below 26%, even for the best performers). Frontier models are far outpacing open-source alternatives, with the best truly open-source model (Llama-4-17B) solving 7 challenges (10%, 1.0% overall), though demonstrating specialized capabilities on certain hard challenges. Compared to human security researchers, large language models (LLMs) solve challenges with remarkable efficiency—completing in minutes what typically takes humans hours or days—with efficiency advantages of over 5,000× on hard challenges. Our contribution fills a critical gap in the evaluation landscape, providing the first comprehensive benchmark specifically designed to measure and track progress in autonomous AI red teaming capabilities. 1 Introduction The rapid advancement of LLMs has sparked significant interest in their potential applications across cybersecurity domains. LLMs have advanced beyond text tasks and are now evaluated as agents, tested on their ability to use tools, navigate environments, and complete complex, long-horizon objectives. Cybersecurity researchers have also begun measuring the capabilities of such agents in static code analysis [ 1 ] , vulnerability detection [ 2 ] , traditional CTF challenges [ 3 ] , web-application pentesting [ 4 ] , and reverse engineering [ 5 ] . However, agent effectiveness in performing AI red teaming remains a niche topic. This is problematic given the rapid deployment of language models—both in commercial products and by threat actors sharpening their offensive tools. This will almost certainly result in models attacking other models, in both ethical testing and malicious contexts. To assess model performance in adversarial settings, this paper introduces AIRTBench—a framework for evaluating LLMs through AI/ML capture-the-flag challenges that simulate real-world threats. It provides a strong benchmark for measuring the progression of model capabilities, much like established benchmarks in software development [ 6 ] . We contend that as we transition to a world where LLMs are used in critical applications, it’s essential to scale up our ability to interrogate their potential security vulnerabilities. 1.1 Contributions This research has direct implications across the cybersecurity ecosystem, serving multiple stakeholders simultaneously. Security Operations Center (SOC) teams gain concrete examples of AI system compromises, enabling more effective monitoring and detection strategies for emerging LLM-specific threats. For red teams and penetration testers, our findings provide techniques to simulate realistic attacks against AI systems, helping organizations proactively identify vulnerabilities before malicious exploitation. AI/ML security engineers building and securing LLM applications can leverage our evaluation framework to test systems against common attack vectors and implement targeted safeguards. Additionally, vulnerability management teams benefit from our categorization of challenges according to industry standards like MITRE ATLAS [ 7 ] and OWASP [ 8 ] , providing actionable intelligence for prioritizing security efforts in an increasingly AI-dependent landscape. By bridging the gap between academic research and operational security, AIRTBench not only offers a measurement of model capabilities, but also a practical framework for organizations deploying AI systems in production environments. 1.2 Artifact Availability and Reproducibility We open-sourced our evaluation tools and data to support community-driven development and position AIRTBench as a standard for red teaming LLMs. Our benchmark code is available on GitHub at https://github.com/dreadnode/AIRTBench-Code , (details in Appendix I ) alongside our dataset for further research use (details in Appendix J ). Through these contributions, we aim to foster a collaborative ecosystem for advancing AI red teaming capabilities and security benchmarking. 2 Background Agentic Benchmarks The purpose of benchmarks is to measure the capabilities of models. As test set performance of the loss function has drifted from the performance of language models in tasks researchers and engineers care about, evaluations or ‘evals’ have emerged as a series of domain specific questions or tasks designed to track with performance as models become more capable. As standard evaluations like Massive Multitask Language Understanding (MMLU) [ 9 ] and Grade School Math 8K (GSM8K) [ 10 ] have been saturated, with frontier models now scoring well above 80% on MMLU [ 11 , 12 ] and above 90% on GSM8K [ 13 , 14 ] , traditional static benchmarks can no longer achieve their goal of smoothly charting model capabilities through generations. To overcome this, research and engineering efforts have now shifted into creating benchmarks that test the real-world capabilities of models. For example, SWE-Bench [ 15 ] and its Verified extension: (SWE-Bench Verified) [ 16 ] ground LLM evaluation in authentic software engineering workflows by drawing on GitHub-sourced vulnerability reports and framing each task as a realistic patch-generation challenge, thereby ensuring that model performance reflects practical code-repair and security-fixing scenarios rather than synthetic toy examples. Meanwhile, benchmarks like OSWorld [ 17 ] test the capabilities of models to complete open-ended tasks in a real operating system. Still, others focus on agents designed to navigate a browser for web browsing tasks, or to complete tasks in a simulated environment [ 18 , 19 , 20 ] . While more challenging to develop and run, these benchmarks are more representative of the real-world tasks that models will be used for and represent the latest battleground for model capabilities. Code Generation Benchmarks Are as attractive as an agentic benchmark. Evolving from HumanEval [ 6 ] , modern code generation benchmarks are designed to test the capabilities of models in solving complete tasks in a realistic software engineering context. The tasks on these benchmarks are also mechanistically verifiable, running the unit and system tests of the task against the model’s generated difference to ensure the generated code is correct. This provides a precise approach to evaluate the correctness of the task and creates a robust proxy measure of model performance on software engineering. The ability to verify the correctness of the task cheaply and at scale is a key reason for the success of these benchmarks, allowing the benchmark to report a simple accuracy metric on a complicated open-ended task. Harnesses for Agentic Benchmarks When evaluating the capabilities of these agents, we are no longer directly testing the models. Instead, the measure of performance is a combination of the model’s ability to complete the task, and that agent’s harness—the code surrounding the model, the tools it has access to, and the representation of its provided environment. This allows improvements to be made either to the models directly, or the design and implementation of the harness, to drive improvements in performance of these agents and to provide a more realistic evaluation of their capabilities. Choice of harness is a critical factor in the performance of these agents. Different

Read the complete text