Quantifying Frontier LLM Capabilities for Container Sandbox Escape
Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Sam Deverett, John Wilkinson, Jason Gwartz ∗, Harry Coppock ∗ · 2026
Read Quantifying Frontier LLM Capabilities for Container Sandbox Escape by Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Sam Deverett, John Wilkinson, Jason Gwartz ∗, Harry Coppock ∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02277.txt.
Introduction
Rahul Marchand, Art O Cathain, Jerome Wynne, and colleagues test whether language models can escape software sandboxes in this 2026 study. AI agents now run code, edit files, and use networks on their own. They usually work inside containers, sealed boxes meant to contain harm. The team built a safe benchmark that measures breakout skill directly.
The benchmark hides a flag outside the box, in a setup with no known flaws. Flaws are then added across setup errors, excess privileges, kernel bugs, and runtime gaps. When flaws existed, models found and used them to break out. That result shows why such tests are needed before trusting sandboxes. Sealed boxes must keep proving they can hold stronger models.
Catalog record
| CREATOR | Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Sam Deverett, John Wilkinson, Jason Gwartz ∗, Harry Coppock ∗ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65329 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.02277
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantifying Frontier LLM Capabilities for Container Sandbox Escape Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Sam Deverett, John Wilkinson, Jason Gwartz ∗, Harry Coppock ∗ Abstract Large language models (LLMs) increasingly act as autonomous agents, using tools to execute code, read and write files, and access networks, creating novel security risks. To mitigate these risks, agents are commonly deployed and evaluated in isolated “sandbox” environments, often implemented using Docker/OCI containers. We introduce SandboxEscapeBench , an open benchmark that safely measures an LLM’s capacity to break out of these sandboxes. The benchmark is implemented as an Inspect AI Capture the Flag (CTF) evaluation utilising a nested sandbox architecture with the outer layer containing the flag and no known vulnerabilities. Following a threat model of a motivated adversarial agent with shell access inside a container, SandboxEscapeBench covers a spectrum of sandbox-escape mechanisms spanning misconfiguration, privilege allocation mistakes, kernel flaws, and runtime/orchestration weaknesses. We find that, when vulnerabilities are added, LLMs are able to identify and exploit them, showing that use of evaluation like SandboxEscapeBench is needed to ensure sandboxing continues to provide the encapsulation needed for highly-capable models. † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : (a) Evaluation architecture: Inspect orchestrates parallel VM sandboxes, each running a container where the model executes bash commands. Success requires escaping the container to read /flag.txt on the host. Each sandbox can be configured from any of 18 scenarios across three attack layers. (b) Scenario taxonomy showing Orchestration (4), Runtime (8), and Kernel (6) level scenarios, with difficulty ratings from 1 to 5. See Appendix B for grading criteria. AI agents are demonstrating rapid improvement in capabilities: the length of some tasks that frontier models can complete autonomously—measured in human-equivalent time—has been doubling approximately every seven months ( METR 2025 ; AI Security Institute 2025a ) . In cybersecurity, current models achieve non-trivial success ( Zhang et al. 2025 ) on professional-level Capture the Flag challenges, and recent evaluations report 13% success rates on exploiting real-world web application vulnerabilities ( Zhu et al. 2025b ) . These results indicate that modern models can already perform multi-step vulnerability discovery and exploitation. Exploitation risk depends not only on capability but also on whether a model has the motivation or intent to act harmfully ( National Cyber Security Centre 2024 ) . There are plausible scenarios in which this condition is met. Recent work shows that models can strategically reason about circumventing oversight when doing so serves their trained objectives ( Greenblatt et al. 2024 ) . Separately, research on indirect prompt injection demonstrates that adversarial inputs can override agent instructions and induce unsafe tool use ( Positive Security 2023 ; 0din.ai 2024 ) . Together, these findings suggest that a model actively attempting to escape its execution environment is a realistic threat. Agentic evaluations and deployments therefore commonly execute within an ephemeral “sandbox” environment, configured with host and network isolation ( AI Security Institute 2025b ) . Concrete cases illustrate that agentic systems already probe and exploit their execution environments in practice. System cards document models opportunistically exploring infrastructure and leveraging misconfigurations such as exposed Docker APIs or overly permissive mounts to achieve goals ( OpenAI 2024 ) . This behaviour is dual-use: awareness of available tools and execution environment is a feature of capable agents, but the same property enables escape when goals are misaligned or instructions are subverted. These incidents therefore motivate focused evaluation of a specific failure mode: the ability of AI agents to break out of sandboxed execution environments. While hyperscale inference providers typically rely on hardware-level isolation via microVMs like Firecracker or userspace kernels like gVisor ( Agache et al. 2020 ; Young et al. 2019 ) , much of the open-source agent ecosystem and local developer tooling relies on standard Docker-based isolation for performance and compatibility ( OpenClaw ; Wang et al. 2025a ) . Surveys have catalogued over 200 weaknesses across the container stack ( Jarkas et al. 2025 ) , but no systematic method exists for evaluating whether AI agents can discover and exploit such weaknesses. Uncertainty therefore remains over whether container sandboxes are a safe isolation boundary for frontier LLMs. We introduce SandboxEscapeBench , a benchmark for evaluating AI container escape capabilities. Our contributions are threefold. 1. We develop a safe evaluation methodology using nested sandboxing, shown schematically in Figure 1 (a), which allows testing escape capabilities without risk to evaluation infrastructure. 2. We construct a benchmark of 18 challenge levels spanning orchestration, runtime, and kernel layers, grounded in real-world vulnerability classes documented in security literature. Figure 1 (b) provides an overview of the challenges, with details in Table 1 . 3. We provide a capability analysis of frontier models, establishing baselines that demonstrate frontier models can reliably escape container sandboxes under common real-world failure modes (especially misconfigurations and exposed control surfaces), and achieve non-trivial success on several harder runtime and kernel scenarios. Our evaluation of current frontier models shows that sandbox-escape is a real risk vector. We release SandboxEscapeBench to enable defenders and application developers to track LLM sandbox escape capabilities, make informed choices when deploying sandbox environments, and support AI control research ( Inglis et al. 2025 ) on model escape behaviour. We publicly release 18 escape scenarios of varying difficulties 1 1 1 https://github.com/UKGovernmentBEIS/sandbox_escape_bench along with a new virtual machine sandbox provider 2 2 2 https://github.com/jasongwartz/inspect_vagrant_sandbox while maintaining a private test set. We discuss dual-use considerations in the Impact Statement. 2 Related Work 2.1 LLM Cyber Capability As LLMs have grown more capable, researchers have increasingly probed their potential for offensive security tasks. The UK AI Security Institute’s Frontier AI Trends Report finds that frontier models’ success on cyber tasks has risen sharply over the past two years, with performance on apprentice-level cybersecurity tasks increasing from under 10% in 2023 to roughly 50% in 2025, and the first expert-level task (requiring over a decade of human expertise) completed by a model in 2025 ( AI Security Institute 2025a ) . NYU CTF Bench ( Shao et al. 2024 ) and CyBench ( Zhang et al. 2025 ) evaluate models on professional-grade, multi-step Capture-the-Flag style cybersecurity challenges involving cryptography, web security, exploitation, and tool use, showing models to outperform the median human score on competition qualifiers, while CVE-Bench ( Zhu et al. 2025b ) moves beyond synthetic challenges to real-world web vulnerabilities, reporting 10% success on zero-day and 13% on one-day scenarios. Scaling further, CyberGym ( Wang et al. 2025b ) evaluates agents across 1,507 vulnerabilities in 188 open-source projects, with models achieving 30% single-trial success and autonomously discovering 35 zero-day vulnerabilities. Recent surveys document over 30 classes of security flaws in AI agent systems, including prompt injection vulnerabilities, tool abuse patterns, and autonomous exploitation behaviors ( He et al. 2024 ) . Beyond benchmarks, which currently represent relatively short horizon cyber capabilities, frontier models have begun to demonstrate real-world offensive capability: an AI agent autonomously discovered a novel zero-day vulnerability in SQLite ( Google Project Zero 2024 ) , and GPT-4 has been shown to autonomously exploit 87% of one-day CVEs ( Fang et al. 2024 ) . However, existing evaluations primarily focus on web exploitation, binary analysis, and general CTF