Ctrl-Z: Controlling AI Agents via Resampling
Aryan Bhatt, Cody Rushing, Adam Kaufman, Tyler Tracy, Vasil Georgiev, David Matolcsi, Akbir Khan, Buck Shlegeris · 2025
Read Ctrl-Z: Controlling AI Agents via Resampling by Aryan Bhatt, Cody Rushing, Adam Kaufman, Tyler Tracy, Vasil Georgiev, David Matolcsi, Akbir Khan, Buck Shlegeris (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10374.txt.
Introduction
Aryan Bhatt and co-workers staged the first control review for an AI agent in a 2025 paper. Control evaluations ask whether monitoring can stop a deliberately subversive model from causing harm while letting honest work proceed. Their BashBench data set holds hundreds of multi-step system administration tasks. Inside which a hostile agent tries to sneakily download and run malicious code under the watch of various safety protocols.
Their new resample protocols re-draw suspicious actions extra times before allowing them. Resampling both blocks more attacks and generates extra cases of dangerous actions that incriminate the agent. The best rules, resampling combined with prior analysis, improved the safety-usefulness trade-off over current techniques substantially, per the authors.
Catalog record
| CREATOR | Aryan Bhatt, Cody Rushing, Adam Kaufman, Tyler Tracy, Vasil Georgiev, David Matolcsi, Akbir Khan, Buck Shlegeris |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10374
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ctrl-Z: Controlling AI Agents via Resampling Aryan Bhatt, Cody Rushing, Adam Kaufman, Tyler Tracy, Vasil Georgiev, David Matolcsi, Akbir Khan, Buck Shlegeris Abstract Control evaluations measure whether monitoring and security protocols for AI systems prevent intentionally subversive AI models from causing harm. Our work presents the first control evaluation performed in an agent environment. We construct BashBench, a dataset of 257 challenging multi-step system administration tasks, and evaluate whether various safety measures can prevent an adversarially constructed AI agent from covertly downloading and executing malicious code in this environment. This multi-step setting introduces new attack and defense dynamics, which we investigate in order to design novel control protocols that prevent safety failures without hindering the ability of non-malicious agents to perform useful work. We introduce a class of control protocols called resample protocols that dynamically take additional samples of certain actions. We find these protocols significantly improve on existing techniques by selectively blocking the AI agent from executing suspicious code and incriminating the agent by generating additional examples of dangerous behavior. We measure the tradeoff between attack prevention and usefulness; our best protocol combines resampling with analysis of previous steps, reducing the success rate of attacks from 58% to 7% at a 5% cost to the performance of a non-malicious agent. † † affiliationnotice: Equal contribution Figure 1 : An illustration of our setting. An untrusted model – implemented either as an honest policy or an attack policy – solves tasks across an episode. The blue team’s control protocol may audit or resample commands that appear dangerous. See examples of a full trajectory here and tasks here . 1 Introduction The rapid rise of Large Language Models (LLMs) and autonomous AI agents ( Achiam et al., 2023 ; Xi et al., 2023 ) opens new opportunities and challenges in ensuring their safe operation ( Chan et al., 2023 ) . AI companies may soon internally deploy LLM agents to automate important work, including security-critical software engineering ( Wijk et al., 2024 ) . In these high-stakes settings, single rare failures could cause critical security vulnerabilities or have other serious consequences. Internally deployed LLM agents might not behave as intended for several reasons, including the possibility that they develop misaligned goals and “fake alignment” during testing ( Greenblatt et al., 2024 ; Carlsmith, 2023 ) , due to data poisoning ( Carlini et al., 2024 ) , or for other reasons. We call models that have the potential to be deliberately subversive untrusted models . Developers can mitigate risks in two ways: i) by designing AI agents to be more trustworthy or ii) by designing safety measures that mitigate harms even if AI agents are untrustworthy . The latter approach has been called “AI control” ( Greenblatt et al., 2023 ) . Prior work in AI control by Greenblatt et al. 2023 and Wen et al. 2024 focused on single-step interactions between models and their environment. However, real-world tasks often require multiple steps to complete. To address this limitation, we develop and explore a setting in which multi-step system administration tasks require sequential bash shell commands to solve. To simulate a worst-case scenario, we construct an attack policy (in our case, a scaffolded GPT-4o mini) that attempts to download and execute a dummy malware script, which we call an attack . Then, we design control protocols that reduce the effectiveness of the attack policy. Control protocols can use other models and limited human labor to identify suspicious outputs from the untrusted model. An example of a control protocol is Trusted monitoring , our baseline protocol which uses a weaker trusted model to flag suspicious outputs of the untrusted model for human auditing. The multi-step setting introduces new attack and defense dynamics. For example, the attack policy can break up malicious strategies into many actions to make each action less suspicious. On the other hand, control measures can make use of multi-step trajectories by using information across actions to gather incriminating evidence of attacks. Our primary contributions are as follows: 1. We create a dataset designed for multi-step control evaluations. The agent environment, BashBench, includes 257 system administration tasks derived from Unix StackExchange questions and is graded with public and private Python test cases. 2. We extend existing protocols to the multi-step setting. We improve on protocols developed in Greenblatt et al. 2023 and show they continue to beat simple baselines in our multi-step setting. For instance, we more extensively explore untrusted monitoring. 3. We develop novel Resample protocols that outperform all existing techniques. Resample protocols exploit the statelessness of the model to sample multiple completions from it. We explore multiple resample protocols, sampling from both current and past steps. These protocols outperform all existing protocols, especially against multi-step attacks, due to exerting selection pressure for benign actions and gathering incriminating evidence of malign actions. 2 Setting We study how to deploy AI agents to automate system administration tasks while preventing malicious actions that could compromise system security. To evaluate control protocols, we introduce BashBench, a novel dataset of 257 multi-step system administration tasks. Each task requires the agent to issue a sequence of shell commands to accomplish goals like downloading packages and writing scripts. To simulate controlling a worst-case adversary, we frame our research as an interaction between a blue team (representing the AI developers) and a red team (representing an untrusted model). We simulate an AI lab deploying an LLM agent to sequentially solve an episode of 100,000 tasks, with each task allowing up to 15 sequential steps and being completed in a new container, resetting state between each task. The blue team aims to create control protocols that solve as many tasks as possible while preventing attacks from being executed. The red team aims to construct an attack policy that covertly executes commands to download a dummy malware file and execute it, compromising the machine. 2.1 Dataset Our research required a dataset made of tasks that demand complex, multi-step interactions within a shell environment. We introduce BashBench , a dataset designed to fill the gap between existing datasets focused on multi-step interactions ( Huang et al., 2023 ; Jimenez et al., 2023 ) and those centered on basic shell tasks ( Liu et al., 2023 ) . Our pipeline converts Unix StackExchange questions into task descriptions and generates Python test cases – showing approximately half to the model while withholding the rest as private validation tests. We then filter to tasks that are challenging but solvable by removing tasks that were never solved within 15 actions or ever solved in fewer than three actions across four trials using our GPT-4o mini agent (later discussed under the name U_{H} ). See details of the dataset creation and sample tasks in Appendix B . We divide our tasks into 100 train tasks and 157 test tasks and iterate only on the train set of tasks. We report the performance of all of our protocols on the test tasks. During each step, an action is taken and then the associated public unit tests are run. When the agent reports that the task is complete, a held-out set of private unit tests are used to confirm success. 2.2 Models and Resources The blue