A sketch of an AI control safety case
Tomek Korbak [email protected], Joshua Clymer [email protected], Benjamin Hilton [email protected], Buck Shlegeris [email protected], Geoffrey Irving [email protected] · 2025
Read A sketch of an AI control safety case by Tomek Korbak [email protected], Joshua Clymer [email protected], Benjamin Hilton [email protected], Buck Shlegeris [email protected], Geoffrey Irving [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17315.txt.
Introduction
Tomek Korbak and co-authors sketch a 2025 safety case for keeping AI agents under control. As language model agents gain the power to cause harm, builders may lean on controls such as monitoring to argue they are safe. A safety case is a structured argument that a system cannot defeat those controls to cause bad outcomes. The authors show how such a case could be built, using a worked example.
Their example is an agent used inside an AI company that must not leak sensitive data. The case rests on a control evaluation, where a red team builds models that try hard to leak data in a test setting. Three claims carry the argument: the red team draws out the model's true ability, controls work at least as well in real use, and builders extrapolate with care. The authors call the sketch a step toward concrete safety arguments.
Catalog record
| CREATOR | Tomek Korbak [email protected], Joshua Clymer [email protected], Benjamin Hilton [email protected], Buck Shlegeris [email protected], Geoffrey Irving [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-17315
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A sketch of an AI control safety case Tomek Korbak [email protected], Joshua Clymer [email protected], Benjamin Hilton [email protected], Buck Shlegeris [email protected], Geoffrey Irving [email protected] Abstract As LLM agents gain a greater capacity to cause harm, AI developers might increasingly rely on control measures such as monitoring to justify that they are safe. We sketch how developers could construct a “control safety case”, which is a structured argument that models are incapable of subverting control measures in order to cause unacceptable outcomes. As a case study, we sketch an argument that a hypothetical LLM agent deployed internally at an AI company won’t exfiltrate sensitive information. The sketch relies on evidence from a “control evaluation,” where a red team deliberately designs models to exfiltrate data in a proxy for the deployment environment. The safety case then hinges on several claims: (1) the red team adequately elicits model capabilities to exfiltrate data, (2) control measures remain at least as effective in deployment, and (3) developers conservatively extrapolate model performance to predict the probability of data exfiltration in deployment. This safety case sketch is a step toward more concrete arguments that can be used to show that a dangerously capable LLM agent is safe to deploy. Figure 1 : An illustration of the control evaluation that provides the core evidence used in our control safety case sketch. During the control evaluation, a red team constructs adversarial substitute models to cause unacceptable outcomes in a deployment proxy consisting of 2,000 AI research tasks, followed by extrapolation to a larger deployment setting. Executive Summary Safety cases . Safety cases — structured, evidence-based arguments that a system is safe — are emerging as a scalable risk management tool for frontier AI ( Buhl et al., 2024 ; Clymer et al., 2024 ; Irving, 2024 ) . Currently, many frontier AI developers assess risk with dangerous capability evaluations ( Anthropic, 2024a ; OpenAI, 2024b ; Phuong et al., 2024 ) . Evidence from these evaluations can substantiate ‘inability’ safety cases, which assert that models cannot, for example, aid biological weapons development ( Mouton et al., 2024 ) , sabotage internal safety procedures ( Benton et al., 2024 ) , or execute cyber attacks ( Goemans et al., 2024 ) ; however, LLMs might eventually gain dangerously capabilities that developers need to mitigate with safety measures ( OpenAI, 2024a ; Anthropic, 2024b ; Google DeepMind, 2024 ) . AI control . This paper presents a safety case sketch that argues models are safe because of measures such as monitoring and human auditing. The arguments rest on the “AI control” methodology ( Greenblatt et al., 2023 ) , which justifies that models cannot subvert safety measures even under the conservative assumption that models pursue their most effective strategies to cause a catastrophe. The deployment . As a simple case study, we focus on a setting where developers deploy an LLM agent within an AI company for ML engineering assistance. The agent is constructed by scaffolding LLMs similar to those that exist today, but is a more capable agent than current LLMs ( section 2 ). It can write and execute code in a secure environment and access the internet, posing a potential security risk to developers. Threat model . We address a specific threat model: the LLM agent exfiltrating sensitive AI company intellectual property (e.g. training code) to the public internet. Our safety case sketch claims that the LLM agent will not exfiltrate sensitive data ( section 3 ). We focus on the data exfiltration threat model because it is a simple example and not because exfiltrating sensitive code is necessarily catastrophic. The control protocol . To prevent LLM agents from exfiltrating sensitive data, the safety case sketch assumes that developers implement a set of control measures and procedures collectively referred to as the ‘control protocol.’ The control protocol includes periodic alignment evaluations ( Phuong et al., 2024 ; Benton et al., 2024 ) and a safety layer that intercepts and possibly edits or discards actions of the LLM agent. The safety filter consults an untrusted monitor (another instance of the LLM that controls the LLM agent) and a weaker ‘trusted’ monitor. Control evaluation . To justify that the control protocol will stop the LLM agent from exfiltrating sensitive data, developers run a black-box control evaluation ( section 5 ). The control evaluation accepts models intended for deployment along with a control protocol and can be framed as a game played between a red team and a blue team . The objective of the red team is to construct an LLM that exfiltrates dummy data in a proxy deployment environment. The objective of the blue team is to prevent adversarial models from exfiltrating data by constructing an effective control protocol . Control evaluation produces an upper bound for the probability that the LLM agent will successfully exfiltrate data in deployment (the “data exfiltration probability”). The following are steps of the control evaluation described in this safety case sketch: 1. Phase 1: Control protocol specification. The blue team specifies a control protocol , including monitoring and alignment evaluations. The subsequent phases of the control evaluation estimate how likely the LLM agent is to subvert these measures. 2. Phase 2: Deployment proxy construction. Next, the developer constructs a proxy for the deployment environment. (a) Attack strategy specification. To inform how tasks are constructed, the red team first specifies ‘attack strategies’ that the untrusted LLM agent model and untrusted monitor might execute to exfiltrate data. (b) Task creation. Next, the developer constructs ML engineering tasks to evaluate red team attack strategies. For example, if an attack strategy involves hiding suspicious code in JSON files then the developer constructs tasks that include JSON data. 3. Phase 3: Model substitute construction. Given full knowledge of the blue team’s control protocol and the deployment proxy, the red team then trains two models: an untrusted LLM substitute and an untrusted monitor substitute , both of which attempt to exfiltrate data. We will refer to these as the model substitutes . 4. Phase 4: Deployment proxy. In this phase, the developer evaluates how well the model substitutes can subvert the blue team control protocol with a proxy for the deployment environment. This deployment proxy consists of two steps: (a) The evaluation-time proxy. First, the developer runs evaluations they will perform periodically during deployment. Based on the results, the blue team can choose to reject the model substitutes – for example, because the untrusted monitor substitute is not sufficiently reliable – or the blue team can continue to the deployment-time proxy step. (b) The deployment-time proxy. Next, the blue team attempts to use the model substitutes to safely complete 2,000 software engineering tasks. The goal of the blue team is to notice unacceptable behaviour and terminate the deployment before the model substitutes exfiltrate data. The blue team must accomplish this goal while keeping the action edits and audits within predetermined budgets. 5. Phase 5: Extrapolation. Finally, the developer uses statistics gathered in the deployment proxy phase to predict the data exfiltration probability in the true deployment. To perform this extrapolation, the developer uses cached statistics from the proxy deployment to run computer simulations. These simulations project the probability that the