How to evaluate control measures for LLM agents? A trajectory from today to superintelligence
Tomek Korbak [email protected], Mikita Balesni [email protected], Buck Shlegeris [email protected], Geoffrey Irving [email protected] · 2025
Read How to evaluate control measures for LLM agents? A trajectory from today to superintelligence by Tomek Korbak [email protected], Mikita Balesni [email protected], Buck Shlegeris [email protected], Geoffrey Irving [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05259.txt.
Introduction
Tomek Korbak, Mikita Balesni, Buck Shlegeris, and Geoffrey Irving ask how to test safety controls on language model agents fairly as the agents grow stronger. Control evaluations pit a red team, playing a misaligned agent, against the safeguards meant to contain it. If the red team is granted superhuman attack skills, the test misprices the real risk.
Their framework scales the red team's affordances to the actual capability profile of the model being deployed. Five fictional models of rising ability anchor five AI control levels. For each level the authors sketch fitting evaluation rules, control measures, and safety cases. The aim is evaluations that stay practical and affordable without going soft.
Catalog record
| CREATOR | Tomek Korbak [email protected], Mikita Balesni [email protected], Buck Shlegeris [email protected], Geoffrey Irving [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05259
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How to evaluate control measures for LLM agents? A trajectory from today to superintelligence Tomek Korbak [email protected], Mikita Balesni [email protected], Buck Shlegeris [email protected], Geoffrey Irving [email protected] Abstract As LLM agents grow more capable of causing harm autonomously, AI developers will rely on increasingly sophisticated control measures to prevent possibly misaligned agents from causing harm. AI developers could demonstrate that their control measures are sufficient by running control evaluations: testing exercises in which a red team produces agents that try to subvert control measures. To ensure control evaluations accurately capture misalignment risks, the affordances granted to this red team should be adapted to the capability profiles of the agents to be deployed under control measures. In this paper we propose a systematic framework for adapting red teams affordances to advancing AI capabilities. Rather than assuming that agents will always execute the best attack strategies known to humans, we demonstrate how knowledge of an agent’s actual capability profile can inform proportional control evaluations, resulting in more practical and cost-effective control measures. We illustrate our framework by considering a sequence of five fictional models (M1-M5) with progressively advanced capabilities, defining five distinct AI control levels (ACLs). For each ACL, we provide example rules for control evaluation, control measures, and safety cases that could be appropriate. Finally, we show why constructing a compelling AI control safety case for superintelligent LLM agents will require research breakthroughs, highlighting that we might eventually need alternative approaches to mitigating misalignment risk. Figure 1 : We sketch a trajectory of how control evaluations might evolve through capability levels. The trajectory is based on a sequence of five fictional models (M1-M5) with progressively advanced capabilities. For each model, we define an AI Control Level (ACL) based primarily on its threat model-specific capabilities. By mapping these capabilities to appropriate control evaluation methodologies, we provide a systematic approach for adapting safety measures to AI systems as they advance in capabilities. 1 Introduction As large language model (LLM) agents grow more capable of acting autonomously in the world ( Kwa et al., 2025 ) , they may develop goals that diverge from human intent ( Carlsmith, 2023 ; Greenblatt et al., 2024 ) . This misalignment poses significant threats, particularly as agents with misaligned goals might actively conceal their intentions until they find an opportunity to pursue harmful objectives—–a phenomenon known as scheming ( Carlsmith, 2023 ; Meinke et al., 2024 ) . The potential severity of this risk increases with autonomy: current models require significant human oversight to be useful, but future models may provide greater economic value with minimal supervision, creating strong incentives for reduced oversight while simultaneously increasing risk. These tensions call for robust mitigations that can scale as model capabilities advance. While many approaches to AI safety focus on alignment during training, control measures offer a complementary strategy that limits a model’s ability to cause harm even if it harbors misaligned goals ( Greenblatt et al., 2023 ) . This approach, known as AI control , focuses on designing mechanisms to constrain AI systems rather than relying solely on training techniques to shape their behavior. Such measures include monitoring agents’ reasoning processes, requiring human approval for consequential actions, restricting access to sensitive resources, and maintaining the ability to terminate a deployment if suspicious behavior is detected. AI control adopts a conservative stance: it assumes that models might be scheming and designs protections accordingly, rather than assuming models can be made inherently safe through training alone. Several frontier AI developers have expressed interest in this approach, with some incorporating control evaluations into their safety frameworks ( Google DeepMind, 2024 ) or including them in their roadmaps for responsible AI deployment ( Shah et al., 2025 ; Bowman, 2024 ) . Figure 2 : Traditionally, control evaluations have been designed to require the red team to upper bound models in terms of capabilities (red dashed line). We argue that this bound should be tight (violet dashed curve). To assess whether control measures adequately mitigate risks, developers can use control evaluations that frame safety as a game between two teams ( Greenblatt et al., 2023 ; Wen et al., 2024 ) . A red team designs models to cause harm by subverting controls, while a blue team implements monitoring and other protective measures to prevent this. By measuring the red team’s success rate across a representative set of tasks, developers can estimate an upper bound on the probability that deployed models could cause harm. A successful evaluation demonstrates that even adversarially designed models cannot cause unacceptable outcomes when properly controlled, providing empirical evidence for safety claims. The results from these control evaluations serve as crucial inputs to a structured safety argument. Safety cases integrate this evidence with other safety considerations to justify that a particular AI system is safe to deploy in a specific context ( Clymer et al., 2024 ) . These cases are emerging as a scalable risk management tool for frontier AI ( Irving, 2024 ; Hilton et al., 2025 ) , offering transparency about safety claims and their supporting evidence. While many frontier AI developers already assess risk with dangerous capability evaluations ( Anthropic, 2024 ; OpenAI, 2024 ; Phuong et al., 2024 ) , these often focus on what models cannot do rather than on the effectiveness of control measures when models can cause harm. As models gain capabilities that necessitate robust safety measures, AI control safety cases will become increasingly important ( Balesni et al., 2024 ; Korbak et al., 2025 ) . Despite increasing interest in AI control, current research tends to focus on narrow contexts with fixed capability levels or specific deployment scenarios. What is missing is a systematic understanding of how control measures should evolve as capabilities advance—–from today’s models to future systems approaching artificial superintelligence. We need a principled framework that adapts control evaluations to different capability levels and provides safety arguments that remain valid as models evolve. Building on these insights and addressing these gaps, this paper has the following goals. Accounting for capability limitations of models Previous work on AI control focused on control measures intended to be robust to models that execute best attack strategies known to humans. Assuming that models will execute best-known strategies will often be overly conservative: in the near future, models will not be capable of optimal responses to control measures and we can learn this from capability evaluations. We can use our knowledge of model limitations when designing control evaluations and safety cases and we should: it will result in cheaper control evaluations and control measures that are more likely to be adopted (see Figure 2 ). This paper suggests a methodology for translating capability evaluation results into rules of control evaluations. Arguing that AI developers are already doing AI control Most research on AI control proposes evaluations addressing risks from models that do not exist yet. However, many components of control evaluations (e.g. red-teaming) and control measures (e.g. access control, human approval) are used today to address risks models already pose. We argue that AI developers should start thinking about those components in terms of minimum viable products of AI control and start a conversation about a roadmap for scaling them up. Tracing