An Example Safety Case for Safeguards Against Misuse
Joshua Clymer †, Jonah Weinbaum ‡*, Robert Kirk §*, Kimberly Mai §, Selena Zhang ⋄, Xander Davies §, † Redwood Research, ‡ MATS, ⋄ MIT, § UK AI Security Institute · 2025
Read An Example Safety Case for Safeguards Against Misuse by Joshua Clymer †, Jonah Weinbaum ‡*, Robert Kirk §*, Kimberly Mai §, Selena Zhang ⋄, Xander Davies §, † Redwood Research, ‡ MATS, ⋄ MIT, § UK AI Security Institute (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18003.txt.
Introduction
Joshua Clymer and co-authors draft a sample safety case for AI misuse guards in a 2025 paper. Tests of such guards now give a patchwork of proof that is hard to tie to real choices. A safety case is one linked argument that the guards cut risk to low levels. The authors build one for a made-up developer to show the shape.
In their sketch, the developer red-teams the guards to gauge the effort needed to slip past them. That gauge feeds a public model of how much the guards deter misuse. The model gives a live risk signal as the product runs, so the team can react fast to new threats. The paper offers this as one concrete path, not the only one.
Catalog record
| CREATOR | Joshua Clymer †, Jonah Weinbaum ‡*, Robert Kirk §*, Kimberly Mai §, Selena Zhang ⋄, Xander Davies §, † Redwood Research, ‡ MATS, ⋄ MIT, § UK AI Security Institute |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18003
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Example Safety Case for Safeguards Against Misuse Joshua Clymer †, Jonah Weinbaum ‡*, Robert Kirk §*, Kimberly Mai §, Selena Zhang ⋄, Xander Davies §, † Redwood Research, ‡ MATS, ⋄ MIT, § UK AI Security Institute Abstract Existing evaluations of AI misuse safeguards provide a patchwork of evidence that is often difficult to connect to real-world decisions. To bridge this gap, we describe an end-to-end argument (a “safety case”) that misuse safeguards reduce the risk posed by an AI assistant to low levels. We first describe how a hypothetical developer red teams safeguards, estimating the effort required to evade them. Then, the developer plugs this estimate into a quantitative “uplift model” to determine how much barriers introduced by safeguards dissuade misuse ( https://www.aimisusemodel.com/ ). This procedure provides a continuous signal of risk during deployment that helps the developer rapidly respond to emerging threats. Finally, we describe how to tie these components together into a simple safety case. Our work provides one concrete path – though not the only path – to rigorously justifying AI misuse risks are low. Figure 1: An illustration of the safeguard evaluation methodology we base our safety case on. Introduction OpenAI recently reported, “several of our… evaluations indicate our models are on the cusp of… cross[ing] our high risk threshold” [ OpenAI, 2025 ] . Anthropic published a similar statement: “we believe there is a substantial probability that our next model may require ASL-3 safeguards” [ Anthropic, 2025a ] , and more recently activated ASL-3 safeguards for their latest model release [ Anthropic, 2025b ] . As models approach dangerous capability thresholds, some developers plan to shield these capabilities from bad actors with misuse “safeguards,” such as monitors and refusal training [ Anthropic, 2024 , Anthropic, 2025b ] . These safeguards might soon become load bearing for safe deployment [ OpenAI, 2025 , Anthropic, 2025a ] , so there’s an urgent need to rigorously assess them. Researchers have taken large strides towards assessing safeguards by conducting large-scale red-teaming exercises [ Sharma et al., 2025 , Gray Swan Arena, 2025 ] , but the results of these exercises can be difficult to connect to real world decisions. The AI security community currently lacks a systematic way to map misuse safeguard evaluations to actionable risk estimates. Our work fills this gap by explaining an example argument (a safety case) that misuse safeguards ensure risk is below a specified level. We focus on mitigations that developers apply to AI assistant APIs, building on the 5-step process laid out in the Principles for Safeguard Evaluations [ UK AISI, 2025 ] . There are also other approaches to ruling out misuse risks, such as arguments that open-weight models will offset their danger by improving societal resilience ( 3. Discussion ), which we leave out of scope. Executive Summary Safety case objective In our example safety case, the developer aims to defend the following claim ( 2.1 Safety case objectives and definitions ): [C0] : Given safeguards, the AI assistant does not pose risk above a certain level, defined as incurring large-scale harm above some threshold T in expectation. 1 1 1 This safety case is agnostic to the particular threat model and the metric of harm, but possible metrics include the expected number of annual fatalities or financial damage. We’ll first explain the procedures the developer follows to justify that this claim holds, and then we’ll organize this evidence into a safety case. Safeguard evaluation The developer first performs a safeguard evaluation to estimate the barriers safeguards create to misuse actors. The safeguard evaluation involves three steps: • First, threat model experts construct a representative set of harmful requests. • Next, members of a red team try to fulfill these requests by evading safeguards ( section 2.6 ). This results in a set of curves – one for each member of the red team – which relate the amount of time that member spent evading safeguards with the number of harmful requests fulfilled. • Finally, the developer adjusts these curves, factoring in components of safeguards assessed separately such as account banning, and extrapolates them into a smooth trendline ( Figure 2 ). Figure 2: Steps of a safeguards evaluation. Uplift model The safeguard evaluation reveals how much time users would need to spend looking for holes in safeguards. Is the effort required sufficient to dissuade misuse? This question might be difficult to answer with qualitative analysis, so we discuss how a developer could construct a quantitative “uplift model.” You can visit https://www.aimisusemodel.com/ to explore an interactive example. Our uplift model involves three steps ( section 2.5 ): • Step 1. The developer first estimates the existing level of risk , assuming a scenario where the AI assistant is not deployed. This risk can be estimated by considering the time-costs of causing harm through the risk pathway of concern, and the willingness of novice actors to pay these costs. • Step 2. Next, the developer estimates the level of risk if the AI assistant is deployed . – Step 2.1. To arrive at this risk estimate, the developer first assesses the risk posed by a pre-mitigation version of the AI assistant. – Step 2.2. Next, the developer determines how much safeguards lower pre-mitigation risk by drawing on results from the safeguard evaluation . Specifically, the developer calculates how safeguard evaluation results shift points along the pre-mitigation time-cost curve . By combining the new “time-cost curve” with the “willingness to pay” distribution mentioned previously, the developer estimates post-mitigation risk . • Step 3. The output of the uplift model is the difference between the post mitigation risk found in step 2 and the existing level of risk estimated in step 1. Responding to signs of risk Safeguard effectiveness can change post-deployment as the pool of adversaries grows [ UK AISI, 2025 ] . The developer follows three procedural policies that define how they respond to evaluations throughout the AI deployment lifecycle: • Policy 1: The developer first runs pre-deployment evaluations . AI assistants estimated to pose risk above the threshold T are not deployed. • Policy 2: Even after an AI assistant is cleared for deployment, the developer continues to perform safeguard evaluations . The red team (which could include participants in a bug bounty program) keeps searching for holes in safeguards – drawing on black markets, social media, and emerging new jailbreaking methods just as users would. • Policy 3: Before the risk estimated by the uplift model exceeds the threshold T , the developer responds to emerging attacks – hardening faulty safeguards, or in an extreme case, restricting access to the AI assistant. Safety case outline Together, the evaluations and procedural commitments described earlier help keep risk low. However, there are many moving parts to keep track of. To maintain these arguments and evidence throughout deployment, a developer might write a “safety case” – a structured argument that their AI assistant poses risks below some threshold [ Clymer et al. [2024] , step 5 of [ UK AISI, 2025 ] ]. Drawing on the evidence discussed, the following summarizes our safety case ( section 2.7 ): Key: [C]: claim Nested claims support parent claims. Safety case outline ( section