An alignment safety case sketch based on debate
Marie Davidsen Buhl [email protected], Jacob Pfau [email protected], Benjamin Hilton [email protected], Geoffrey Irving [email protected] · 2025
Read An alignment safety case sketch based on debate by Marie Davidsen Buhl [email protected], Jacob Pfau [email protected], Benjamin Hilton [email protected], Geoffrey Irving [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03989.txt.
Introduction
Marie Davidsen Buhl and co-authors sketch in 2025 how debate between AI systems could form the base of a safety case for a strong AI. As AI nears or passes human skill at many tasks, people may not be able to judge its work well. Feedback from people would then be a weak steer, which puts safe use at risk.
In a debate, a second strong system hunts for flaws in the first system's work and argues its case. A person can judge the duel more easily than the raw work. The paper sets out what debate would need to work and what proof is still missing. Its test case is an AI research agent inside a firm that might fake results. The sketch is a plan for an argument, not proof of safety, and the authors mark the gaps that research must fill.
Catalog record
| CREATOR | Marie Davidsen Buhl [email protected], Jacob Pfau [email protected], Benjamin Hilton [email protected], Geoffrey Irving [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03989
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An alignment safety case sketch based on debate Marie Davidsen Buhl [email protected], Jacob Pfau [email protected], Benjamin Hilton [email protected], Geoffrey Irving [email protected] Abstract If AI systems match or exceed human capabilities on a wide range of tasks, it may become difficult for humans to efficiently judge their actions – making it hard to use human feedback to steer them towards desirable traits. One proposed solution is to leverage another superhuman system to point out flaws in the system’s outputs via a debate. This paper outlines the value of debate for AI safety, as well as the assumptions and further research required to make debate work. It does so by sketching an “alignment safety case” – an argument that an AI system will not autonomously take actions which could lead to egregious harm, despite being able to do so. The sketch focuses on the risk of an AI R&D agent inside an AI company sabotaging research, for example by producing false results. To prevent this, the agent is trained via debate, subject to exploration guarantees, to teach the system to be honest. Honesty is maintained throughout deployment via online training. The safety case rests on four key claims: (1) the agent has become good at the debate game, (2) good performance in the debate game implies that the system is mostly honest, (3) the system will not become significantly less honest during deployment, and (4) the deployment context is tolerant of some errors. We identify open research problems that, if solved, could render this a compelling argument that an AI system is safe. 1 Introduction As AI capabilities scale - especially if AI systems match or exceed human performance on a wide range of tasks - it could become challenging to steer them towards desirable traits. One particular problem is scalable oversight: how can we correctly reward desired behaviours for superhuman systems? In particular, it may be beyond humans’ ability to efficiently judge whether the behaviour of a superhuman system is desirable (or judge if a justification provided by the superhuman system is valid). One proposed solution to scalable oversight is to leverage another superhuman AI to scrutinise the system’s behaviour via a debate ( Irving et al. 2018 ) . During training, the two systems would debate the answer to a question, the debate would be judged by a human, and the systems would be rewarded accordingly. If the debate set-up incentivises desirable traits such as honesty, the AI systems could acquire those traits via reinforcement learning (RL). This paper sets out what debate can be used to achieve, what gaps remain, and what research is needed to solve them. The paper uses the technique of safety case sketching ( Hilton et al. 2025 ) : outlining an argument a developer could make that a given AI system is safe in a given context. The details in the sketch are hypothetical, often relying on future research. Nonetheless, the sketch helps contextualise how different pieces of evidence fit together and identify the most crucial pieces of missing evidence. The core claim in the sketch is that debate training can upper bound the proportion of model outputs that are misaligned, as long as the model does not deliberately underperform during training. 1 1 1 That is, debate training gets you a deployment-time error bound that allows you to prove regret bounds for online training – given the assumptions stated below. This upper bound can be maintained during deployment if a system is trained continuously. So, in low-stakes settings, where a small number of mistakes are acceptable such that you can use online training ( Christiano 2021 ) , debate training might be sufficient - especially if the error bound is tight. 2 2 2 It is debatable if this safety case should be considered an example of control, alignment, or a combination. Our argument does guarantee robustness to distribution shifts or internalisation of certain values, and it admits some proportion of misaligned outputs. On the other hand, the argument relies primarily on training-level interventions. Further, sufficiently strong exploration guarantees (such as methods for eliciting bad contexts) might enable robustness guarantees, as discussed in ”Extending the safety case to high-stakes contexts”. The sketch focuses on a slightly superhuman system in a limited internal deployment context designed for automated AI research (including alignment research). The sketch argues that this is a low-stakes context, assuming sufficient security in the environment implementation. The paper also contains ideas about pushing low-stakes solutions through to high-stakes contexts (see “Extending the safety case to high-stakes contexts”). Beyond the low-stakes assumption, the paper identifies other assumptions that are required for the safety case to work. The key assumptions are: • Exploration guarantees: We need a solution to exploration hacking (in which the debaters fail to explore certain highly rewarded strategies to avoid having them reinforced). For example, if we can elicit “bad” contexts ( Irving et al. 2025 ) in which models would answer differently under different circumstances, we can use this to detect exploration hacking (and possibly prevent it during training). Notably, with a (mostly) trusted reward signal, we do not need to explicitly look for “scheming” (which is hard to conceptualise and formalise) – we only need an “exploration guarantee” showing that the model is not foregoing substantial reward and therefore likely optimising against the reward signal. • No obfuscated arguments: We need a debate protocol that avoids the obfuscated arguments problem (in which debaters can win by giving a false but hard-to-refute argument, specifically one where it is computationally intractable for both debaters to identify the flaw) ( Barnes 2020 ) . Our sketch is based on a new debate protocol (forthcoming) which we believe evades the obfuscated arguments problem , 3 3 3 However, we think that the argument structure is general enough to carry over to improved scalable oversight protocols with better properties. which would be substantial progress in debate research. However, the protocol requires a stability assumption -- that the computation accelerated by debate is not sensitive to small variations in input probabilities -- to function. 4 4 4 The stability assumption is not directly required for safety – for queries where stability does not hold, the protocol will simply return no answer, so instability does not imply false outputs. However, the protocol is not very useful if stability fails to hold for a large fraction of relevant queries. • Good human input: We need to assume that (simulated) human inputs sufficiently reliably reward correct answers, over the relevant distribution of human-judged queries. The safety case sketch contains four key claims that a developer must defend to show that the system is safe: 1. The training process has reached an approximate global equilibrium of the game. 2. In approximate global equilibria of the game, the system makes mistakes in at most an \varepsilon^{\prime} -fraction of cases. 3. During deployment, the error rate will not drift past \varepsilon given online training. 4. The system cannot cause unacceptable outcomes given an error rate of \varepsilon . We call type of alignment argument an “asymptotic guarantee” (also known as an “incentives” argument ( Shah et al. 2025 ) ): an argument that high performance during training implies (approximate)