Automated Alignment is Harder Than You Think

Aleksandr Bowkis, Marie Davidsen Buhl, Jacob Pfau, Geoffrey Irving · 2026

Read Automated Alignment is Harder Than You Think by Aleksandr Bowkis, Marie Davidsen Buhl, Jacob Pfau, Geoffrey Irving (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06390.txt.

Introduction

Aleksandr Bowkis and three co-authors argue that automating alignment research is harder and riskier than its proponents assume. A leading plan for aligning superintelligence uses AI agents to do a growing share of alignment research itself. Even with no deliberate sabotage, the authors warn, that plan could yield compelling but badly misleading safety assessments. It could end in the unintentional deployment of a misaligned system that everyone believed had been checked thoroughly and passed.

Alignment work is full of fuzzy tasks with no clear scoring, where human judgment is systematically flawed, so agent outputs would carry undetected systematic errors. Optimization pressure concentrates agent mistakes exactly where reviewers are least likely to catch them. Agent errors also differ from human ones, and shared training makes agent outputs correlate with each other. Scalable oversight and generalization, the leading remedies, face novel challenges in this setting.

Catalog record

CREATORAleksandr Bowkis, Marie Davidsen Buhl, Jacob Pfau, Geoffrey Irving
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06390

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automated Alignment is Harder Than You Think Aleksandr Bowkis, Marie Davidsen Buhl, Jacob Pfau, Geoffrey Irving Abstract A leading proposal for aligning artificial superintelligence (ASI) is to use AI agents to automate an increasing fraction of alignment research as capabilities improve. We argue that, even when research agents are not scheming to deliberately sabotage alignment work, this plan could produce compelling but catastrophically misleading safety assessments resulting in the unintentional deployment of misaligned AI. This could happen because alignment research involves many hard-to-supervise fuzzy tasks (tasks without clear evaluation criteria, for which human judgement is systematically flawed). Consequently, research outputs will contain systematic, undetected errors, and even correct research could be incorrectly aggregated into overconfident safety assessments. This problem is likely to be worse for automated alignment research than for human-generated alignment research for several reasons: 1) optimisation pressure means agent-generated mistakes are concentrated among those that human reviewers are least likely to catch; 2) agents are likely to produce errors that do not resemble human mistakes; 3) AI-generated alignment solutions may involve arguments humans cannot evaluate; and 4) shared weights, data and training processes may make AI outputs more correlated than human equivalents. Therefore, agents must be trained to reliably perform hard-to-supervise fuzzy tasks. Generalisation and scalable oversight are the leading candidates for achieving this but both face novel challenges in the context of automated alignment. 1 Introduction Alignment of artificial superintelligence (ASI) is an important, unsolved research problem 1 1 1 The definition of what constitutes ASI, and the degree to which misaligned ASI poses a significant risk, is contested. . AI research agents may help solve this problem, for example via the following plan: • Build AI agents that can do empirical alignment work (e.g. writing code, running experiments, designing evaluations and red teaming) and confirm they are not scheming 2 2 2 We consider an agent to be scheming if it covertly and strategically pursues a hidden objective ( 30 ) . This definition permits isolated misaligned behaviours such as reward hacking. Unlike other common definitions (e.g., 7 ), our definition is not restricted to behaving well in training in order to gain power later. . • Use these agents to build increasingly sophisticated empirical safety cases for each successive generation of agents, gradually automating more of the research process. • Hand over primary research responsibility once agents outperform humans at all relevant alignment tasks. This paper argues that automating alignment research in this manner could produce catastrophically misleading safety cases, causing researchers to believe that an egregiously misaligned AI is safe, even if AI agents are not scheming to deliberately sabotage alignment research. Our core argument (Fig. 1 ) is as follows: 1. The goal of an automated alignment program is to produce an overall safety assessment (OSA) 3 3 3 ”Overall safety assessment” is intended as a more general term than ”safety case”; a safety case is an OSA, but an OSA could also look like a risk report, a compilation of empirical evidence, or a judgement by an individual human or AI). - an estimate of the probability that the next-generation agent is non-scheming 4 4 4 If the next-generation agent will also be used to do automated alignment research, the OSA must additionally estimate the probability that its research outputs can be assessed well enough to produce a calibrated OSA for the next-next-generation. - that is both calibrated and shows low risk . 2. Producing an OSA involves several tasks that are difficult to check. We refer to these as hard-to-supervise fuzzy tasks: tasks without clear evaluation criteria, for which human judgement is systematically flawed . Two particularly important such tasks are 5 5 5 Another important, hard-to-supervise fuzzy task is steering a body of research and allocating resources efficiently between research directions. This involves assessing whether individual research directions are worth pursuing and whether the research corpus is sufficiently broad. : 1. Measuring what you care about: The alignment properties of a sufficiently advanced AI are not directly measurable. We cannot safely deploy the AI in situations where it could cause serious harm and observe whether it takes misaligned actions. Instead, research must target alignment proxies (e.g. honesty evaluations, model organism experiments, whitebox probes) and then make a judgement about how relevant the results are for alignment of the AI. 2. Aggregating correlated evidence: OSAs will likely rely on multiple pieces of evidence. If each piece of evidence is independent, aggregating them into an OSA would be straightforward. In practice, however, there will be uncertainty about the conclusions from each research output and these uncertainties will be correlated. This happens because research outputs share assumptions and methodologies; because research agents share weights, training processes and data; and because human researchers share biases. If these correlations are mis-modelled, the resulting OSA could be catastrophically flawed even if no individual piece of evidence is wrong. Because these tasks are hard to supervise, research outputs (including OSAs) will contain systematic, undetected errors , regardless of whether these tasks are completed by human researchers or research agents trained using human data. We already see examples of such errors in current models, especially on hard tasks. 3. Of course, such errors are also present in human-generated alignment research. However, errors in automated alignment research are likely to be harder to identify than the human baseline . There are five reasons for this: 1. Optimisation pressure: AI research is optimised for human approval to a greater extent than human research, and so any errors are concentrated among those that human reviewers are least likely to catch. 2. Alien mistakes: Agents may introduce errors that do not resemble human mistakes. Reviewers may struggle to identify these, even if they are in principle easy to correct. 3. More correlated research: Shared weights, training processes and data sources for agents introduce additional correlation structure relative to human-generated research. 4. Research volume: OSAs based on AI research may reference much larger evidence bases, increasing the potential for complex and hard-to-spot correlations between research outputs. 5. Non-human-evaluable arguments: Alignment solutions for ASI may rely on arguments that humans are unable to follow, making errors impossible to identify. In most domains, iteration is able to correct for undetected errors. Mistakes not caught during one experiment are often surfaced by subsequent research or real-world system behaviour. Unfortunately, alignment lacks the safe feedback loops that are required for such an error-correction process to work: producing an overly optimistic OSA could result in the deployment of a misaligned AI before the error is caught, which could be catastrophic. Therefore, we need to train agents to reliably perform well on these hard-to-supervise fuzzy tasks the first time. Direct training using human feedback on these hard-to-supervise fuzzy tasks is not sufficient as human approval does not indicate correctness. Two alternatives remain: Generalisation: Train agents on easier-to-supervise training proxies and rely on generalisation to the hard-to-supervise fuzzy research tasks we care about. This requires us to predict generalisation behaviour, since performance on the hard-to-supervise fuzzy task cannot be evaluated directly. Scalable oversight: Improve the reward signal for hard-to-supervise fuzzy tasks, for example by decomposing them into easier-to-supervise subtasks. Existing protocols (e.g. recursive reward modelling or debate) may not work,

Read the complete text