Human-AI Complementarity: A Goal for Amplified Oversight
Rishub Jain, Sophie Bridgers, Lili Janzer, Rory Greig, Tian Huey Teh, Vladimir Mikulik · 2025
Read Human-AI Complementarity: A Goal for Amplified Oversight by Rishub Jain, Sophie Bridgers, Lili Janzer, Rory Greig, Tian Huey Teh, Vladimir Mikulik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26518.txt.
Introduction
Rishub Jain, with five co-authors, studies how people and AI can check each other's work. Human feedback steers AI toward human values, but as AI takes on harder tasks, judging its output gets harder for people alone. The team focuses on fact-checking AI answers, a safety task that already strains human reviewers.
Jain and the team find that blending AI ratings with human ratings, weighted by the AI's confidence, beats either one alone. Giving people an AI fact-checking assistant raised their accuracy too, but the form of help mattered. Showing the AI's explanations, confidence, and labels led people to lean on it too much. Just showing search results and evidence led to better-placed trust. The work speaks to amplified oversight, the task of supervising AI as it grows past human skill.
Catalog record
| CREATOR | Rishub Jain, Sophie Bridgers, Lili Janzer, Rory Greig, Tian Huey Teh, Vladimir Mikulik |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26518
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Human-AI Complementarity: A Goal for Amplified Oversight Rishub Jain, Sophie Bridgers, Lili Janzer, Rory Greig, Tian Huey Teh, Vladimir Mikulik Abstract Human feedback is critical for aligning AI systems to human values. As AI capabilities improve and AI is used to tackle more challenging tasks, verifying quality and safety becomes increasingly challenging. This paper explores how we can leverage AI to improve the quality of human oversight. We focus on an important safety problem that is already challenging for humans: fact-verification of AI outputs. We find that combining AI ratings and human ratings based on AI rater confidence is better than relying on either alone. Giving humans an AI fact-verification assistant further improves their accuracy, but the type of assistance matters. Displaying AI explanation, confidence, and labels leads to over-reliance, but just showing search results and evidence fosters more appropriate trust. These results have implications for Amplified Oversight—the challenge of combining humans and AI to supervise AI systems even as they surpass human expert performance. \reportnumber keywords HCI, Human-AI Complementarity, Amplified Oversight, Scalable Oversight, LLM Rating, AI Assistance, Hybridization 1 Introduction Human oversight is critical for ensuring that Artificial Intelligence (AI) models remain safe and aligned to human values. But AI systems are rapidly advancing in capabilities and are being used to complete ever more complex tasks, making it increasingly challenging for humans to verify AI outputs and provide high-quality feedback. How can we ensure that humans can continue to meaningfully evaluate AI performance? An avenue of research to tackle this problem is “ Amplified Oversight ” (also called “Scalable Oversight”), which aims to develop techniques to use AI to amplify humans’ abilities to oversee increasingly powerful AI systems, even if they eventually surpass human capabilities in particular domains. With this level of advanced AI, we could use AI itself to evaluate other AIs (i.e., AI raters), but this comes with drawbacks (see Section 3.1 ). Importantly, humans and AIs have complementary strengths and weaknesses. We should thus, in principle, be able to leverage these complementary abilities to generate an oversight signal for model training, evaluation, and monitoring that is stronger than what we could get from human raters or AI raters alone. Two promising mechanisms for harnessing human-AI complementarity to improve oversight are: • Hybridization , in which we combine judgments from human raters and AI raters working in isolation based on predictions about their relative rating ability per task instance (e.g., based on confidence), and • Rater Assistance , in which we give human raters access to an AI rating assistant that can critique or point out flaws in an AI output or automate parts of the rating task. In this paper, we empirically investigate how combining Hybridization and Rater Assistance can produce a higher quality oversight signal than relying on either human or AI raters alone. We ground our study in the task of verifying the factual accuracy of AI-generated sentences. We chose this task for two reasons: (1) It is a realistic, pressing safety concern of currently deployed models, which often hallucinate and generate misleading information and (2) It is already a challenging task for human raters. To do so, we developed an AI fact-verification rater that uses a search engine tool to research factuality. This AI rater has a higher overall accuracy than a typical human rater on our realistic evaluation dataset. But critically, when the AI’s confidence is low, it performs worse than humans. This allows us to perform what we call Confidence-based Hybridization : use the AI’s ratings when it has high confidence, and the humans’ ratings when the AI has lower confidence. The AI rater output can also be served to humans as a fact-verification assistant. Using Rater Assistance to achieve complementarity in settings where AI raters already outperform human raters is challenging. A recent review shows that when AI alone performs better, it is harder for humans to add value ( Vaccaro et al. 2024 ) . To still realize the gains of Rater Assistance, we can use Confidence-based Hybridization to identify examples where Humans still outperform AI, thus making it easier for Rater Assistance to achieve complementarity. This leads to our primary research questions: 1. Does confidence-based hybridization improve accuracy above AI or human ratings alone? 2. On the low-AI-confidence data slice assigned to humans, can AI assistance improve human accuracy, even though the assistant is less accurate than the humans on this slice? If so, what form of AI assistance is most effective at improving accuracy while appropriately calibrating human reliance? 1.1 Related Work 1.1.1 Amplified Oversight AI models are rapidly increasing in capabilities. As models improve, we want to ensure that they continue to align with human goals and values. A prominent approach for value alignment of LLMs is fine-tuning based on human feedback, using techniques like Reinforcement Learning from Human Feedback ( Christiano et al. 2017 ) and Direct Preference Optimization ( Rafailov et al. 2024 ) . These techniques rely on humans being able to accurately judge the quality of an AI’s responses, considering several dimensions (e.g. helpfulness, factuality, safety). The increasing capabilities of AI models, however, present a challenge for human oversight: AI systems will be asked to complete increasingly challenging tasks and their capabilities on these tasks may exceed human capabilities making it difficult or highly time consuming for humans to provide feedback. If human evaluation becomes more unreliable, we risk rewarding behavior that looks superficially "good" but isn’t actually desirable ( Krakovna et al. 2020 ) , such as misleading information and sycophancy ( Sharma et al. 2023 ) . This raises a critical question: how can we improve human oversight to keep pace with AI advancements? The field of Amplified Oversight aims to address this issue by leveraging the increasingly advanced capabilities of AI to assist humans in providing high-quality feedback ( Amodei et al. 2016 ) . Current research explores different approaches to AI assistance for human evaluation, such as AI critics that point out errors in a model’s output and AI debates that provide different perspectives, with promising but mixed results ( Irving et al. 2018 ; Saunders et al. 2022 ; McAleese et al. 2024 ; Parrish et al. 2022b ; Michael et al. 2023 , e.g.,) . For example, there is positive evidence of debate helping when there is knowledge asymmetry between the AI assistance and the "judge" rater, but when this asymmetry is removed, the judge receives little benefit ( Michael et al. 2023 ; Parrish et al. 2022b ; Parrish et al. 2022a ; Khan et al. 2024 ; Kenton et al. 2024 ) . Thus far the field has primarily focused on machine learning solutions, but figuring out how to use AI to elicit better reasoning from a human rater is fundamentally a Human Computer Interaction (HCI) problem. Amplified Oversight would benefit from closer collaboration with the HCI community to effectively leverage the complementary strengths of humans and AI in achieving robust and scalable oversight. In this paper, taking an HCI perspective, we explore this collaborative human-AI approach for improving human feedback on the factuality of model outputs. While existing Amplified Oversight and HCI research on AI assistance provide valuable insights, Amplified Oversight has not