Scalable Oversight for Superhuman AI via Recursive Self-Critiquing
Xueru Wen, Jie Lou, Xinyu Lu, Junjie Yang, Yanjiang Liu, Yaojie Lu, Debing Zhang, XingYu · 2025
Read Scalable Oversight for Superhuman AI via Recursive Self-Critiquing by Xueru Wen, Jie Lou, Xinyu Lu, Junjie Yang, Yanjiang Liu, Yaojie Lu, Debing Zhang, XingYu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04675.txt.
Introduction
Wen and seven co-authors present a paper from twenty twenty-five on overseeing superhuman AI through recursive self-critique. Human review fails when AI output outstrips human judgment. The team test two linked guesses about critique. First, judging a critique is easier than critiquing itself. Second, that gap repeats at each higher level of critique. Experiments mix human-human, human-AI, and AI-AI settings. Results support stacked critique as a workable supervision path.
Critiques of critiques give traction where direct grading stalls. Oversight could scale with AI skill under this scheme. The authors frame recursion as the key move. Alignment practice gains a candidate tool.
Catalog record
| CREATOR | Xueru Wen, Jie Lou, Xinyu Lu, Junjie Yang, Yanjiang Liu, Yaojie Lu, Debing Zhang, XingYu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-04675
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scalable Oversight for Superhuman AI via Recursive Self-Critiquing Xueru Wen, Jie Lou, Xinyu Lu, Junjie Yang, Yanjiang Liu, Yaojie Lu, Debing Zhang, XingYu Abstract As AI capabilities increasingly surpass human proficiency in complex tasks, current alignment techniques, including SFT and RLHF, face fundamental challenges in ensuring reliable oversight. These methods rely on direct human assessment and become impractical when AI outputs exceed human cognitive thresholds. In response to this challenge, we explore two hypotheses: (1) Critique of critique can be easier than critique itself , extending the widely-accepted observation that verification is easier than generation to the critique domain, as critique itself is a specialized form of generation; (2) This difficulty relationship holds recursively , suggesting that when direct evaluation is infeasible, performing higher-order critiques (e.g., critique of critique of critique) offers a more tractable supervision pathway. We conduct Human-Human, Human-AI, and AI-AI experiments to investigate the potential of recursive self-critiquing for AI supervision. Our results highlight recursive critique as a promising approach for scalable AI oversight. † † affiliationnotice: Equal contribution 1 Introduction Supervision signals are fundamental to AI alignment ( Bowman et al., 2022 ) , providing the ground truth or preference data necessary to train models that behave in accordance with human expectations. The nature and accessibility of these supervision signals, however, vary substantially across different application domains. From a supervision acquisition perspective, tasks can be categorized into two types: (1) tasks with well-defined criteria, where ground truth can be deterministically obtained with low computational overhead, e.g., Go games and mathematical problems ( Silver et al., 2017 ; Lightman et al., 2023 ) ; (2) tasks involving subjectivity or complex evaluation frameworks, such as business strategy and product design ( Ouyang et al., 2022 ) . The latter type is more prevalent in real-world applications and predominantly relies on human assessment, presenting a fundamental challenge. Current alignment techniques, particularly Supervised Fine-tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), have achieved empirical success with large language models ( Meta, 2024 ; Yang et al., 2024 ; DeepSeek-AI, 2024 ) . SFT ( Chung et al., 2022 ; Wei et al., 2022 ) finetunes models with human-annotated demonstrations, showing particular efficacy in tasks where humans can effectively showcase desired behaviors. RLHF ( Christiano et al., 2023 ; Ouyang et al., 2022 ) employs reinforcement learning with human preference reward models based on pairwise comparisons, extending supervision to more complex tasks where direct solution generation is challenging. However, both approaches rely on direct human feedback, making them unsustainable for tasks where human evaluation becomes infeasible. For example, humans can struggle with time-consuming tasks such as reviewing extensive long-form text ( Stiennon et al., 2022 ) r expertise-intensive tasks such as verifying solutions to complex mathematical problems ( Li et al., 2024b ) . Furthermore, as AI capabilities advance beyond human abilities, obtaining reliable supervision signals becomes increasingly challenging, representing the central problem of scalable oversight ( Casper et al., 2023 ; Ji et al., 2024 ; Kenton et al., 2024b ) . The underlying insight of RLHF is that verification is easier than generation ( Leike et al., 2018 ; Irving et al., 2018b ) . By recognizing critique as a specialized form of generation, we further hypothesize that critique of critique is easier than critique itself . Taking a complex mathematical proof as an example: while direct review can be challenging, assessing its critique is more manageable, as the key steps have already been identified. Moreover, we hypothesize that this difficulty relationship generalizes recursively , where each successive level of meta-evaluation becomes increasingly tractable. This resembles organizational decision-making processes, where managers evaluate their subordinates’ assessments rather than directly reviewing complex details. These hypotheses, if validated, offer a promising pathway for scalable oversight: while directly evaluating sophisticated AI output may exceed human capabilities, performing higher-order critiques could remain feasible. To systematically verify these hypotheses, we first conduct Human-Human experiments where humans evaluate human outputs. We examine the progression from response to critique and then to critique-of-critique ( C^{2} ). By comparing accuracy under similar computational effort, completion time, and confidence levels, we find that higher-order critiques contribute to more effective evaluation than direct assessment. Furthermore, we demonstrate the recursive nature of this relationship by extending experiments to deeper critique chains, i.e., critique of critique of critique ( C^{3} ). Inspired by these human-human findings, we further investigate their applicability for supervising AI: when AI generates self-recursive critiques, can humans provide effective oversight by evaluating these critique chains? To answer this question, we conduct Human-AI experiments, where humans evaluate AI outputs on tasks where AI outperforms average humans. The results are promising across models of varying capabilities. Finally, we examine whether AI can achieve effective oversight through recursive self-critiques in AI-AI experiments across models of different capabilities. Our results demonstrate that recursive self-critiquing is effective in weak-to-strong scenarios, while the optimal critique strategy depends on the relative capabilities between supervised and critic models. In general, our contributions can be summarized as follows: 1. We investigate and validate the hypothesis that critique of critique is easier than critique , extending the principle that verification is easier than generation. 2. We demonstrate that above difficulty relationship can hold recursively , showing how complex evaluation tasks can be simplified by recursive meta evaluations. 3. Through comprehensive Human-Human, Human-AI, and AI-AI experiments, we demonstrate the potential of recursive self-critiquing as a scalable oversight method, providing new valuable insights for supervising advanced AI systems beyond human capabilities. 2 Recursive Self-Critiquing In this section, we introduce the protocols for recursive self-critiquing across multiple evaluation levels, spanning initial response through higher-order critiques. We then present majority voting and naive voting as two representative baselines to provide fair comparisons for evaluating the effectiveness of recursive critique. Figure 1 : Overview of the recursive critique framework. Starting from response generation for a given question, each subsequent level performs pair-wise evaluation of outputs from the previous level, forming a recursive critique chain. C^{1} denotes Critique, C^{2} denotes Critique of Critique, C^{3} denotes Critique of Critique of Critique. 2.1 Protocols As shown in Figure 1 , the hierarchical criticism architecture progresses through multiple levels: from initial response, through first-order critique, to second-order critique of critique ( C^{2} ) and higher-order critiques. Our protocols follow standard RLHF practice ( Ouyang et al., 2022 ) , employing pairwise comparisons at each critique level. This approach leverages humans’ cognitive advantage in relative assessment over absolute evaluation ( Jones and Inglis, 2015 ; Kelly et al., 2022 ) , making recursive evaluation more tractable at each level. Moreover, this design facilitates consistency between human and AI experiments, as the latter requires pairwise preference data for reward model training. Response Response is the initial attempt to answer the question, serving as the foundation of the critique chain. Each response comprises a complete solution process and its corresponding answer: R(Q)\rightarrow(T^{0},A^{0}) (1) where Q denotes the input question, T^{0} represents the solution process which may include reasoning steps, justifications, and intermediate calculations, and A^{0} is the final answer. Including the full solution process rather than merely the final answer enables critiques to better assess the correctness of each response by examining logical consistency, key step validity, and other