FindTheFlaws: Annotated Errors for Detecting Flawed Reasoning and Scalable Oversight Research

Gabriel Recchia, Chatrik Singh Mangat, Issac Li, Gayatri Krishnakumar · 2025

Read FindTheFlaws: Annotated Errors for Detecting Flawed Reasoning and Scalable Oversight Research by Gabriel Recchia, Chatrik Singh Mangat, Issac Li, Gayatri Krishnakumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22989.txt.

Introduction

Gabriel Recchia, Chatrik Singh Mangat, Issac Li, and Gayatri Krishnakumar introduce FindTheFlaws in a 2025 paper. Their project tackles a growing problem: as AI systems solve harder problems, people struggle to check the answers. Oversight methods such as debate and critique need test data with known errors.

FindTheFlaws bundles five datasets covering medicine, mathematics, science, coding, and the Lojban language. Each set holds long questions and long worked answers. Experts marked each answer as correct or flagged the exact spots where the reasoning goes wrong. The authors then tested leading AI models on spotting the flaws. Scores varied widely by topic, and experts still beat the models in some areas. The data gives researchers a fair yardstick for oversight tools.

Catalog record

CREATORGabriel Recchia, Chatrik Singh Mangat, Issac Li, Gayatri Krishnakumar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE113 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22989

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FindTheFlaws: Annotated Errors for Detecting Flawed Reasoning and Scalable Oversight Research Gabriel Recchia, Chatrik Singh Mangat, Issac Li, Gayatri Krishnakumar Abstract 1 Abstract As AI models tackle increasingly complex problems, ensuring reliable human oversight becomes more challenging due to the difficulty of verifying solutions. Approaches to scaling AI supervision include debate, in which two agents engage in structured dialogue to help a judge evaluate claims; critique, in which models identify potential flaws in proposed solutions; and prover-verifier games, in which a capable ‘prover’ model generates solutions that must be verifiable by a less capable ‘verifier’. Evaluations of the scalability of these and similar approaches to difficult problems benefit from datasets that include (1) long-form expert-verified correct solutions and (2) long-form flawed solutions with annotations highlighting specific errors, but few are available. To address this gap, we present FindTheFlaws, a group of five diverse datasets spanning medicine, mathematics, science, coding, and the Lojban language. Each dataset contains questions and long-form solutions with expert annotations validating their correctness or identifying specific error(s) in the reasoning. We evaluate frontier models’ critiquing capabilities and observe a range of performance that can be leveraged for scalable oversight experiments: models performing more poorly on particular datasets can serve as judges/verifiers for more capable models. Additionally, for some task/dataset combinations, expert baselines exceed even top model performance, making them more beneficial for scalable oversight experiments. 2 Introduction As AI systems become more sophisticated, ensuring reliable human oversight becomes a growing challenge: verifying AI-generated solutions is often difficult, even for domain experts. The limitations of human feedback reduce our ability to trust AI in high-stakes scenarios and raise concerns about robustness, reliability, and alignment ( Amodei et al., 2016 ) . To address this, researchers have proposed protocols to enable ‘scalable oversight’ ( Amodei et al., 2016 ; Bowman et al., 2022 ; Irving et al., 2018 ; Christiano et al., 2018 ; Kenton et al., 2024 ) : methods that could theoretically allow humans to effectively evaluate AI systems’ outputs as AI capabilities increase, even as the task of verifying these outputs becomes too difficult or costly for human experts to accomplish directly. Most scalable oversight protocols ultimately aim to help a judge (a human or trusted model, such as an AI judge trained to predict human judgments ( Irving, 2019 ) ) to identify errors in solutions produced by potentially misaligned or deceptive models. However, accurate labels for errors in long-form reasoning are expensive and time-consuming to produce. As a result, existing work has largely focused on whether oversight protocols can help judges arrive at correct binary decisions about solution validity ( Bowman et al., 2022 ; Irving et al., 2018 ; Kenton et al., 2024 ; Khan et al., 2024 ; Kirchner et al., 2024 ) , with far less attention paid to the key question of whether judges actually identify the true underlying problems when they reject flawed solutions (but see Lightman et al. (2023) ; Uesato et al. (2022) ). While achieving correct judgments might seem sufficient, we cannot be as confident that these approaches will generalize without understanding the specific nature of the (supposed) errors identified—what it is about the argument that causes it to fail, whether this takes the form of a natural language explanation or an indication of which step of the argument is flawed. In the following paragraphs, we highlight how studies on the efficacy of three families of scalable oversight protocols—critique, debate, and prover-verifier games—could benefit from access to this kind of ground-truth error information. We then present our FindTheFlaws datasets as a resource for conducting such analyses. Critique models. Critique models are trained to write natural language assessments that identify potential flaws or limitations in AI-generated outputs, aiming to help human or AI evaluators more effectively assess complex solutions. In pioneering work, Saunders et al. (2022) conducted rigorous experiments on topic-based summarization and synthetic tasks (like arithmetic and boolean satisfaction problems) where ground truth was clearly defined. Their models wrote natural language critiques of proposed solutions, identifying issues like inaccuracies and important missing information. These critiques effectively assisted human evaluators—showing evaluators model-generated critiques alongside summaries helped them identify about 50% more flaws than they found unassisted. Sun et al. (2024) enumerate several subsequent works that make use of LLM-generated critiques to improve LLM outputs, and propose methods for automated evaluation of critique quality which yield scores that are more correlated with human judgments than direct quality rating by GPT-4. While these results are promising, the tasks studied were relatively simple compared to more complex problems where evaluation requires deep domain expertise, where flaws require sophisticated reasoning to identify, and where the space of possible errors is larger and more open-ended. In order to scale critique-based oversight to such domains, we need datasets containing not just examples of correct solutions, but also examples of flawed solutions where both the presence and specific nature of the errors are known and validated by domain experts. This would extend the approach of using synthetic tasks with hand-coded oracles to more sophisticated settings, allowing researchers to directly measure whether models’ critiques align with actual expert-identified flaws in complex reasoning. Debate. Debate is an approach in which two AI agents engage in a structured back-and-forth dialogue to help a less capable judge evaluate claims and determine truth ( Irving et al., 2018 ) . The key hypothesis is that in many domains, it may be easier to identify flaws in incorrect statements than to directly verify correct ones. The debate protocol aims to exploit this asymmetry: if making false claims is reliably harder than refuting them, then honest debaters should have an advantage over dishonest ones, and optimal play may even converge to honest behavior, with agents racing to point out flaws in their opponent’s reasoning rather than constructing elaborate deceptions. This could enable meaningful oversight of superhuman AI systems by allowing human judges to leverage the competitive dynamic between debating agents to surface important considerations and verify claims, rather than having to directly evaluate every aspect of complex solutions. Early empirical results from debate experiments have demonstrated that agents trained in debate settings can achieve higher accuracy in identifying truthful claims compared to settings where they view output from a single agent randomly assigned to advocate for a correct or incorrect answer ( Kenton et al., 2024 ; Khan et al., 2024 ; Michael et al., 2023 ) , and that judgment accuracy scales with the capabilities of the debate models (keeping judge skill constant) ( Kenton et al., 2024 ; Khan et al., 2024 ) . However, this trend may break down as the difficulty of the problems that debaters are asked to consider increases. Another issue is that in some cases, agents may rely more on heuristic reasoning and may struggle to fully articulate their decision-making process. Consider a debate agent provided an incomplete document on a complex topic, which is tasked with convincing a judge of an answer to the question "What is the most likely word to come next?" It might assess the most likely next word to be w1 , but it might not be able to articulate all of the statistical patterns, contextual cues, and interactions

Read the complete text