ReFACT: A Benchmark for Scientific Confabulation Detection with Positional Error Annotations

Yindong Wang, Martin Preiß, Margarita Bugueño, Jan Vincent Hoffbauer, Abdullatif Ghajar, Tolga Buz, Gerard de Melo · 2025

Read ReFACT: A Benchmark for Scientific Confabulation Detection with Positional Error Annotations by Yindong Wang, Martin Preiß, Margarita Bugueño, Jan Vincent Hoffbauer, Abdullatif Ghajar, Tolga Buz, Gerard de Melo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25868.txt.

Introduction

Yindong Wang and six co-authors present ReFACT in this 2025 paper, a benchmark for catching made up science in language model answers. The name blends Reddit False And Correct Texts. It holds 1,001 question and answer pairs drawn from a science forum on Reddit. Experts marked the exact spans where errors sit. The team used the set to test nine top language models.

One failure mode dominated: most wrong span picks pointed at text that had nothing to do with the real errors. That pattern held from small models to the largest, hinting that scale alone will not fix it. A second surprise was that judging two answers side by side proved harder than checking one alone. Even GPT-4o scored lower in the paired mode. The authors warn that using models as judges of science claims is shakier than often assumed.

Catalog record

CREATORYindong Wang, Martin Preiß, Margarita Bugueño, Jan Vincent Hoffbauer, Abdullatif Ghajar, Tolga Buz, Gerard de Melo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25868

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ReFACT: A Benchmark for Scientific Confabulation Detection with Positional Error Annotations Yindong Wang, Martin Preiß, Margarita Bugueño, Jan Vincent Hoffbauer, Abdullatif Ghajar, Tolga Buz, Gerard de Melo Abstract Large Language Models (LLMs) frequently confabulate scientific facts,severely undermining their trustworthiness. Addressing this challenge requires benchmarks that go beyond binary factuality and enable fine-grained evaluation. We introduce ReFACT ( Reddit False And Correct Texts ), a benchmark of 1,001 expert-annotated question–answer pairs spanning diverse scientific domains for the detection of scientific confabulation. Each instance includes both a scientifically correct answer and a non-factual counterpart annotated with precise error spans and error-types . ReFACT enables multi-stage evaluation: (1) confabulation detection, (2) fine-grained error localization, and (3) correction. We benchmark 9 state-of-the-art LLMs, revealing limited performance ( ∼ \sim 50% accuracy). Even top models such as GPT-4o fail to distinguish factual from confabulated scientific answers, raising concerns about the reliability of LLM-as-judge evaluation paradigms. Our findings highlight the need for fine-grained, human-validated benchmarks to detect and correct scientific confabulation in domain-specific contexts. Dataset is released on GitHub 1 1 1 We provide the dataset at: https://github.com/ddz5431/ReFACT . Figure 1: Overview of the ReFACT Evaluation Pipeline (Entity Replacement Example). Given a factual Reddit answer and a minimally transformed counterpart containing a subtle instance of scientific confabulation (“your DNA” \rightarrow “your RNA”), the model is evaluated on: (1) Judgment – detecting confabulation, (2) Span Localization – identifying the corrupted entity span, and (3) Correction – recovering the original entity span. The figure illustrates a representative case from the entity replacement subset of ReFACT. 1 Introduction As Large Language Models (LLMs) increasingly mediate access to scientific and medical information, their tendency to generate plausible but false content poses significant risks. Unlike obvious factual errors that can be easily verified, scientific misinformation is often subtle and domain-specific. Such inaccuracies may appear credible and require expert knowledge to detect, presenting a critical challenge for reliable knowledge dissemination in online discourse. Numerous benchmarks have emerged to assess LLM reliability, yet even fundamental terminology remains unsettled. For example, Ji et al. 2023 define hallucination as “unfaithfulness to the provided source”, while Augenstein et al. 2023 expand the definition to include inconsistencies with world knowledge. One recent work, such as Bang et al. 2025 , distinguishes between hallucination and factuality by introducing intrinsic (input-inconsistent) and extrinsic (training-inconsistent) error types. These varying perspectives reflect ongoing debate about what constitutes a model error–whether errors are rooted in the input, a model’s internal knowledge, or its training data. In this work, we focus on output-level phenomena, independent of model internals or training data provenance. We adopt the term scientific confabulation to describe a distinct class of LLM failure: the generation of fluent, plausible, and contextually appropriate scientific text that is nonetheless factually incorrect. Our use of confabulation draws on its clinical meaning—fabricated, but coherent and convincing, memories or explanations that are ultimately false. While Sui et al. 2024b highlight potential narrative affordances, we adopt the term to emphasize epistemic failures in truth-sensitive scientific contexts. Scientific confabulations are particularly insidious because of their surface fluency and domain-specific language, often requiring expert knowledge to detect. For instance, a model might erroneously describe DNA replication using RNA mechanisms (see Figure 1 ). To address this gap in the evaluation landscape, we introduce ReFACT ( Re ddit F alse A nd C orrect T exts), the first benchmark specifically designed to detect, localize and correct scientific confabulation in LLMs. ReFACT is constructed from authentic, human-authored scientific discourse in r/AskScience–a community with over 23 million members and rigorous moderation standards. Our contributions include: • A Benchmark for Scientific Confabulation : 1,001 question–answer pairs across 10 scientific domains, each with a factual answer and a confabulated counterpart, annotated with error types and span-level error annotations. • Three-Tier Evaluation Framework : Comprehensive assessment of (1) binary confabulation judgment, (2) fine-grained error localization at the span level, and (3) correction generation. • Human-Verified Generation Pipeline : Combining LLM-driven transformations with multi-annotator human verification to ensure high-quality, domain-relevant confabulations. • Systematic Analysis of Model Failures : Benchmarking 9 state-of-the-art LLMs, revealing that even the best models achieve only \sim 54% accuracy. Our analysis highlights persistent challenges in distinguishing factual from confabulated content, underscoring the need for fine-grained evaluation. 2 Related Work Figure 2: Fact Transformation Pipeline (Data Creation Process). Given a filtered Reddit question–answer pair, factual statements are extracted and systematically corrupted through: (1) Fact Extraction – identify factual claims from the original answer; (2) Selection – selecting the most convincing fact for negation or a keyphrase for entity replacement; (3) Transformation – applying either negation (flipping the factual assertion) or entity replacement (substituting technical terms, e.g., “your DNA” \rightarrow “your RNA”); (4) Annotation — marking all generated confabulations with precise span-level error tags. The figure shows the systematic pipeline used to generate span-annotated confabulated answers for the ReFACT benchmark. 2.1 Factuality, Hallucination and Confabulation A range of prior work has sought to define and taxonomize errors produced by Large Language Models (LLMs), but terms such as hallucination , factuality and faithfulness are often used inconsistently ( Ji et al. 2023 ) . Factuality typically refers to agreement with external or real-world facts, while Hallucination , adapted from psychology, describes model outputs that introduce information not grounded in reality or that contradict the input. In contraset, Faithfulness is defined as strict consistency with the source input: a model is unfaithful if it generates content not directly supported by the input, regardless of factual correctness ( Maynez et al. 2020 ; Augenstein et al. 2023 ) . Recently, confabulations , have gained traction describing an error subset of fluent, coherent and contextually plausible content that nontheless contains subtle factual inaccuracies ( Sui et al. 2024a ) . For example, a model might claim, "Mitochondria produce glucose through cellular respiration.", which sounds plausible to non-experts but is biologically inaccurate. Notably, most prior work on confabulation has focused on general patterns, with limited attention to domain-specific epistemic failures in scientific context. To address this gap, we introduce a benchmark specifically designed for fine-grained evaluation of scientific confabulation. 2.2 Factuality and Hallucination Benchmarks The emergence of hallucinations in LLM-generated text has shifted misinformation detection research beyond traditional settings focused on fake news and social media diffusion, toward methods that operate without reliance on external cues ( Chen and Shu 2023 ) . While early approaches relied on binary classification of texts as factual or not, real-world scenarios increasingly demand fine-grained detection in complex, user-generated content. Binary Factuality Benchmarks TruthfulQA Lin et al. 2022 serves as a widely used benchmark, providing a set of questions and a fine-tuned GPT-3 model to evaluate the factuality of an LLM response. Recent methods include prompt-based detection ( Lee et al. 2023 ) and agent-based workflows with enhanced explainability ( Li et al. 2024b ) . Although some studies argue that LLMs inherently have some capability to detect misinformation Guan et al. 2024 ; Huang and Sun 2024 , Hu et al. 2024 show that a specially trained BERT-based architecture outperforms current LLMs on their evaluation set. Fine-Grained Factuality Benchmarks While binary factuality detection remains widely used, real-world scenarios increasingly require fine-grained evaluation that can localize and characterize subtle errors. FactScore ( Min

Read the complete text