Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights

Wenbo Chen, Veena Padmanabhan, Tootiya Giyahchi, Elaine Wong, Leman Akoglu · 2026

Read Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights by Wenbo Chen, Veena Padmanabhan, Tootiya Giyahchi, Elaine Wong, Leman Akoglu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11330.txt.

Introduction

Chen and four colleagues rethink how hallucination detectors for language models should be judged. They set out a list of traits good benchmarks should have, and find no existing benchmark has them all. Two gaps stand out: grounded benchmarks built on retrieval with long context, and realistic noisy labels for stress tests. Their new open benchmark, TRIVIA+, fills both gaps, and tests show current detectors still have ample room to improve.

Catalog record

CREATORWenbo Chen, Veena Padmanabhan, Tootiya Giyahchi, Elaine Wong, Leman Akoglu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61522
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11330

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights Wenbo Chen, Veena Padmanabhan, Tootiya Giyahchi, Elaine Wong, Leman Akoglu Abstract Hallucination, broadly referring to unfaithful, fabricated, or inconsistent content generated by LLMs, has wide-ranging implications. Therefore, a large body of effort has been devoted to detecting LLM hallucinations, as well as designing benchmark datasets for evaluating these detectors. In this work, we first establish a desiderata of properties for hallucination detection benchmarks (HDBs) to exhibit for effective evaluation. A critical look at existing HDBs through the lens of our desiderata reveals that none of them exhibits all the properties . We identify two largest gaps: (1) RAG-based grounded benchmarks with long context are severely lacking (partly because length impedes human annotation); and (2) Existing benchmarks do not make available realistic label noise for stress-testing detectors although real-world use-cases often grapple with label noise due to human or automated/weak annotation. To close these gaps, we build and open-source a new RAG-based HDB called Trivia + + that underwent a rigorous human annotation process. Notably, our benchmark exhibits all desirable properties including (1) Trivia + + contains samples with the longest context in the literature; and (2) we design and share four sets of noisy labels with different, both sample-dependent and sample-independent, noise schemes. Finally, we perform experiments on RAG-based HDBs , including our Trivia + + , using popular SOTA detectors that reveal new insights : (i) ample room remains for current detectors to reach the performance ceiling on RAG-based HDBs, (ii) the basic LLM-as-a-Judge baseline performs competitively, and (iii) label noise hinders detection performance. We expect that our findings, along with our proposed benchmark 1 1 1 We release the benchmark dataset with expert annotations at github.com/amazon-science/hallucination-benchmark-trivialplus . , will motivate and foster needed research on hallucination detection for RAG-based tasks. 1 Introduction Modern large language models (LLMs) have propelled advances in various real-world domains including e-commerce Jiang et al. 2024 , medicine Thirunavukarasu et al. 2023 , law tho ; Greenstein , to name a few. While LLM-driven AI technologies offer real-world impact, the fact that they hallucinate remains a big obstacle in the safe and secure usage of generative AI tools Natalie Sherman ; Magesh et al. 2024 ; Hong et al. 2024 . Various efforts have been devoted to preventing LLM hallucinations. Mitigation strategies include retrieval-augmented generation (RAG) Li et al. 2024 , reasoning Dhuliawala et al. 2024 , self-reflection Ji et al. 2023 , self-refinement Madaan et al. 2023 , among others Zhang et al. 2023 . Nevertheless, hallucinations persist making both proactive mitigation and post hoc detection essential Luo et al. 2024 . As a result, the field has seen a surge of interest on hallucination detection approaches Huang et al. 2025 ; Luo et al. 2024 ; Zhang et al. 2023 and a large body of hallucination detection benchmark (HDB) datasets have also been developed (see Table 1 ). Despite the long list of HDBs, we point to a large gap: only a handful of RAG-based HDBs exists in the literature . These HDBs are especially hard for humans to annotate, as they (1) exhibit considerably long context settings, and (2) associate with knowledge-intensive tasks Lewis et al. 2020 . In this work, we build and open-source a new RAG-based HDB called Trivia + ; representative of knowledge-intensive LLM tasks and thus grounded on often long context—rendering hallucination detection even more challenging. Furthermore, we identify characteristics that a HDB ideally exhibits. While admittedly not exhaustive, the list offers a systematic lens to assess existing HDBs. Lastly, experiments on RAG-based HDBs report new findings. The following summarizes the contributions of this work. • Desiderata for HDBs (Hallucination Detection Benchmarks): We compose a list of desirable properties for a HDB to exhibit, through the lens of which we scrutinize the existing HDBs—revealing that none of them satisfies all the criteria in our desiderata. (See Table 1 .) • A New Benchmark: We introduce Trivia + , a new RAG-based HDB that exhibits seven key properties in our desiderata. Namely, it consists of (i) organic , naturally-occurring hallucinations; (ii) human-verified labels, with each sample annotated up to 6 times; (iii) long-context (i.e. high-demand, hard-to-label) LLM tasks; (iv) training labels with realistic noise (for supervised detectors); (v) extrinsic and intrinsic hallucinations; from (vi) multiple LLMs; and (vii) multiple domains. • Empirical Findings: Experiments on RAG-based HDBs find that (1) hallucination prevails in grounded tasks and current detectors leave ample room to ceiling performance, (2) while supervised and fine-tuned models can be effective, the simple LLM-as-a-Judge performs competitively, in contrast to reports in past literature Niu et al. 2024 , and (3) sample-dependent (as opposed to random) label noise hinders detection significantly. • Open Problems: Our findings highlight the scarcity of (a) HDBs with long context—and effective detectors on RAG-based generations, (b) HDBs with realistic label noise—and detectors leveraging robust learning; and (c) HDBs that span from multiple domains—and detectors that can generalize across domains. 2 A Desiderata for Hallucination Detection Benchmarking Table 1: Comparison of HDBs w.r.t. desirable desiderata (Sec. 2 ). Proposed Trivia + exhibits all seven properties; with (i) organic generations, (ii) human-verified labels (for evaluation), (iii) four sets of realistic label noise (for training), on (iv) RAG-based long-context tasks (knowledge-intensive, hard-to-annotate), with (v) both extrinsic and intrinsic hallucinations, from (vi) multiple modern LLMs and (vii) multiple domains. None of the existing HDBs fully meets the desiderata. RAG-based HDBs are few, and are highlighted in red . Those marked with × do not make LLM generations available. Symbol ? denotes that it is unclear whether the dataset spans multiple domains. Hal.s: Test-Labels: Train-Labels: Context: Type: LLMs: Domains: # Dataset Organic Human Realistic noise Long/RAG Faith. Multiple Multiple 1 HADES Liu et al. 2022 ✓ ? 2 Frank Marfurt and Henderson 2022 ✓ ✓ 3 TLHD-CNNDM Marfurt and Henderson 2022 ✓ ✓ ✓ 4 SummaC Laban et al. 2022 ✓ ✓ ✓ ✓ 5 WikiBio+ Manakul et al. 2023 ✓ ✓ 6 HaluEval Li et al. 2023 ✓ ✓ ✓ 7 HILT Rawte et al. 2023 ✓ × ✓ ✓ ? 8 PHD Yang et al. 2023 ✓ ✓ ✓ 9 DelucionQA Sadat et al. 2023 ✓ × ✓ ✓ ✓ 10 RAGTruth Niu et al. 2024 ✓ ✓ ✓ ✓ ✓ ✓ 11 TofuEval Tang et al. 2024 ✓ ✓ ✓ ✓ ? 12 FAVABench Mishra et al. 2024 ✓ ✓ ✓ ✓ 13 SHROOM Mickus et al. 2024 ✓ ✓ \ast ✓ ✓ ✓ 14 DiaHalu Chen et al. 2024a ✓ ✓ ✓ ✓ ✓ 15 ERBench Oh et al. 2024 ✓ auto ✓ ✓ 16 Dolly (AC) Hu et al. 2024 ✓ ✓ ✓ ✓ ✓ 17 Dolly (NC) Hu et al. 2024 ✓ ✓ ✓ ✓ ✓ ✓ 18 ANAH Ji et al. 2024 ✓ ✓ ✓ ✓ ✓ 19 HaluEval-Wild Zhu et al. 2025 ✓ × ✓ 20 FACTS Jacovi et al. 2025 ✓ × ✓ ✓ ✓ ✓ 21 Mu-SHROOM Vázquez et al. 2025 ✓ ✓ \ast ✓ ✓ 22 FaithEval Ming et al. 2025 ✓ ✓ ✓ ✓ 23 FaithBench Bao et al. 2025 ✓ ✓ ✓ ✓ ✓ 24 Trivia + (this paper) ✓

Read the complete text