Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights
Wenbo Chen, Veena Padmanabhan, Tootiya Giyahchi, Elaine Wong, Leman Akoglu · 2026
Read Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights by Wenbo Chen, Veena Padmanabhan, Tootiya Giyahchi, Elaine Wong, Leman Akoglu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11330.txt.
Introduction
Chen and four colleagues rethink how hallucination detectors for language models should be judged. They set out a list of traits good benchmarks should have, and find no existing benchmark has them all. Two gaps stand out: grounded benchmarks built on retrieval with long context, and realistic noisy labels for stress tests. Their new open benchmark, TRIVIA+, fills both gaps, and tests show current detectors still have ample room to improve.
Catalog record
| CREATOR | Wenbo Chen, Veena Padmanabhan, Tootiya Giyahchi, Elaine Wong, Leman Akoglu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61522 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11330
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights Wenbo Chen, Veena Padmanabhan, Tootiya Giyahchi, Elaine Wong, Leman Akoglu Abstract Hallucination, broadly referring to unfaithful, fabricated, or inconsistent content generated by LLMs, has wide-ranging implications. Therefore, a large body of effort has been devoted to detecting LLM hallucinations, as well as designing benchmark datasets for evaluating these detectors. In this work, we first establish a desiderata of properties for hallucination detection benchmarks (HDBs) to exhibit for effective evaluation. A critical look at existing HDBs through the lens of our desiderata reveals that none of them exhibits all the properties . We identify two largest gaps: (1) RAG-based grounded benchmarks with long context are severely lacking (partly because length impedes human annotation); and (2) Existing benchmarks do not make available realistic label noise for stress-testing detectors although real-world use-cases often grapple with label noise due to human or automated/weak annotation. To close these gaps, we build and open-source a new RAG-based HDB called Trivia + + that underwent a rigorous human annotation process. Notably, our benchmark exhibits all desirable properties including (1) Trivia + + contains samples with the longest context in the literature; and (2) we design and share four sets of noisy labels with different, both sample-dependent and sample-independent, noise schemes. Finally, we perform experiments on RAG-based HDBs , including our Trivia + + , using popular SOTA detectors that reveal new insights : (i) ample room remains for current detectors to reach the performance ceiling on RAG-based HDBs, (ii) the basic LLM-as-a-Judge baseline performs competitively, and (iii) label noise hinders detection performance. We expect that our findings, along with our proposed benchmark 1 1 1 We release the benchmark dataset with expert annotations at github.com/amazon-science/hallucination-benchmark-trivialplus . , will motivate and foster needed research on hallucination detection for RAG-based tasks. 1 Introduction Modern large language models (LLMs) have propelled advances in various real-world domains including e-commerce Jiang et al. 2024 , medicine Thirunavukarasu et al. 2023 , law tho ; Greenstein , to name a few. While LLM-driven AI technologies offer real-world impact, the fact that they hallucinate remains a big obstacle in the safe and secure usage of generative AI tools Natalie Sherman ; Magesh et al. 2024 ; Hong et al. 2024 . Various efforts have been devoted to preventing LLM hallucinations. Mitigation strategies include retrieval-augmented generation (RAG) Li et al. 2024 , reasoning Dhuliawala et al. 2024 , self-reflection Ji et al. 2023 , self-refinement Madaan et al. 2023 , among others Zhang et al. 2023 . Nevertheless, hallucinations persist making both proactive mitigation and post hoc detection essential Luo et al. 2024 . As a result, the field has seen a surge of interest on hallucination detection approaches Huang et al. 2025 ; Luo et al. 2024 ; Zhang et al. 2023 and a large body of hallucination detection benchmark (HDB) datasets have also been developed (see Table 1 ). Despite the long list of HDBs, we point to a large gap: only a handful of RAG-based HDBs exists in the literature . These HDBs are especially hard for humans to annotate, as they (1) exhibit considerably long context settings, and (2) associate with knowledge-intensive tasks Lewis et al. 2020 . In this work, we build and open-source a new RAG-based HDB called Trivia + ; representative of knowledge-intensive LLM tasks and thus grounded on often long context—rendering hallucination detection even more challenging. Furthermore, we identify characteristics that a HDB ideally exhibits. While admittedly not exhaustive, the list offers a systematic lens to assess existing HDBs. Lastly, experiments on RAG-based HDBs report new findings. The following summarizes the contributions of this work. • Desiderata for HDBs (Hallucination Detection Benchmarks): We compose a list of desirable properties for a HDB to exhibit, through the lens of which we scrutinize the existing HDBs—revealing that none of them satisfies all the criteria in our desiderata. (See Table 1 .) • A New Benchmark: We introduce Trivia + , a new RAG-based HDB that exhibits seven key properties in our desiderata. Namely, it consists of (i) organic , naturally-occurring hallucinations; (ii) human-verified labels, with each sample annotated up to 6 times; (iii) long-context (i.e. high-demand, hard-to-label) LLM tasks; (iv) training labels with realistic noise (for supervised detectors); (v) extrinsic and intrinsic hallucinations; from (vi) multiple LLMs; and (vii) multiple domains. • Empirical Findings: Experiments on RAG-based HDBs find that (1) hallucination prevails in grounded tasks and current detectors leave ample room to ceiling performance, (2) while supervised and fine-tuned models can be effective, the simple LLM-as-a-Judge performs competitively, in contrast to reports in past literature Niu et al. 2024 , and (3) sample-dependent (as opposed to random) label noise hinders detection significantly. • Open Problems: Our findings highlight the scarcity of (a) HDBs with long context—and effective detectors on RAG-based generations, (b) HDBs with realistic label noise—and detectors leveraging robust learning; and (c) HDBs that span from multiple domains—and detectors that can generalize across domains. 2 A Desiderata for Hallucination Detection Benchmarking Table 1: Comparison of HDBs w.r.t. desirable desiderata (Sec. 2 ). Proposed Trivia + exhibits all seven properties; with (i) organic generations, (ii) human-verified labels (for evaluation), (iii) four sets of realistic label noise (for training), on (iv) RAG-based long-context tasks (knowledge-intensive, hard-to-annotate), with (v) both extrinsic and intrinsic hallucinations, from (vi) multiple modern LLMs and (vii) multiple domains. None of the existing HDBs fully meets the desiderata. RAG-based HDBs are few, and are highlighted in red . Those marked with × do not make LLM generations available. Symbol ? denotes that it is unclear whether the dataset spans multiple domains. Hal.s: Test-Labels: Train-Labels: Context: Type: LLMs: Domains: # Dataset Organic Human Realistic noise Long/RAG Faith. Multiple Multiple 1 HADES Liu et al. 2022 ✓ ? 2 Frank Marfurt and Henderson 2022 ✓ ✓ 3 TLHD-CNNDM Marfurt and Henderson 2022 ✓ ✓ ✓ 4 SummaC Laban et al. 2022 ✓ ✓ ✓ ✓ 5 WikiBio+ Manakul et al. 2023 ✓ ✓ 6 HaluEval Li et al. 2023 ✓ ✓ ✓ 7 HILT Rawte et al. 2023 ✓ × ✓ ✓ ? 8 PHD Yang et al. 2023 ✓ ✓ ✓ 9 DelucionQA Sadat et al. 2023 ✓ × ✓ ✓ ✓ 10 RAGTruth Niu et al. 2024 ✓ ✓ ✓ ✓ ✓ ✓ 11 TofuEval Tang et al. 2024 ✓ ✓ ✓ ✓ ? 12 FAVABench Mishra et al. 2024 ✓ ✓ ✓ ✓ 13 SHROOM Mickus et al. 2024 ✓ ✓ \ast ✓ ✓ ✓ 14 DiaHalu Chen et al. 2024a ✓ ✓ ✓ ✓ ✓ 15 ERBench Oh et al. 2024 ✓ auto ✓ ✓ 16 Dolly (AC) Hu et al. 2024 ✓ ✓ ✓ ✓ ✓ 17 Dolly (NC) Hu et al. 2024 ✓ ✓ ✓ ✓ ✓ ✓ 18 ANAH Ji et al. 2024 ✓ ✓ ✓ ✓ ✓ 19 HaluEval-Wild Zhu et al. 2025 ✓ × ✓ 20 FACTS Jacovi et al. 2025 ✓ × ✓ ✓ ✓ ✓ 21 Mu-SHROOM Vázquez et al. 2025 ✓ ✓ \ast ✓ ✓ 22 FaithEval Ming et al. 2025 ✓ ✓ ✓ ✓ 23 FaithBench Bao et al. 2025 ✓ ✓ ✓ ✓ ✓ 24 Trivia + (this paper) ✓