Language models should be subject to repeatable, open, domain-contextualized hallucination benchmarking
Justin D. Norman, Michael U. Rivera, D. Alex Hughes · 2025
Read Language models should be subject to repeatable, open, domain-contextualized hallucination benchmarking by Justin D. Norman, Michael U. Rivera, D. Alex Hughes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17345.txt.
Introduction
Justin Norman, Michael Rivera, and Alex Hughes argue that the field lacks a sound way to gauge hallucination in language models. Text that sounds right but is wrong is widely thought to be common. It is also thought to threaten safe use. Yet little work gauges how common it is in full.
The team argues that models should be tested in a way that can be run again, is open, and fits each domain. Experts should be grounded in the process. They give a taxonomy of hallucinations to structure the task. A case study then shows the stakes. When experts are absent at the early stages of benchmark data work, the scores lack validity and use in practice. The paper calls for sound test tools before the numbers are trusted.
Catalog record
| CREATOR | Justin D. Norman, Michael U. Rivera, D. Alex Hughes |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17345
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language models should be subject to repeatable, open, domain-contextualized hallucination benchmarking Justin D. Norman, Michael U. Rivera, D. Alex Hughes Abstract Plausible, but inaccurate, tokens in model-generated text are widely believed to be pervasive and problematic for the responsible adoption of language models. Despite this concern, there is little scientific work that attempts to measure the prevalence of language model hallucination in a comprehensive way. In this paper, we argue that language models should be evaluated using repeatable, open, and domain-contextualized hallucination benchmarking. We present a taxonomy of hallucinations alongside a case study that demonstrates that when experts are absent from the early stages of data creation, the resulting hallucination metrics lack validity and practical utility. 1 Introduction Language models should be subject to repeatable, open, domain-contextualized hallucination benchmarking. The current environment of hallucination benchmarking is hampering the development of models that could perform better and with fewer errors and hallucinations. Specifically, because benchmarks are (a) not open; (b) not repeatable; (c) are either domain ignorant or ablate expertize behind private walls, it is not possible for researchers and commercial developers to openly compare model performance in the pursuit of better functioning, safer, and less-carbon intensive frameworks. By developing open and repeatable benchmarks, the language modeling community can establish a shared set of tasks that support model training and experimentation prior to submitting models to public evaluation. When these benchmarks incorporate expert knowledge, they help ensure that resulting models are both valid and practically useful. To identify and mitigate hallucinations in contemporary LLMs, a crucial first step is the development of a process to create a golden dataset that supports domain-specific hallucination benchmarks. One key lesson we learned—the hard way—is that domain expertise must be present from the inception of the dataset. Without domain expert involvement in the early stages of data creation, hallucination metrics lack scientific validity and real-world utility. This paper proceeds as follows: we begin by outlining the problem space of hallucination benchmarking, summarizing the state of the field, providing a working definition of model hallucination, and providing three examples the consequences of model hallucination in the real-world. We then argue why a repeatable, open, domain-contextualized approach will produce better outcomes. We then demonstrate how expert responses differ from model-generated text, drawing on one case study from medicine and a second from politics. We conclude by evaluating alternative perspectives. 2 Language Model Hallucination Although there are several descriptive features about language models --- the amount of time or carbon to train a model, the speed with which a model generates tokens at inference, inter alia --- we focus specifically on controlling incorrect or misleading text, i.e., hallucination benchmarking. 1 1 1 See Appendix B for a brief survey of model performance and accuracy. Recent advances in open and repeatable benchmarks ( srivastava2023 ; bbeh2025 , e.g.,) have helped to constrain general-purpose, model-generated text. The result is that LLM-generated text has become increasingly, though only seemingly, coherent. But the shine, has come off the stochastic parrot — simply generating seemingly coherent language is no longer sufficient, and language models are being pushed to produce outputs that serve concrete business, policy, medical and scientific purposes. Stakeholders increasingly want language models to take expert-like actions in high-stakes domains, but current models are not ready to operate at that level of rigor or reliability. Indeed, as models grow larger, the hallucination problem appears to be worsening ( Zhou2024larger_and_more ) . Early efforts in LLM hallucination benchmarking primarily centered on summarization tasks that compared model-generated summaries against reference texts (e.g., CNN and DailyMail ( hermann2015 ; narayan2018 ) and XSum ( narayan2018 ) ) to assess factual consistency. As LLMs evolved and were used to generate seemingly novel text, new benchmarks that did not rely upon scoring against reference texts became necessary. When benchmarking metrics scored only perceived coherence or helpfulness, teams’ trained models to score better on these measures, but with the side effect of wild hallucinations — entirely ungrounded references, links, and factual claims (see, for example, subsection 2.2 ) ( huang2025survey ) . Recent advancements have moved from evaluating the perceived coherence of language, to assessing the factual accuracy of model-generated text in zero-shot question answering ( Hughes_Vectara_Hallucination_Leaderboard_2023 ; li2023haluevallargescalehallucinationevaluation , see, e.g.,) . We highlight two recent advancements in hallucination detection: the Vectara hallucination leaderboard and HaluEval. The Vectara hallucination leaderboard provides an open and continuously updated platform for evaluating hallucination rates across LLMs using document summary metric, the Hughes Hallucination Evaluation Model (HHEM) ( Hughes_Vectara_Hallucination_Leaderboard_2023 ; hhem-2.1-open ) . HHEM assesses factuality using a zero-shot question-answering setup over grounded documents; humans define ground-truth, binary judgments of correctness. In a separate project, HaluEval collects 45,000 human-annotated samples across multiple generation tasks (e.g., question answering, summarization, open-domain generation), distinguishing between extractive, abstractive, and factual hallucinations ( li2023haluevallargescalehallucinationevaluation ) . Crucially, HaluEval 2.0, provides a domain-relevant benchmarks across biomedicine, finance, science, education, and as well as an open category ( li2024dawndarkempiricalstudy ) . Despite these advancements, we observe that the field has focused on only a small subset of possible dimensions from the full range available to evaluate and constrain models. These selected dimensions often include tasks like causal reasoning, geometric rotation, and calendar scheduling under conditions of incomplete information ( bbeh2025 , see, e.g.) . Certainly these are difficult tasks for machines — without the ability to intervene, data simply cannot possess the full compliment of counterfactual measurements to define a tractable loss function — but they are trivial for any human. It is unsurprising that model performance tends to degrade on zero-shot tasks that fall outside the scope of these evaluations. The use of ever larger models has resulted in improved performance on in-benchmark tasks but also increased hallucination on out-of-benchmark tasks ( Zhou2024larger_and_more ) . 2.1 Defining LLM Hallucination While there is a considerable body of work surrounding language model hallucination, the field has not settled on a single operational definition of what, specifically constitutes a hallucination (see e.g., ziwei2023 ; huang2025survey , for a survey of the field) . The term, hallucination, arises in 2017 in the context of a data-to-text generation task ( wiseman2017challenges ) , the concept features in abstractive summarization in 2020 ( maynez2020faithfulness ) , and features prominently in Stochastic Parrots ( bender2021dangers ) . To create a point of reference within this paper, we define language model hallucinations in the following way: Definition 1 An LLM hallucination is a generated text statement presented without qualifying language that is: incoherent, factually incorrect, or unverified in the judgment of a human expert capable of making that assessment. Qualifying Language A response that does not include qualifying language fails to convey the probabilistic nature of the output, instead presenting ideas as definitive facts. Responses that appropriately include qualifying language — for example, in medicine — might use phrases such as “Those symptoms could be a sign of [diagnosis] but they could also be signs of [alternative diagnosis],” or “It is important to consult with a healthcare provider…” to communicate uncertainty and encourage additional verification. Incoherence The greatest trick language models ever pulled was convincing the world they were coherent. Current generation language models produce syntactically correct, seemingly coherent strings of text ( bender2021dangers )