VeriTrail: Closed-Domain Hallucination Detection with Traceability
Dasha Metropolitansky, Jonathan Larson · 2025
Read VeriTrail: Closed-Domain Hallucination Detection with Traceability by Dasha Metropolitansky, Jonathan Larson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21786.txt.
Introduction
Dasha Metropolitansky and Jonathan Larson present a tool for catching made-up content in text that AI models were told to base on sources. Their 2025 paper names this failure closed-domain hallucination. The risk grows when a task runs through several AI steps, since errors can sneak in midway and hide in the final text.
Their method, VeriTrail, does more than flag a bad final answer. It traces where the false content likely entered and how true content was carried through each step. They also built the first test sets that keep every in-between output, with human labels on the final ones. VeriTrail beat older methods on both sets.
Catalog record
| CREATOR | Dasha Metropolitansky, Jonathan Larson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 122 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21786
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VeriTrail: Closed-Domain Hallucination Detection with Traceability Dasha Metropolitansky, Jonathan Larson Abstract Even when instructed to adhere to source material, language models often generate unsubstantiated content – a phenomenon known as “closed-domain hallucination.” This risk is amplified in processes with multiple generative steps (MGS), compared to processes with a single generative step (SGS). However, due to the greater complexity of MGS processes, we argue that detecting hallucinations in their final outputs is necessary but not sufficient: it is equally important to trace where hallucinated content was likely introduced and how faithful content may have been derived from the source material through intermediate outputs. To address this need, we present VeriTrail, the first closed-domain hallucination detection method designed to provide traceability for both MGS and SGS processes. We also introduce the first datasets to include all intermediate outputs as well as human annotations of final outputs’ faithfulness for their respective MGS processes. We demonstrate that VeriTrail outperforms baseline methods on both datasets. 1 Introduction Language models (LMs) are widely used to generate content based on source text. However, even when instructed to adhere to the source, LMs are known to produce unsupported content – a phenomenon known as “closed-domain hallucination” ( OpenAI et al., 2024 ) . Detecting hallucination is important in many settings (e.g., a doctor seeking guidance from medical literature, a lawyer summarizing precedent cases, a customer service agent answering questions based on policy documents). Closed-domain hallucination detection is also known as “faithfulness evaluation” – we use both terms (as well as “hallucination detection” for brevity) interchangeably throughout this paper. Processes that use LMs to generate content based on source material can be divided into two categories: processes with a single generative step (SGS) and processes with multiple generative steps (MGS) . In SGS processes, the LM produces only a final output, without generating any intermediate outputs. One limitation of SGS processes is that they may be less reliable for long documents or large collections of documents: if the source material exceeds the model’s context window, truncation or retrieval is required, increasing the risk of information loss. Given the limitations of SGS processes, applications of LMs increasingly rely on MGS processes, where intermediate outputs generated by the LM are used as inputs to subsequent steps. In MGS processes, since the source material (or the task) can be split into smaller parts, truncation or retrieval may not be necessary, reducing the risk of information loss. However, MGS processes are more susceptible to hallucination, as each step presents an additional opportunity for errors to arise and propagate. Therefore, as the use of MGS processes accelerates, effective hallucination detection methods become increasingly important. We argue, however, that detection alone is not enough. In many settings, we need to understand how the output may have been derived from the source ( provenance ) and where errors may have been introduced ( error localization ). Provenance helps users verify and trust the output, while error localization is critical for addressing hallucinations and understanding which parts of the process are most error-prone. We refer to provenance and error localization collectively as traceability . The transparency enabled by traceability is especially important for MGS processes due to their complexity. Prior works frame faithfulness evaluation as assessing whether a given LM output is supported by the source material and do not distinguish between intermediate and final outputs (see § 5 for examples). For SGS processes, this simplification is reasonable: since there is only one generative step, evaluating its output against the source text is sufficient both to assess faithfulness and to provide traceability. In MGS processes, however, while we can still evaluate the final output against the source text to determine whether hallucination occurred, we cannot achieve traceability without utilizing the intermediate outputs. A simplistic approach to traceability is to check the final output against each individual intermediate output. However, this approach can be prohibitively expensive when there are many intermediate outputs (e.g., one of the processes studied in this paper produced over 100,000 intermediate outputs). The simplistic approach also fails when the final output is based on a combination of intermediate outputs. For instance, suppose the final output states: “Company X acquired two startups in 2020 as part of its expansion into healthcare.” One intermediate output mentions a single acquisition in 2020; a second references another acquisition in 2020; and a third describes Company X’s overall acquisition strategy in 2020 as focused on expansion into healthcare. Taken individually, none of these intermediate outputs supports the claim – but together, they do. These limitations highlight the need for an approach to traceability that goes beyond evaluating outputs in isolation. This paper makes the following contributions: 1. We propose a conceptual framework that provides a unified representation of generative processes for the purpose of faithfulness evaluation. 2. We introduce VeriTrail, the first closed-domain hallucination detection method to provide traceability for MGS and SGS processes. We also demonstrate that VeriTrail outperforms baseline methods in hallucination detection while remaining cost-effective. 3. We construct FABLES+ and DiverseSumm+, the first datasets to include all intermediate outputs as well as human annotations of final outputs’ faithfulness for their respective MGS processes. 1 1 1 The datasets will be released at https://aka.ms/veritrail-datasets . 2 Conceptual Framework We define a non-generative step as an operation that outputs only unmodified text spans from the input (e.g., noun phrase extraction using spaCy; Honnibal et al., 2020 ). In contrast, a generative step may modify the input text or introduce new information. Steps involving LMs are typically generative, although exceptions exist (e.g., constrained decoding; Geng et al., 2024 ). A generative process is a sequence of steps that produces a final output from a set of source documents D and includes at least one generative step. At each step, the input consists of text spans from D and/or outputs from earlier steps. Generative processes can be categorized as containing either a single generative step (SGS) or multiple generative steps (MGS) . An example of an SGS process is Retrieval-Augmented Generation (RAG) where a non-generative retrieval system selects a subset of the source material to provide as input to an LM ( Lewis et al., 2020 ) . Although RAG involves two steps, it is an SGS process because only one step is generative. We provide examples of MGS processes in Appendix A , including two processes used in our experiments: hierarchical summarization ( Wu et al., 2021 ; Chang et al., 2023 ) and GraphRAG ( Edge et al., 2025 ) . We model a generative process as a directed acyclic graph (DAG) G=(V,E) , where each node v\in V represents a text span, either originating from D or produced by a step. Each directed edge (u,v)\in E indicates that node u was included as an input to the step that produced node v . For any node v , we define its source nodes as \operatorname{src}(v)=\{u\in V\mid(u,v)\in E\} – the set of nodes used as input to produce v . Root nodes V_{0}\subseteq V have no incoming edges (i.e., \forall\,v\in V_{0},\ \operatorname{src}(v)=\emptyset ) and correspond to text spans from D . The terminal node v^{*}\in V has no outgoing edges and represents the final output. We refer to