CLATTER: Comprehensive Entailment Reasoning for Hallucination Detection

Ron Eliav 1 Arie Cattan 1 Eran Hirsch 1 Shahaf Bassan 2 Elias Stengel-Eskin 3 Mohit Bansal 3 Ido Dagan 1 · 2025

Read CLATTER: Comprehensive Entailment Reasoning for Hallucination Detection by Ron Eliav 1 Arie Cattan 1 Eran Hirsch 1 Shahaf Bassan 2 Elias Stengel-Eskin 3 Mohit Bansal 3 Ido Dagan 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05243.txt.

Introduction

Language models sometimes state false facts, called hallucinations, and one way to catch them is to check whether a source text supports each claim. Ron Eliav and co-authors proposed a guided reasoning method for that check in 2025, called CLATTER. Instead of asking a model for one quick verdict, the method walks it through three steps in order.

First the text is split into small claims. Then each claim is matched to evidence in the source and classified, and finally the verdicts are combined. Guided this way, models made finer and more accurate calls. The authors also define scores for the quality of each middle step, which backed the gain.

Catalog record

CREATORRon Eliav 1 Arie Cattan 1 Eran Hirsch 1 Shahaf Bassan 2 Elias Stengel-Eskin 3 Mohit Bansal 3 Ido Dagan 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05243

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CLATTER: Comprehensive Entailment Reasoning for Hallucination Detection Ron Eliav 1 Arie Cattan 1 Eran Hirsch 1 Shahaf Bassan 2 Elias Stengel-Eskin 3 Mohit Bansal 3 Ido Dagan 1 Abstract A common approach to hallucination detection casts it as a natural language inference (NLI) task, often using LLMs to classify whether the generated text is entailed by corresponding reference texts. Since entailment classification is a complex reasoning task, one would expect that LLMs could benefit from generating an explicit reasoning process, as in CoT reasoning or the explicit “thinking” of recent reasoning models. In this work, we propose that guiding such models to perform a systematic and comprehensive reasoning process—one that both decomposes the text into smaller facts and also finds evidence in the source for each fact—allows models to execute much finer-grained and accurate entailment decisions, leading to increased performance. To that end, we define a 3-step reasoning process, consisting of (i) claim decomposition, (ii) sub-claim attribution and entailment classification, and (iii) aggregated classification, showing that such guided reasoning indeed yields improved hallucination detection. Following this reasoning framework, we introduce an analysis scheme, consisting of several metrics that measure the quality of the intermediate reasoning steps, which provided additional empirical evidence for the improved quality of our guided reasoning scheme. 1 Introduction Figure 1: An example of CLATTER reasoning framework to evaluate a claim. The process begins by decomposing the claim into its two sub-claims. Each sub-claim is checked against the source via attribution and entailment analysis. Finally, the results are aggregated to reach a not supported verdict for the overall claim. The output of Large Language Models (LLMs) is often required to be faithful to some reference texts. Such texts might be provided by the user, as in text summarization, retrieved sources, as in RAG settings, or retrieved references against which parametric-based generation is verified for factuality. In such settings, a critical challenge is to detect if the generated output contains unsupported claims, known as hallucinations ( Tian et al. 2020 ; Thorat et al. 2025 ; Ádám Kovács and Recski 2025 ; Paudel et al. 2025 ) . Automated hallucination detection methods can inform users of suspected hallucinations ( Leiser et al. 2024 ; Zhao et al. 2024 ) , correct hallucinations by editing the output ( Wadhwa et al. 2024 ) , or guide models to avoid hallucinations through reinforcement learning ( Roit et al. 2023 ) and controlled decoding ( Wan et al. 2023 ) . Figure 2: Overview of CLATTER process. (i) Decomposition: the original claim is split into individual sub-claims. (ii) Attribution & Entailment: each sub-claim is checked against the source for supporting evidence, refuting evidence, or no evidence. (iii) Aggregation: if all sub-claims are supported, the claim is accepted; otherwise, it is rejected. The task of hallucination detection is mostly seen as an entailment classification task Dagan et al. 2005 ; Bowman et al. 2015 , where the hypothesis is a model-generated output claim while the premise is the source text. Hallucination detection is then implemented using either fine-tuned entailment classifiers ( Zha et al. 2023 ; Kamoi et al. 2023 ; Tang et al. 2024a ) , or via prompting LLMs to complete the entailment task ( Kamoi et al. 2023 ; Laban et al. 2023 ; Min et al. 2023 ; Tang et al. 2024b ) . In our work we focus on the latter scenario, where LLMs are often preferred thanks to their broad domain and language coverage, robustness and accessibility. Since entailment classification is a complex reasoning task, we expect that LLMs might benefit from generating an explicit reasoning process, as in CoT reasoning or the explicit “thinking” of recent reasoning models (Large Reasoning Models, or LRMs). Given such model-generated entailment reasoning, two research questions arise: RQ1: How well do models perform such reasoning on their own, in an un-guided manner? This question is posed with respect to both bottom line entailment classification performance as well as the validity of the reasoning process itself. RQ2: Is it possible to improve such reasoning, by guiding models to perform systematic reasoning steps that follow the inherent semantics of entailment decision-making? Toward addressing these questions, we first formulate a systematic and comprehensive reasoning process for entailment classification, which we term CLATTER: C laim L ocalization & ATT ribution for E ntailment R easoning. This process consists of three steps, namely (i) claim decomposition, (ii) sub-claim attribution and entailment classification, and (iii) aggregated classification, as illustrated in Figures 1 and 2 . Further, we define a set of metrics that measure the validity of the different steps involved in such entailment reasoning. While prior work also decomposes entailment reasoning based on sub-claims, to the best of our knowledge, we are the first to investigate a principled decomposition of this sort as a single LLM reasoning process, as opposed to prior pipeline architectures, which often involve targeted fine-tuned models ( Kamoi et al. 2023 ; Manakul et al. 2023a ; Min et al. 2023 ) . Our experiments show that CLATTER-guided reasoning does improve bottom-line entailment classification, relative to un-guided reasoning. Importantly, CLATTER-guided LRMs perform sub-claim attribution and entailment classification much more accurately, successfully following the prescribed reasoning steps. Overall, our contributions include: (1) introducing CLATTER as a comprehensive multi-step reasoning process for entailment classification by LLMs (Section 2 ); (2) defining assessment metrics for the involved reasoning steps (Section 3 ); (3) analyzing both unguided and CLATTER-guided reasoning, in both CoT and LRM settings, showing the advantages of CLATTER reasoning in both entailment classification and reasoning quality. In the following sections, we describe the CLATTER approach in detail (§ 2 ), present evaluation metrics for the entailment reasoning steps (§ 3 ), describe our experimental setup (§ 4 ), present our results and ablations (§ 5 ), discuss insights from our manual analysis (§ 6 ), and finally contrast with related work (§ 7 ). 2 Comprehensive NLI Reasoning In the following section, we formulate the CLATTER reasoning process, which, in our setting, models are instructed to follow when making an entailment decision. We take the view that a natural-language sentence can be presented as a conjunction of smaller facts Davidson 1967 ; Partee 2008 , all sharing a consistent interpretation, where the sentence is semantically equivalent to the union of these facts. Then, a hypothesis is entailed if all its facts are entailed by the source, contradicted if at least one is contradicted, and neutral otherwise. Consequently, for detecting a hallucination in a given claim, we first decompose a claim into sub-claims. Each sub-claim is then classified by checking for a corresponding piece of evidence in the source: entailed if supported, contradicted if opposed, and neutral if no match is found. Finally, we aggregate the decisions of each sub-claim to provide a prediction for the whole claim. We propose guiding models to follow a systematic process aligned with this perspective. As shown in Fig. 2 , the entailment prediction of a generated claim \mathcal{H} relative to a source \mathcal{P} involves three steps: (i) decomposition, (ii) attribution and entailment classification, and (iii) aggregation. Through the reasoning process, CLATTER provides a set of triples {(h_{i},p_{i},\hat{y}_{i})} , where h_{i} is a sub-claim, p_{i} is the

Read the complete text