Entailed Opinion Matters: Improving the Fact-Checking Performance of Language Models by Relying on their Entailment Ability
Gaurav Kumar, Ayush Garg, Debajyoti Mazumder, Aditya Kishore, Babu kumar, Jasabanta Patro · 2025
Read Entailed Opinion Matters: Improving the Fact-Checking Performance of Language Models by Relying on their Entailment Ability by Gaurav Kumar, Ayush Garg, Debajyoti Mazumder, Aditya Kishore, Babu kumar, Jasabanta Patro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15050.txt.
Introduction
Gaurav Kumar and five colleagues improve automated fact-checking by leaning on a language model's sense of what an opinion entails, in a 2025 paper. Fact-checking systems still fall short of real-world accuracy despite training tricks and retrieval help. The team's idea: have a generative model sort evidence and write entailed justifications, then train smaller encoder models on those outputs.
They compared the approach against recent systems and many prompting and fine-tuning strategies. Extra studies covered ablations, error patterns, explanation quality, and transfer to new domains. The authors present the results as a full picture of when borrowed entailment judgments make a fact-checker stronger and where the method still breaks down.
Catalog record
| CREATOR | Gaurav Kumar, Ayush Garg, Debajyoti Mazumder, Aditya Kishore, Babu kumar, Jasabanta Patro |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15050
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Entailed Opinion Matters: Improving the Fact-Checking Performance of Language Models by Relying on their Entailment Ability Gaurav Kumar, Ayush Garg, Debajyoti Mazumder, Aditya Kishore, Babu kumar, Jasabanta Patro Abstract Automated fact-checking has been a challenging task for the research community. Prior work has explored various strategies, such as end-to-end training, retrieval-augmented generation, and prompt engineering, to build robust fact-checking systems. However, their accuracy has not been high enough for real-world deployment. We, on the other hand, propose a new learning paradigm, where evidence classification and entailed justifications made by generative language models (GLMs) are used to train encoder-only language models (ELMs). We conducted a rigorous set of experiments, comparing our approach with recent works along with various prompting and fine-tuning strategies. Additionally, we performed ablation studies, error analysis, quality analysis of model explanations, and a domain generalisation study to provide a comprehensive understanding of our approach. 1 Introduction: Figure 1: A schematic diagram explaining our learning paradigm, which breaks down into three stages. (a) Entailment : Given a claim and its corresponding evidences, a GLM first distinguishes evidences into supporting/ refuting evidence via entailment. (b) Evidence consolidation : Then the same GLM consolidates these two groups into concise supporting and refuting justification. (c) Veracity prediction : Using the claim and both justifications, an ELM is trained to predict veracity. The spread of misinformation on the internet has become a pressing social issue. Its consequences have manifested across social, political, and commercial domains Mozur (2018) ; Allcott and Gentzkow (2017) . Manually verifying facts and detecting misinformation is a slow and costly process. To overcome this, the research community has proposed several approaches for automated fact-checking. Their interest can be gauged by the fact that more than 1,200 research articles and 50 datasets spanning different languages and modalities have been published on this topic Alnabhan and Branco (2024) ; Guo et al. (2022) . Most of them use deep neural architectures to learn latent features for a given input. Researchers, in this case, use attention maps to assess the model’s decision-making process. Recently, with the emergence of generative language models (GLMs), researchers have begun exploring prompt engineering approaches. Zero-shot prompting, few-shot prompting, and CoT-based prompting Qiao et al. (2023) have been among the front-runners. However, the accuracy of existing systems is not high enough to deploy them in the real world. The primary reason behind this is that fact verification is a complex process. For a given claim, language models have to parse through multiple pieces of evidence, which often contradict each other. The contradictory evidence confuses the language models, leading to poor performance. Prompt-engineering approaches, on the other hand, have achieved human-level performance when used in conjunction with a large generative language model (with over a trillion parameters) Qiao et al. (2023) . However, their deployment is costly, as large GLMs usually remain behind a paywall. Using prompting approaches on smaller GLMs such as Mistral-7B Jiang et al. (2023) , Llama-7B AI@Meta (2024) , etc., may seem to be a solution, but (i) their performance is lower than training-based approaches, and (ii) the model outcome often suffers from bias and hallucination. To overcome this, we propose a simple yet effective strategy, formulating a new training paradigm for fact-checking. Here, we classify evidence and generate entailed justifications using GLMs, and finetune ELMs for veracity predictions based on these justifications. Particularly, we follow a three-step process as illustrated in Figure 1 . We conducted a variety of training (TBE)- and inference (IBE)-based experiments to evaluate the effectiveness of our approach across different input scenarios. All of our experiments were done around three research questions. They are, • R1: "How well do the language models perform when only claim and raw evidence sentences are available?" • R2: "Does prompted claim-evidence understanding improve the performance of the language models?" • R3: "Can prompted entailed justifications improve the claim veracity prediction?" Some of our key contributions and observations in this work can be summarised as follows: • We conducted three training-based (TBEs) and four inference-based (IBEs) experiments along the line of R1, R2, and R3 for two monolingual English datasets, i.e., LIAR-RAW and RAW-FC. For RAW-FC, training with raw evidence sentences (TBE-1) and overall claim-evidence understanding (TBE-2) registered an improvement of up to 8.20% and 16.39% in macro-F1 ( MF1 ) over the baselines. Training with the entailed justifications (TBE-3), on the other hand, outperformed the baselines by a significant margin (up to 28.57% and 44.26% for LIAR-RAW and RAW-FC, respectively). • We performed an ablation study by removing individual supporting and refuting entailed justifications. We found that both of the entailed justifications were important, as their removal led to a decline in MF performance. • We considered the entailed justifications generated by GLMs as model explanations and evaluated them using two strategies: (i) checking lexical overlap and semantic matching, and (ii) conducting subjective evaluation by the GLMs themselves. Notably, explanations generated by Llama received the highest subjective scores, correlating with the highest veracity prediction performance. We performed a thorough linguistic analysis of the explanations by looking into the model attention scores. We found that ELMs assigned higher attention scores to appropriate keywords and factual elements in samples with extreme labels (e.g., true or false). However, for borderline samples (e.g., half-true labels), attention scores seemed scattered. These studies were kept in the appendix due to space constraints. • We extended our evaluation (TBE-3) to multilingual (X-Fact and Ru22fact) and multimodal (Factify-2, MOCHEG, and VERITE) datasets to check its domain generalisability. Our approach resulted in performance improvements (in MF1 ) of up to 19.70% in multilingual and 54.10% in multimodal domains. 2 Related Works: Automated fact-checking has been extensively studied in the literature, with prior work spanning diverse datasets, modeling approaches, and task settings Guo et al. (2022) ; Vladika and Matthes (2023) . While most fact-checking research has focused on text-only, English-language claims, recent works have expanded the literature to multimodal and multilingual settings Akhtar et al. (2023) ; Tufchi et al. (2023) . Overall approaches: Early approaches relied heavily on manual feature engineering and retrieval from external knowledge sources Guo et al. (2022) ; Zeng et al. (2021) . With the rise of deep learning, encoder-only language models (ELMs) such as BERT-like architectures became a dominant approach through fine-tuning Lee et al. (2020) . In recent years, larger generative language models (GLMs) have shown dominance in this field due to their vast pretraining knowledge Vykopal et al. (2024) . While effective, language models often struggle with bias and hallucination when evidence is noisy or contradictory Huang et al. (2025) . Handling evidence: To mitigate these issues, recent works have introduced evidence discrimination, aggregation, and conflict resolution methods. State-of-the-art pipelines have explored diverse strategies for handling evidence in fact-checking. For instance, Yue et al. (2024) used retrieved evidence to generate supporting and opposing arguments and applied few-shot prompting for claim verification. However, they did not explicitly divide evidence sets that may contain contradictory statements. Similarly, L-Defense Wang et al. (2024a) attempted to filter or weight evidence before prediction. However, the authors still relied on complex attention mechanisms (or learned relevance scores), which may fail when many pieces of evidence conflict with each other. Apart from