Recon, Answer, Verify: Agents in Search of Truth

Satyam Shukla Himanshu Dutta Pushpak Bhattacharyya · 2025

Read Recon, Answer, Verify: Agents in Search of Truth by Satyam Shukla Himanshu Dutta Pushpak Bhattacharyya (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03671.txt.

Introduction

Checking political claims fast, before any verdict exists, is the setting for a 2025 paper by Satyam Shukla, Himanshu Dutta, and Pushpak Bhattacharyya. Most test sets for automated fact checking leak hints from the checkers' own write-ups, which real claims do not come with.

The team built a clean test set of 2,982 claims from politifact.com, with all later analysis and labels stripped out by hand. Models lost about 22 percent of their score on this clean set, which shows how much the hints had helped. Their fix is a three-agent system that poses sub-questions, answers them, and only then assigns a label. The system works across topics and label sets, and it beat prior methods.

Catalog record

CREATORSatyam Shukla Himanshu Dutta Pushpak Bhattacharyya
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03671

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Recon, Answer, Verify: Agents in Search of Truth Satyam Shukla Himanshu Dutta Pushpak Bhattacharyya Abstract Automated fact-checking with large language models (LLMs) offers a scalable alternative to manual verification. Evaluating fact-checking is challenging as existing benchmark datasets often include post-claim analysis and annotator cues, which are absent in real-world scenarios where claims are fact-checked immediately after being made. This limits the realism of current evaluations. We present Politi-Fact-Only (PFO) , a 5-class benchmark dataset of 2,982 political claims from politifact.com 1 1 1 https://www.politifact.com/ , where all post-claim analysis and annotator cues have been removed manually. This ensures that models are evaluated using only the information that would have been available prior to the claim’s verification. Evaluating LLMs on PFO, we see an average performance drop of 22% in terms of macro-f1 compared to PFO’s unfiltered version. Based on the identified challenges of the existing LLM-based fact-checking system, we propose RAV (Recon-Answer-Verify) , an agentic framework with three agents: question generator, answer generator, and label generator. Our pipeline iteratively generates and answers sub-questions to verify different aspects of the claim before finally generating the label. RAV generalizes across domains and label granularities, and it outperforms state-of-the-art approaches on well-known baselines RAWFC (fact-checking, 3-class) by 25.28% , and on HOVER (encyclopedia, 2-class) by 1.54% on 2-hop, 4.94% on 3-hop, and 1.78% on 4-hop, sub-categories respectively. RAV shows the least performance drop compared to baselines of 16.3% in macro-f1 when we compare PFO with its unfiltered version. 1 Introduction Figure 1: An instance from the PFO dataset where sentences with strike-through represent information added only after the claim was verified (post-publication analysis or annotator cues). These lines were manually deleted to ensure the evidence contained only factual details related to the claim. We also show the metadata available for an instance in PFO. Several studies have explored automated fact-checking, including work by Wang 2017 , Augenstein et al. 2019 , and Gupta and Srikumar 2021 , who created widely-used benchmark datasets. While these datasets contain real-world claims, they are primarily derived from fact-checking websites, where each claim is often accompanied by detailed analysis and verdicts written after the claim was made. This post-claim analysis frequently includes annotator judgments or interpretive statements that explicitly indicate the truthfulness of the claim, what we refer to as leakage . Leakage, also called post-claim analysis or annotator cues, provides information that would not be available in a real-time fact-checking scenario and can inadvertently guide model predictions. To address this, some researchers, such as Yang et al. 2022 and Khan et al. 2022 , have instead used contemporaneous source articles documents published before the claim to simulate more realistic settings. However, even when using pre-claim documents, the evidence may still be insufficient or too loosely related to support or refute the claim effectively, limiting the reliability of model evaluation. To address this problem, we propose a benchmark dataset in the political domain, Politi-Fact-Only (PFO) (see Section 4 ), a subset of Misra 2022 . In this dataset, we manually remove post-claim analysis and annotator cues provided by fact-checkers. For example, phrases such as “But one old picture is being mischaracterized as showing the aftermath of a crash in Eminence, Indiana.” and “but this photo doesn’t show it” (see Figure 1 ) directly signal that the claim is false. Such cues are unavailable in real-world scenarios, where evidence typically consists of content published prior to the claim itself. In PFO, we retain only pre-claim content, ensuring that fact-checking models are evaluated under conditions that more closely resemble real-time verification. The rapid spread of misinformation on digital platforms, especially social media, has intensified the need for domain-agnostic fact-checking systems Vosoughi et al. 2018 . Manual verification is slow and resource-intensive Graves 2018 , and less than half of published articles undergo any fact-checking at all Lewis et al. 2008 . This creates a pressing demand for automated approaches that can scale across domains and support fine-grained labelling. One such effort is ProgramFC Pan et al. 2023 , which uses program-guided reasoning, encoding verification steps as intermediate executable programs. Labels are predicted based on the logical truth of Boolean expressions: true indicates support, and false indicates refutes. However, this approach is restricted to binary classification, limiting its applicability to more nuanced multi-class fact-checking scenarios. Zhang and Gao 2023 worked on decomposing a claim into sub-components and generating questions to verify each of the components. Their study is limited to the political domain. They used multiple in-context examples from the political domain, and their question generation process is based on only those in-context examples. As no task description is provided, it becomes limited to the political domain. To address this challenge, we propose Recon-Answer-Verify (RAV) , a generalized agentic framework. Our approach starts with a Question Generator agent (QG agent ), which produces a sub-question and brief reasoning based on the claim and earlier sub-questions. The reasoning links the claim with previous questions and answers to identify parts that are still unverified. Then, a related sub-question is generated. This makes the process interpretable and applicable across domains. The Answer Generator agent (AG agent ) answers each question using the evidence provided with the claim. The process is iterative: at each step, the QG agent checks whether all aspects of the claim are covered or more questions are needed. Once the claim is fully explored, the Label Generator agent (LG agent ) generates reasoning that connects the questions, answers, and claim, and then predicts the final label. Reasoning is generated during both question and label steps, making fact-checking more transparent. Our contributions are: 1. Politi-Fact-Only (PFO) (Section 4 ): We introduce a benchmark dataset of 2,982 instances evenly distributed across five classes: true, mostly-true, half-true, mostly-false, and false. We manually remove all post-claim analyses and annotator commentary, keeping only factual evidence (see Fig. 1 ) to ensure unbiased input and better grounding in reality. When evaluated on PFO as opposed to its unfiltered version. LLM show a performance drop of 22% on average (see Table 2 ). Content length reduction confirms that 17.79% of the original data consisted of commentary or verdict cues (see Table 1 ). 2. Recon-Answer-Verify (RAV) (Section 5 ): An agentic pipeline that breaks fact verification into a question-answering process. RAV uses domain-agnostic prompts, allowing it to generalize across domains and label granularities. It outperforms state-of-the-art baselines on RAWFC (3-class, multi-domain) by 25.28% , and on HoVer (2-class, encyclopedia) by 1.54% (2-hop), 4.94% (3-hop), and 1.78% (4-hop). RAV shows the least performance drop of 16.3% in macro-f1, when we evaluate it on PFO as opposed to its unfiltered version (Table 3 ). 3. Comprehensive evaluation of the RAV pipeline (Section 7.2 ): We systematically study the effect of different question-generation strategies and question types: Verification (True/False answerable) and inquiry (requires explanation) question on the fact-checking task (see Table 4 ). We conduct experiments with LLMs ranging from 7B to 70B parameters, and analyse how performance over the fact-checking task is affected by the scale of LLMs. We show that the final RAV pipeline, designed to reason iteratively and generate both verification and inquiry questions , achieves better performance compared to other variants (see Table 3 ). 2 Problem

Read the complete text