Don’t Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning

Yuehan Qin, Shawn Li, Yi Nian, Xinyan Velocity Yu, Yue Zhao, Xuezhe Ma · 2025

Read Don’t Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning by Yuehan Qin, Shawn Li, Yi Nian, Xinyan Velocity Yu, Yue Zhao, Xuezhe Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06438.txt.

Introduction

Yuehan Qin and co-authors present a method that checks the premises inside a user's question before a language model answers, in a 2025 paper. False premises can lead a model to invent facts that fit the error.

The method first turns the question into a logical form. It then looks up facts to test each premise, using retrieval from outside sources. The verdicts are fed back into the prompt, so the reply can correct the record. Tests showed fewer invented claims and better factual accuracy. The approach needs no access to the model's inner scores and no large retraining, which suits live use.

Catalog record

CREATORYuehan Qin, Shawn Li, Yi Nian, Xinyan Velocity Yu, Yue Zhao, Xuezhe Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06438

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Don’t Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning Yuehan Qin, Shawn Li, Yi Nian, Xinyan Velocity Yu, Yue Zhao, Xuezhe Ma Abstract Large language models (LLMs) have shown substantial capacity for generating fluent, contextually appropriate responses. However, they can produce hallucinated outputs, especially when a user query includes one or more false premises —claims that contradict established facts. Such premises can mislead LLMs into offering fabricated or misleading details. Existing approaches include pretraining, fine-tuning, and inference-time techniques that often rely on access to logits or address hallucinations after they occur. These methods tend to be computationally expensive, require extensive training data, or lack proactive mechanisms to prevent hallucination before generation, limiting their efficiency in real-time applications. We propose a retrieval-based framework that identifies and addresses false premises before generation. Our method first transforms a user’s query into a logical representation, then applies retrieval-augmented generation (RAG) to assess the validity of each premise using factual sources. Finally, we incorporate the verification results into the LLM’s prompt to maintain factual consistency in the final output. Experiments show that this approach effectively reduces hallucinations, improves factual accuracy, and does not require access to model logits or large-scale fine-tuning. 1 Introduction Large Language Models (LLMs) generate fluid, context-aware responses but can produce hallucinations when prompted with queries that include hidden factual errors ( Manakul et al. 2023 ; Zheng et al. 2023 ) . These errors, known as false premises , are statements in a user’s question that conflict with real-world facts. Even when LLMs can store accurate information, they may trust the incorrect assumptions embedded in the query and generate misleading outputs ( Yuan et al. 2024 ) . This is especially problematic in sensitive applications such as finance or healthcare, where mistakes can cause serious harm ( Pal et al. 2023 ) . Prior research distinguishes between factuality hallucinations—where the output conflicts with known facts—and faithfulness hallucinations—where the response diverges from the provided context or user instructions ( Snyder et al. 2024a ) . We focus on factuality hallucinations, particularly those driven by incorrect assumptions (false premises). These errors are common among LLM outputs ( Huang et al. 2025 ; Snyder et al. 2024b ) , where Figure 1 presents an example of a question with a false premise and the resulting hallucination. Figure 1: LLM experiences factuality hallucination when faced with a false premise question, where both entities ( The Lord of the Rings: The Return of the King, AACTA Award for Best Adapted Screenplay ) exist but are not correctly aligned. Many methods attempt to address false premises after an LLM has already produced an answer. They include fine-tuning the model to detect invalid assumptions ( Hu et al. 2023 ) , applying contrastive decoding to surface inconsistencies ( Shi et al. 2023 ; Chuang et al. 2024 ) , and using uncertainty-based measures or logits to gauge inaccuracies ( Pezeshkpour 2023 ; Varshney et al. 2023 ) . Although effective in some contexts, these approaches can be computationally demanding and do not necessarily prevent misinformation from appearing in the first place. Additionally, questions with false premises often maintain normal semantic flow, changing only a few tokens so that they are difficult to identify using traditional out-of-distribution detection ( Vu et al. 2023 ) . Even advanced LLMs can struggle with real-time truth evaluation, lacking the context or capacity to fully check every assumption ( Hu et al. 2023 ; Liu et al. 2024c ) . To address this challenge, we focus on preventing hallucinations rather than mitigating them post hoc. In our framework, outlined in Section 3 , we first transform the user’s query into a logical form that highlights key entities or relations (§ 3.2 ). We then employ retrieval-augmented generation (RAG) to check the accuracy of these statements against a knowledge graph (§ 3.3 ). If contradictions are found, the query is flagged as containing a false premise (§ 3.1 ), prompting the model to correct or reject the assumption before formulating a final answer. This process, shown in Figure 2 , ensures that the LLM does not rely on erroneous details during response generation. By informing the LLM about any detected false premise in advance (§ 3.4 ), we reduce the likelihood of hallucinations without requiring access to model logits or large-scale fine-tuning. We summarize our contributions as follows: Logical Form Representation : We first introduce logical forms to represent input queries and demonstrate their effectiveness across various types of graph retrievers. This logical approach enables accurate and systematic evaluation of statements provided in user prompts, particularly handling queries that may include false premises. Explicit False Premise Detection : Our method improves the reliability of LLM-generated responses by explicitly detecting false premises and informing the LLM if a question contains a false premise. Hallucination Mitigation Without Output Generation or Model Logits : Our approach reduces factual hallucinations without actual generation of responses or LLM logits and, therefore, can be seamlessly integrated into existing LLM frameworks and pipelines, offering a straightforward enhancement for improving factual accuracy. 2 Related Works False Premise . A False Premise Question (FPQ) is a question that contains incorrect facts that are not necessarily explicitly stated but might be mistakenly believed by the questioner ( Yu et al. 2022 ; Kim et al. 2021 ) . Recent studies ( Yuan et al. 2024 ) have demonstrated that FPQs can induce factuality hallucination in LLMs, as these models often respond directly to FPQs without verifying their validity. Notably, existing prompting techniques like few-shot prompting ( Brown et al. 2020 ) and Chain-of-Thought ( Wei et al. 2023 ) , tend to increase hallucinations. Conversely, directly prompting LLMs to detect false premises negatively impacts their performance on questions containing valid premises ( Vu et al. 2023 ) . Knowledge Graph Fact Checking and Question Answering . In the related field of fact checking, Retrieval-Augmented Generation (RAG) approaches are utilized for checking whether data are true and accurate. Among different RAG tasks, knowledge graph-driven RAG has drawn attention due to its ability to effectively leverage structured knowledge. Recent works include 1) prompt based : ( Pan et al. 2023 ) asks LLMs if evidence is sufficient. If not, they further ask LLMs questions critical for verifying a claim. ( Sun et al. 2024 ) uses the LLM to retrieve relevant facts hop-by-hop. 2) graph based : ( He et al. 2024 ) extracts subgraph from knowledge graph by formulating RAG over graph as Prize-Collecting Steiner Tree optimization problem. ( Mavromatis & Karypis 2024 ) uses graph neural network to reason over a dense subgraph and retrieve answer candidates for the given question. 3) training based : ( Zheng et al. 2024 ) trains two encoders for retrieval: one for embedding the queries, and one for embedding the subgraph evidences. ( Liu et al. 2024a ) trains encoder to conduct the following retrieval and ranker processes. However, it requires that entity exists in the knowledge graph and uses results generated from prompts for training. Hallucination Mitigation . Sources of LLM hallucinations originate from different stages in the LLM life cycle ( Zhang et al. 2023a ) , leading existing mitigation methods to target specific stages:

Read the complete text