Removal of Hallucination on Hallucination: Debate-Augmented RAG

Wentao Hu, Wengyu Zhang, Yiyang Jiang, Chen Jason Zhang, Xiaoyong Wei, Qing Li · 2025

Read Removal of Hallucination on Hallucination: Debate-Augmented RAG by Wentao Hu, Wengyu Zhang, Yiyang Jiang, Chen Jason Zhang, Xiaoyong Wei, Qing Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18581.txt.

Introduction

Wentao Hu and co-authors wrote this machine learning paper, published in two thousand twenty-five, on a failure mode of retrieval-augmented generation. Such systems fetch facts before writing, but bad fetches can mislead the writer and stack new errors on old ones, a loop the authors name hallucination on hallucination.

Their fix, Debate-Augmented RAG, makes software agents argue the case at both stages. In retrieval, backers, skeptics, and a judge hash out which documents to trust. In writing, agents hold uneven information and debate, which hardens the reasoning. No training is needed; the debate runs on top of existing models. Across several tasks, the method improved retrieval quality and cut the stacked errors. Factual accuracy rose overall. The team has released its code in public.

Catalog record

CREATORWentao Hu, Wengyu Zhang, Yiyang Jiang, Chen Jason Zhang, Xiaoyong Wei, Qing Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18581

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Removal of Hallucination on Hallucination: Debate-Augmented RAG Wentao Hu, Wengyu Zhang, Yiyang Jiang, Chen Jason Zhang, Xiaoyong Wei, Qing Li Abstract Retrieval-Augmented Generation (RAG) enhances factual accuracy by integrating external knowledge, yet it introduces a critical issue: erroneous or biased retrieval can mislead generation, compounding hallucinations, a phenomenon we term Hallucination on Hallucination . To address this, we propose D ebate-Augmented RAG ( DRAG ), a training-free framework that integrates Multi-Agent Debate (MAD) mechanisms into both retrieval and generation stages. In retrieval, DRAG employs structured debates among proponents, opponents, and judges to refine retrieval quality and ensure factual reliability. In generation, DRAG introduces asymmetric information roles and adversarial debates, enhancing reasoning robustness and mitigating factual inconsistencies. Evaluations across multiple tasks demonstrate that DRAG improves retrieval reliability, reduces RAG-induced hallucinations, and significantly enhances overall factual accuracy. Our code is available at https://github.com/Huenao/Debate-Augmented-RAG . 1 Introduction Large Language Models (LLMs) have demonstrated remarkable natural language understanding and reasoning capabilities Achiam et al. 2023 ; Touvron et al. 2023 . However, their reliance on parametric knowledge introduces a critical challenge, hallucination, where the generated content deviates from factual correctness Ji et al. 2023 ; Huang et al. 2024 . This issue severely limits their reliability, particularly in knowledge-intensive tasks where factual accuracy is paramount. Figure 1: Demonstration of Hallucination in Retrieval-Augmented Generation. In the first example, the error stems from retrieving information about a movie with the same name instead of the correct entity (a band). In the second, despite retrieving accurate information, retrieval noise still leads to an incorrect response. To mitigate hallucinations, Retrieval-Augmented Generation (RAG) has been proposed as a framework that integrates external knowledge retrieval to enhance LLM outputs Gao et al. 2023 . By conditioning responses on retrieved documents, RAG reduces reliance on the model’s parametric knowledge, thereby aiming to improve factual correctness. However, as shown in Figure 1 , RAG introduces a new challenge: Biased or erroneous retrieval results can mislead the generation, compounding the hallucination problem rather than solving it . In other words, the LLM’s inherent hallucinations are further amplified when retrieval provides incomplete, biased, or misleading information, leading to what we term Hallucination on Hallucination . Addressing this challenge requires optimizing the entire RAG pipeline, encompassing both retrieval and generation stages. In the retrieval stage, hallucinations frequently arise from insufficient retrieval, where incomplete or biased results mislead the generation phase. Even when retrieval strategies are optimized, the generation stage remains vulnerable to retrieval noise and misinformation Ngo et al. 2007 , which can propagate hallucinations rather than mitigate them. Existing approaches to reducing retrieval bias include iterative retrieval Trivedi et al. 2023 ; Shao et al. 2023 ; Feng et al. 2024 and autonomous retrieval Jiang et al. 2023 ; Su et al. 2024 ; Yu et al. 2024 ; Wang et al. 2025 , both of which seek to counteract hallucination stemming from knowledge deficiency and retrieval noise. Additionally, reflection during generation Asai et al. 2023 and retrieval summarization Kim et al. 2024 have been proposed to enhance the robustness and factual consistency of generation. Existing methods do not systematically optimize the entire pipeline. These methods also rely on LLMs for self-decision-making and self-optimization, making them susceptible to inherent biases, which limits their ability to generate diverse queries and robustness Wei and Ngo 2008 . Recent advancements in Multi-Agent Debate (MAD) offer a promising paradigm for enhancing LLM robustness, factual accuracy, and reasoning diversity Becker 2024 . MAD introduces multiple independent LLM agents that engage in structured debates, iteratively refining their responses through critical evaluation and multi-agent verification. This process improves factual consistency and enhances robustness in reasoning Du et al. 2024 . Furthermore, MAD fosters perspective diversity by assigning distinct roles to agents, thereby mitigating biases that arise in single-agent approaches Yin et al. 2023 ; Liang et al. 2024 . Motivated by this, we leverage MAD as a mechanism to address the challenge in RAG and propose D ebate Augmented RAG ( DRAG ). DRAG is a novel training-free framework integrating multi-agent debate mechanisms across retrieval and generation phases. In the retrieval phase, multiple agents engage in structured debates to assess retrieval adequacy. DRAG maintains a dynamic query pool by incorporating proponents advocating retrieval strategies, opponents questioning query sufficiency, and judges evaluating completeness, ensuring broader knowledge coverage and reduced factual biases. In the generation phase, we establish asymmetric information roles among agents, reducing the LLM’s over-reliance on retrieved content while promoting structured adversarial debate to enhance reasoning robustness and mitigate factual inconsistencies. We evaluate DRAG on multiple tasks and demonstrate that this structured debate process leads to more reliable retrieval, enhanced reasoning robustness, and substantially reduced RAG-induced hallucinations. Overall, our main contributions can be summarized as follows: • We propose a novel perspective on the challenges in RAG: Hallucination on Hallucination . • We introduce DRAG, a novel training-free framework that integrates Multi-Agent Debate (MAD) mechanisms into both retrieval and generation phases of RAG. • We evaluate DRAG on multiple tasks, demonstrating its effectiveness in improving retrieval reliability, reasoning robustness, and reducing RAG-induced hallucinations. 2 Related Work 2.1 Retrieval-Augmented Generation LLMs have demonstrated remarkable success across a wide range of tasks. However, they are prone to hallucinations, particularly when handling queries beyond their training data, such as in domain-specific or knowledge-intensive tasks. To mitigate this issue, Retrieval-Augmented Generation (RAG) enhances LLMs by incorporating external knowledge retrieval into the response generation process Gao et al. 2023 . The most straightforward approach involves using the initial input as a query to retrieve information from an external corpus, which is then integrated into the model’s input as supplementary knowledge Guu et al. 2020 ; Lewis et al. 2020 ; Izacard and Grave 2021 ; Izacard et al. 2022 ; Shi et al. 2024 . However, single-round retrieval based solely on the initial input may fail to retrieve essential external knowledge for complex tasks and introduce harmful noise. To meet complex retrieval needs, IRCoT Trivedi et al. 2023 and ITER-RETGEN Shao et al. 2023 introduce iterative retrieval-generation loops, allowing LLMs to refine their retrieval strategy dynamically to address insufficient retrieval. Auto-RAG Yu et al. 2024 further extends this by enabling autonomous query decision-making, enhancing retrieval flexibility. FLARE Jiang et al. 2023 and DRAGIN Su et al. 2024 adopt a token-triggered retrieval mechanism, activating retrieval only when necessary and improving efficiency. However, focusing on local tokens may neglect global reasoning needs. In contrast, Auto-RAG leverages LLM reasoning to optimize queries autonomously. Self-RAG Asai et al. 2023 autonomously retrieves via reflection and follows a reflection-retrieval-regeneration process to improve factual consistency between generated and retrieved content. RECOMP Xu et al. 2023 and SuRe Kim et al. 2024 optimize LLM generation through summarized retrieval, enhancing the utilization of external information, reducing information overload, and improving factual consistency. Existing methods lack a comprehensive analysis and optimization of RAG. Moreover, relying on a single-agent decision-making and reasoning process may introduce biases that impact performance. 2.2 Multi-Agent Debate Recent studies have demonstrated that multi-agent interactions significantly enhance LLM capabilities across various dimensions Becker 2024 . EoT Yin et al. 2023 mitigates biases and blind spots inherent in single-agent approaches by integrating external insights from other agents. Moreover, Multi-Agent Debate (MAD) enables agents to

Read the complete text