Pref RAG : Preference-Driven Multi-Source Retrieval Augmented Generation

Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Jie Tang · 2024

Read Pref RAG : Preference-Driven Multi-Source Retrieval Augmented Generation by Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Jie Tang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00689.txt.

Description

Abstract Retrieval-Augmented Generation (RAG) has emerged as a reliable external knowledge augmentation technique to mitigate hallucination issues and parameterized knowledge limitations in Large Language Models (LLMs). Existing adaptive RAG (ARAG) systems excel at in-depth exploration within a single source but struggle to effectively and controllably explore different retrieval sources, as they fail to foresee their internal knowledge features.

Catalog record

CREATORQingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Jie Tang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00689

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pref RAG : Preference-Driven Multi-Source Retrieval Augmented Generation Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Jie Tang Abstract Retrieval-Augmented Generation (RAG) has emerged as a reliable external knowledge augmentation technique to mitigate hallucination issues and parameterized knowledge limitations in Large Language Models (LLMs). Existing adaptive RAG (ARAG) systems excel at in-depth exploration within a single source but struggle to effectively and controllably explore different retrieval sources, as they fail to foresee their internal knowledge features. We develop a novel multi-source ARAG system, PrefRAG, which enhances RAG by enabling in-depth and controllable exploration of diverse retrieval sources through preference-driven adaptive retrieval and self-reflection. PrefRAG first fully explores controllable local sources in adaptive retrieval and supplements with the web when appropriate, ultimately selecting the optimal source for knowledge observation. Subsequently, PrefRAG feeds answer quality feedback into the retrieval process, optimizing it from the generation perspective to produce higher-quality responses. Extensive experiments confirm its superiority, high retrieval efficiency, and knowledge controllability. PrefRAG outperforms Vanilla RAG and the leading MS-ARAG by up to 25.6% and 13.9% respectively. Additionally, PrefRAG trained with DPO achieves higher performance. The code and data are available at https://github.com/QingFei1/PrefRAG.git . 1 Introduction In the question answering (QA) task ( Kwiatkowski et al., 2019 ; Rajpurkar et al., 2016 ), even the leading Large Language Models (LLMs) ( OpenAI, 2023 ; Zeng et al., 2024 ; Touvron et al., 2023 ) are restricted by the scope of their parametric knowledge and struggle with hallucination ( Chen et al., 2023 ) and insufficient knowledge ( Kandpal et al., 2023 ) . Figure 1: Comparison of Different Methods. Single-source adaptive RAG enables in-depth exploration but cannot integrate cross-source knowledge. PrefRAG addresses this limitation by enabling efficient and adaptive exploration of different retrieval resources. Retrieval-Augmented Generation (RAG) ( Lewis et al., 2020 ) serves as a powerful technique that mitigates these challenges by supplementing external knowledge with a non-parametric form, generating high-quality and reliable answers. Mainstream retrieval sources for RAG typically include local retrieval sources, e.g., Wikipedia corpus ( Izacard et al., 2023 ) or web retrieval sources, e.g., Bing, each with distinct data characteristics ( Williams, 2000 ) . Generally, local retrieval sources are carefully curated, highly structured, and offer greater control and security due to their on-premise storage. In contrast, web-based retrieval sources provide large-scale, diverse, and real-time information but are inherently less controllable. These differences indicate that each retrieval source has its own advantages and limitations. A pilot study conducted on a multi-hop dataset ( Ho et al., 2020 ) , as illustrated in Fig. 1 , reveals that knowledge from local and web sources can be mutually reinforcing, leading to enhanced performance. However, existing RAG remain underdeveloped in their ability to effectively and controllably leverage multiple retrieval sources with distinct characteristics. As depicted in Fig. 1 , Adaptive RAG (ARAG) ( Jiang et al., 2023 ; Jeong et al., 2024 ) typically focus on exploring a single retrieval source (either local or web) in depth, overlooking the complementary contributions of multiple sources. Recently, an LLM-based agent paradigm, ReAct ( Yao et al., 2023 ) can be instantiated as Multi-Source ARAG (MS-ARAG) and allow retrieval from multiple sources throughout the iterative process. However, ReAct struggles to foresee the data characteristics in different retrieval sources before retrieval. Its source selection decision relies on retrieval source descriptions and the model’s internal parameterized representation, which may fail to align with the real retrieval demands. Another direct strategy for leveraging diverse sources is concatenating knowledge from different sources. This strategy risks direct exposure of problematic web content to the LLM, potentially generating undesirable outputs and requiring more retrieval counts. To bridge these gaps, we develop PrefRAG, a novel MS-ARAG system designed for efficient, controlled, and adaptive exploration of retrieval sources with diverse characteristics. As illustrated in Fig. 2 , PrefRAG consists of two core processes: preference-driven adaptive retrieval (Pref-AR) and self-reflection. During the Pref-AR process, the LLM decides whether to retrieve and what to retrieve based on the original query and accumulated context, enabling adaptive retrieval. Once a retrieval action is determined, we retrieve the preset preferred source (e.g., the local source) and then guide the LLM to analyze the retrieved knowledge before deciding whether to switch to another source (e.g., the web source). This enables the system to conduct in-depth knowledge analysis and make well-considered retrieval source decisions. Moreover, such an orderly retrieval process transitioning from the relatively controlled local source to the web source helps minimize the risk of exposing the LLM to uncontrolled knowledge from the web when local retrieval suffices. During the self-reflection process, the LLM assesses the reliability of responses and provides specific improvement suggestions through self-feedback ( Madaan et al., 2023 ; Shinn et al., 2023 ), thereby guiding subsequent retrieval and reasoning processes to enhance the final response quality. To summarize, our main contributions are as follows: 1) We develop a novel MS-ARAG system with preference-driven adaptive retrieval and self-reflection mechanisms. The system leverages preference constraints to guide the RAG system in selecting appropriate retrieval sources and refines subsequent retrieval through self-reflection, enabling deep and controllable knowledge utilization from diverse retrieval sources to generate high-quality answers. 2) We propose an automated pipeline for constructing preference-driven retrieval training data, which generates high-quality data for Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) fine-tuning, further enhancing the system’s capability. 3) Extensive empirical studies conducted on four datasets demonstrate the effectiveness of PrefRAG. Experimental results show that our method significantly outperforms Vanilla RAG (by up to 25.6%) and the leading MS-RAG (by up to 13.9%) while maintaining high retrieval efficiency. In real-world applications, we further validate the superior performance of PrefRAG in controllable knowledge retrieval. 2 Related Work Knowledge Source Exploration for RAG. In the era of LLM, RAG ( Lewis et al., 2020 ; Guu et al., 2020 ) builds on the versatile LLM as a foundation and serves as a bridge between external knowledge and the model’s internal parameterized knowledge by following the " Retriever-and-Reader " paradigm ( Chen et al., 2017 ; Das et al., 2019 ). For various downstream tasks ( Zhu et al., 2021 ; Zhou et al., 2023 ; Cai et al., 2019 ), RAG systems retrieve accessible sources as comprehensively as possible to enhance generation, especially for knowledge-intensive question answering task ( Khattab et al., 2022 ) . In terms of the manner of retrieval sources, recent advanced RAG research can be divided into two categories. One line of study conducts in-depth exploration within a single retrieval source, referred to as Single-Source RAG (SS-RAG). It primarily includes multi-step RAG methods ( Trivedi et al., 2023 ; Ram et al., 2023 ; Borgeaud et al., 2022 ) that use subqueries for iterative retrieval and ARAG methods ( Yao et al., 2023 ; Asai et al., 2024 ; Dhole, 2025 ) that flexibly determine " when and what to retrieve " for a more adaptive and in-depth retrieval process. For Single-Source ARAG (SS-ARAG), the limitation of a single retrieval source imposes an upper bound on the capability of the RAG system. Another line of research focuses on Multi-Source RAG (MS-RAG). CRAG ( Yan et al., 2024 ) uses the web as

Read the complete text