PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization
Jiayi Wu, Hengyi Cai, Lingyong Yan, Hao Sun, Xiang Li, Shuaiqiang Wang, Dawei Yin, Ming Gao · 2024
Read PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization by Jiayi Wu, Hengyi Cai, Lingyong Yan, Hao Sun, Xiang Li, Shuaiqiang Wang, Dawei Yin, Ming Gao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14510.txt.
Introduction
Jiayi Wu and seven co-authors present PA-RAG in this 2024 paper, a way to tune the generator inside retrieval-augmented generation systems. Such systems fetch documents and then have a language model answer from them. General models often give thin answers, break under poor documents, or cite sources badly. Past fixes added extra steps or used plain fine tuning, yet still missed the full set of needs.
PA-RAG aligns the generator from several angles at once. The team built fine tuning data plus preference data by sampling answers of varied quality under varied document quality. They then trained the model in two stages, ending with direct preference optimization. Tests on four question answering data sets and three base models showed clear gains. The authors release their code and data. They frame the work as a full alignment of the generator with what retrieval based answering demands.
Catalog record
| CREATOR | Jiayi Wu, Hengyi Cai, Lingyong Yan, Hao Sun, Xiang Li, Shuaiqiang Wang, Dawei Yin, Ming Gao |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14510
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization Jiayi Wu, Hengyi Cai, Lingyong Yan, Hao Sun, Xiang Li, Shuaiqiang Wang, Dawei Yin, Ming Gao Abstract The emergence of Retrieval-augmented generation (RAG) has alleviated the issues of outdated and hallucinatory content in the generation of large language models (LLMs), yet it still reveals numerous limitations. When a general-purpose LLM serves as the RAG generator, it often suffers from inadequate response informativeness, response robustness, and citation quality. Past approaches to tackle these limitations, either by incorporating additional steps beyond generating responses or optimizing the generator through supervised fine-tuning (SFT), still failed to align with the RAG requirement thoroughly. Consequently, optimizing the RAG generator from multiple preference perspectives while maintaining its end-to-end LLM form remains a challenge. To bridge this gap, we propose Multiple Perspective P reference A lignment for R etrieval- A ugmented G eneration (PA-RAG), a method for optimizing the RAG generator to align with RAG requirements comprehensively. Specifically, we construct high-quality instruction fine-tuning data and multi-perspective preference data by sampling varied quality responses from the generator across different prompt documents quality scenarios. Subsequently, we optimize the generator using SFT and Direct Preference Optimization (DPO). Extensive experiments conducted on four question-answer datasets across three LLMs demonstrate that PA-RAG can significantly enhance the performance of RAG generators. Our code and datasets are available at https://github.com/wujwyi/PA-RAG . 1 Introduction Figure 1: PA-RAG retains the end-to-end form of the generator while enabling the generator to learn multi-perspective preference information. Pre-trained large language models (LLMs) demonstrate impressive question-answering capabilities but also reveal certain drawbacks, such as generating outdated or hallucinatory information ( Tonmoy et al., 2024 ; Turpin et al., 2023 ) . The emergence of retrieval-augmented generation (RAG) ( Gao et al., 2023b ) has alleviated this issue by incorporating retrieved documents relevant to the question into the prompt, thereby providing the LLMs with information they may not know. However, while RAG systems help improve factual grounding, the role of the generator in previous work is usually far from well-aligned. Despite their impressive pre-training, off-the-shelf LLMs are not optimally suited for the specific requirements of RAG tasks without proper task alignment. 1) Helpfulness for RAG application (response informativeness) ( Dong et al., 2024b ) : The generator should possess a refined ability to identify and utilize valuable information from the provided context. When presented with a mix of high-quality and less relevant information, it should actively and effectively leverage the valuable content while disregarding the rest. 2) Harmlessness for RAG application (response robustness) ( Yoran et al., 2024a ; Zhu et al., 2024 ) : A well-aligned generator should demonstrate robust noise resistance, capable of filtering out misleading information without being adversely affected. This ability ensures the generated responses remain accurate and coherent even when the input context is imperfect. 3) Honesty for RAG application (citation quality) ( Gao et al., 2023a ; Sun et al., 2023 ) : The generator should be capable of producing responses that are firmly rooted in the provided documents while maintaining appropriate attribution. This involves explicitly citing the source documents when necessary, and ensuring that the generated content is not only factually accurate but also traceable to the retrieved documents. To satisfy the aforementioned RAG requirements, existing RAG systems mainly fall into two categories: end-to-end architecture and pipeline architecture. These approaches attempt to bridge the gap between off-the-shelf LLMs and the specific demands of RAG, but both face significant limitations. End-to-end architectures primarily rely on supervised fine-tuning (SFT) of general LLMs ( Yoran et al., 2024a ; Fang et al., 2024 ) . These approaches focus on constructing high-quality responses in RAG scenarios. However, through further analysis, we found that the requirements for the generators in RAG tasks are highly context-dependent and often interwoven. These varying and intertwined demands make it challenging for a model to meet all RAG objectives through standard SFT alone, as it does not incorporate the necessary preference information required for adapting to different retrieval scenarios. On the other hand, pipeline architectures introduce additional steps beyond generation, such as re-ranking retrieved documents, filtering irrelevant information, or employing post-hoc verification to ensure that citations support the claims ( Dong et al., 2024a ; Yu et al., 2024 ; Wang et al., 2024 ; Sun et al., 2023 ) . However, these additional steps can only satisfy certain RAG requirements and still have a considerable gap in aligning with the global RAG requirements. Moreover, these additional steps introduce extra computational costs and time consumption, making it less practical for large-scale deployment. This further inspires us to think: Is it possible to fully align the generator with the diverse RAG requirements while retaining the simplicity and efficiency of an end-to-end architecture? To this end, in this paper, we propose Multi-Perspective P reference A lignment for R etrieval- A ugmented G eneration (PA-RAG), aiming to optimize the generator of RAG systems to align comprehensively with specific RAG requirements. As illustrated in Figure 1 , PA-RAG maintains the end-to-end architecture of the generator while enabling it to learn multi-perspective preference information. The training of PA-RAG is divided into two phases. The first phase is foundational capability training, where the generator acquires the basic ability to utilize and cite documents through instruction fine-tuning. To construct high-quality instruction fine-tuning data, we utilize ChatGPT to generate complete and correct answers and employ a citation rewrite mechanism to ensure citation quality. The second phase is the multi-perspective preference optimization phase, in which the generator is trained using Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) to learn preference information from different perspectives. This phase encompasses three sub-stages, sequentially enhancing the generator’s response informativeness, response robustness, and citation quality. To construct high-quality preference data, we sample responses of varying quality from the generator across scenarios with different document qualities in prompt to build preference data for informativeness and robustness. Additionally, we employ the citation rewrite mechanism to construct preference data for citation quality. We conducted extensive experiments on four QA datasets and three LLMs, demonstrating that PA-RAG significantly enhances the generator’s performance. The improvement achieved by PA-RAG far surpasses the performance gains using only SFT or additional steps. LLMs trained with PA-RAG show an average absolute improvement of 13.97% in correctness, 49.77% in citation recall, and 39.58% in citation precision. Our contributions can be summarized as follows: • We propose PA-RAG, which achieves comprehensive alignment of LLMs with specific RAG requirements through a multi-stage, multi-perspective training while preserving its end-to-end architecture. • We publicly release our training data, which includes 58.9k instruction fine-tuning data and 48.7k preference optimization data. • Through extensive experiments, we demonstrate the effectiveness of PA-RAG, showing that the preference optimization from each perspective is effective. 2 Preliminaries In this section, we describe the optimization objectives of the RAG generator, the motivation for multi-perspective optimization, and the rationale for choosing preference optimization over mere instruction fine-tuning. 2.1 Optimization Objectives of the RAG Generator As illustrated in Figure 1 , when answering a question, the RAG generator outputs the corresponding response after receiving the question and multiple relevant documents retrieved by the retriever. Our optimization goal is to enable the generator to fully utilize valuable documents and accurately cite the references