RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao · 2024

Read RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment by Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13746.txt.

Introduction

Zhuoran Jin and co-authors present a 2024 paper on how to grade the graders of retrieval-augmented chatbots. Reward models score chatbot answers and steer training toward human taste, but nobody knew which ones stay reliable when answers lean on fetched sources. Their benchmark, RAG-RewardBench, is the first built for that setting.

Four hard scenarios anchor the tests: multi-step reasoning, fine-grained citation, knowing when to abstain, and staying firm when sources conflict. The data mixes eighteen retrieval sets, six retrievers, and answers from twenty-four models. A language model judge labels preferences, matching human labels closely. Putting forty-five reward models through the bench exposed blind spots in all of them. Trained chatbots themselves also carry hidden alignment gaps.

Catalog record

CREATORZhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13746

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao Abstract Despite the significant progress made by existing retrieval augmented language models (RALMs) in providing trustworthy responses and grounding in reliable sources, they often overlook effective alignment with human preferences. In the alignment process, reward models (RMs) act as a crucial proxy for human values to guide optimization. However, it remains unclear how to evaluate and select a reliable RM for preference alignment in RALMs. To this end, we propose RAG-RewardBench , the first benchmark for evaluating RMs in RAG settings. First, we design four crucial and challenging RAG-specific scenarios to assess RMs, including multi-hop reasoning, fine-grained citation, appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG subsets, six retrievers, and 24 RALMs to increase the diversity of data sources. Finally, we adopt an LLM-as-a-judge approach to improve preference annotation efficiency and effectiveness, exhibiting a strong correlation with human annotations. Based on the RAG-RewardBench, we conduct a comprehensive evaluation of 45 RMs and uncover their limitations in RAG scenarios. Additionally, we also reveal that existing trained RALMs show almost no improvement in preference alignment, highlighting the need for a shift towards preference-aligned training. 1 Introduction Retrieval augmented generation (RAG) ( Guu et al. 2020 ; Lewis et al. 2020 ; Izacard et al. 2023 ; Asai et al. 2024b ) has emerged as a widely adopted approach for enabling large language models (LLMs) to access long-tailed and up-to-date knowledge by retrieving relevant information from external sources at inference. Existing retrieval augmented language models (RALMs) leverage RAG to address the inherent knowledge limitations of LLMs, effectively reducing factual errors Vu et al. 2024 and providing better attributions Gao et al. 2023 . Figure 1: An illustration of (a) traditional and (b) preference-aligned RAG training paradigms. A direct approach ( Ram et al. 2023 ; Shi et al. 2024 ) to building RALMs involves leveraging the in-context learning of LLMs to generate responses based on the retrieved documents. However, this plug-and-play method may cause the model to generate unfaithful responses or become distracted by noise. Recent works ( Asai et al. 2024a ; Lin et al. 2024 ; Yu et al. 2024c ) have proposed constructing specialized RAG datasets and applying supervised fine-tuning (SFT) to further increase the usability of RALMs. However, these SFT-based methods may cause RALMs to overly rely on and fit training data, lacking a feedback mechanism that enables the model to capture human preferences. As shown in Figure 1 (a), the SFT RALM may cite satirical content from the internet and generate harmful responses, or provide responses that lack sufficient information and fail to fully address the user’s needs. To better integrate human preferences like helpful and harmless Bai et al. 2022 into RALMs, we argue that RALMs should shift towards a new training paradigm, namely preference-aligned RAG training . The alignment process, as illustrated in Figure 1 (b), involves the reward model (RM) acting as a proxy for human values by providing feedback on the generated responses. Based on the signals from the reward model, preference learning algorithms, such as PPO Schulman et al. 2017 and DPO Rafailov et al. 2023 , optimize the policy model, ultimately resulting in the aligned RALM. Reward models are central to this process. However, whether they can provide high-quality reward modeling for RALMs remains underexplored. Constructing a comprehensive benchmark for reward models in RAG settings requires consideration of the following three key factors: (1) Designing well-crafted RAG scenarios : Existing benchmarks for reward models primarily focus on evaluation in general scenarios. However, in RAG scenarios, human preferences introduce new alignment requirements. For instance, privacy protection requires that RALMs must not disclose any user privacy information from the private retrieval database Zeng et al. 2024 . Additionally, users often prefer generated responses that properly attribute information to the retrieved documents; (2) Collecting diverse data sources : Data collection should encompass a wide range of diverse sources, avoiding reliance on a single domain, retriever, or RALM, to prevent any biases in the evaluation of the reward model Liu et al. 2024b ; (3) Providing high-quality preference judgments : Compared to RewardBench Lambert et al. 2024b with an average prompt length of 47, RAG needs to incorporate a much larger number of retrieved documents in the prompt. This makes it challenging for human annotators to efficiently process the long context and provide reliable preference judgments. In this paper, we propose RAG-RewardBench , a benchmark for systematically evaluating reward models in RAG settings to facilitate the alignment of RALMs. Our RAG-RewardBench is designed based on the three key factors mentioned above: (1) Beyond general helpfulness and harmlessness, we carefully design four crucial and challenging RAG-specific scenarios, including multi-hop reasoning ( i.e. , users prefer logically coherent reasoning paths, rather than inconsistent ones), fine-grained citation ( i.e. , users favour precise and relevant citations, rather than lengthy or excessive ones), appropriate abstain ( i.e. , when unable to answer with retrieved documents, actively abstaining or seeking more information is preferred over fabricating a response), and conflict robustness ( i.e. , when conflicts arise in the retrieved documents, the response should prioritize truthful facts, rather than being misled by false information). (2) To increase the diversity of data sources, we sample real-world queries from 18 subsets across different domains. To mitigate biases introduced by retrieval results, we select six retrievers, including Google Search, sparse retrieval, and dense retrieval. We adopt 24 RALMs to generate responses, ranging from open-source models (3B to 70B parameters) to commercial models ( e.g. , o1-mini, GPT-4o, Gemini-1.5-Pro, Claude 3.5 and Command R). (3) When facing the challenges of RAG’s long-context prompts, we adopt an LLM-as-a-judge approach to improve annotation efficiency and effectiveness. Specifically, we select 4 state-of-the-art commercial models to rate the responses based on carefully designed evaluation dimensions ( Ru et al. 2024 ; ES et al. 2024 ) , such as correctness, faithfulness, citation granularity, logical consistency, etc. Then, we filter out responses with inconsistent scores among judges. As a result, the preference pairs in RAG-RewardBench achieve a Pearson correlation coefficient of 0.84 with human annotations. Based on the RAG-RewardBench, we conduct a systematic evaluation of 45 reward models, including discriminative RMs ( Wang et al. 2024c ) , generative RMs ( Yuan et al. 2024 ) and implicit RMs ( Lambert et al. 2024a ) . Experimental results demonstrate that RAG-RewardBench is highly challenging, with the top-ranked RM, Skywork-Critic-Llama-3.1-70B Shiwen et al. 2024 , achieving only 78.3% accuracy. Meanwhile, in the four RAG-specific scenarios we designed, the RM’s performance decreases to varying extents, underscoring the need for specialized RMs tailored specifically for RALMs. We obtain the following meaningful conclusions: (1) RMs that perform well are generative or discriminative RMs that have been specifically trained with 27B or 70B parameters, whereas those implicit RMs tend to perform poorly on RAG-RewardBench. (2) Although state-of-the-art trained RALMs ( Asai et al. 2024a ; Liu et al. 2024c ) demonstrate significant improvements on certain RAG datasets, their performance on RAG-RewardBench shows only a minimal gain of 0.6% compared to the original LLMs. This suggests that the RALM

Read the complete text