OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learning
Jiawei Zhou, Lei Chen · 2025
Read OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learning by Jiawei Zhou, Lei Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08398.txt.
Introduction
Jiawei Zhou and Lei Chen present OpenRAG in a computing paper from 2025. Many chat systems first search a store of texts and then write an answer from what they find. The search step is trained in the usual way, yet the team shows that such scores can misfit what the answer step needs.
OpenRAG tunes the search step end to end for its role in context. Across many tasks, the tuned search beat the base search by four points on their scale. It also beat top rival tools. A small tuned model even topped gains from much larger ones. The lesson is that cheap tuning can lift a whole system.
Catalog record
| CREATOR | Jiawei Zhou, Lei Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08398
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learning Jiawei Zhou, Lei Chen Abstract In this paper, we analyze and empirically show that the learned relevance for conventional information retrieval (IR) scenarios may be inconsistent in retrieval-augmented generation (RAG) scenarios. To bridge this gap, we introduce Open-Rag , a RAG framework that is OP timized EN d-to-end by tuning the retriever to capture in-context relevance, enabling adaptation to the diverse and evolving needs. Extensive experiments across a wide range of tasks demonstrate that Open-Rag , by tuning a retriever end-to-end, leads to a consistent improvement of 4.0% over the original retriever, consistently outperforming existing state-of-the-art retrievers by 2.1%. Additionally, our results indicate that for some tasks, an end-to-end tuned 0.2B retriever can achieve improvements that surpass those of RAG-oriented or instruction-tuned 8B large language models (LLMs), highlighting the cost-effectiveness of our approach in enhancing RAG systems. \lst@InstallKeywords kattributesattributestyle attributestyleld 1 Hong Kong University of Science and Technology 1 Introduction As large language models (LLMs) ( Zhao et al., 2023 ; Minaee et al., 2024 ) scale, they face a data bottleneck where the high-quality internet data unable to meet growing training demands. Meanwhile, the volume of downstream data is expanding rapidly but often remains unusable for pre-training due to their real-time availability ( Wang et al., 2024b ; Liu et al., 2023 ) , privacy concerns ( Arora et al., 2023 ) , licensing restrictions ( Min et al., 2024 ) , and ethical concern ( Serouis & Sèdes, 2024 ; Ayyamperumal & Ge, 2024 ) . Retrieval-augmented generation (RAG) ( Lewis et al., 2020 ; Guu et al., 2020 ; Gao et al., 2023 ) emerges as a promising solution to this challenge. Rather than relying solely on well-curated internet data, RAG leverages information retrieval (IR) to fetch relevant data from external sources and incorporates it as context to enhance generation quality. This is valuable as RAG enables the use of rapidly expanding yet often inaccessible downstream data, which are more scalable and up-to-date than the heavily processed and regulated internet data used in pre-training. Figure 1 : Comparison of query-document relevance in IR scenario and RAG scenario. Despite their success, existing RAG frameworks typically rely on off-the-shelf retrievers trained on QA datasets, which can lead to inconsistencies between the learned retrieval relevance and the needs of downstream tasks. This discrepancy highlights key relevance gaps between IR and RAG scenarios. We explore these gaps in detail below, drawing on insights from prior research. First, there is the broadening of tasks : traditional IR datasets ( Kwiatkowski et al., 2019 ; Bajaj et al., 2016 ) are designed mainly for open-domain question-answering (OpenQA), while RAG framework are applied to a wider range of tasks, such as recommendation ( Manzoor & Jannach, 2022 ) , dialog systems ( Liu et al., 2024 ) , and role-playing ( Wang et al., 2023 ) , where task requirements can be flexibly written as instructions. We refer to relevance in these two cases as QA relevance and in-context relevance , respectively, as shown in Figure 1 . Second, the role of retrieved documents has shifted: in IR, retrieved documents are the final output provided to users, whereas in RAG, they are fed into the LLM to generate a response. Recent studies ( Cuconasu et al., 2024a ; Cuconasu et al., 2024b ; Wu et al., 2024 ) have shown that including more answer-containing documents, which align with QA relevance in IR scenarios, can harm RAG performance, while documents without direct answers may actually help. These findings challenge traditional IR assumptions in the RAG setting. Finally, the complexity of queries has increased: unlike traditional IR, where queries are typically simple questions, RAG queries tend to be more diverse and noisy, reflecting varying levels of task complexity. Several studies highlight the challenges of complex queries and suggest that refining queries ( Chan et al., 2024 ) or generating task-specific queries ( Wu & Cao, 2024 ; Koo et al., 2024 ) based on documents can significantly enhance RAG performance. To address this gap, we introduce Open-Rag , a RAG framework that is OP timized EN d-to-end by tuning the retriever to capture in-context relevance. Unlike existing retrievers, which are constrained to training on specific corpora and tasks with human annotations provided, our framework is OPEN to training on any task, with any corpus and any LLM. During training, Open-Rag retrieves documents on-the-fly and identifies them as positives or negatives for contrastive learning.To reduce training costs, we use approximation techniques to bypass the autoregressive generation process and employ semi-parametric retrieval to avoid the need for re-indexing. Our training requires only four GPUs and can be completed within a day. Extensive experiments demonstrate that our method leads to significant improvements, consistently outperforming state-of-the-art (SOTA) retrievers. For certain tasks, our improvements surpass those achieved by tuning an 8B LLM, showcasing that end-to-end retrieval learning is a cost-effective approach for enhancing RAG systems. Our contribution can be summarized as follows: \bullet We investigate the relevance gap between IR and RAG scenarios, providing empirical evidence of when and how this gap negatively impacts RAG performance. \bullet Through our experiments, we identify potential biases in prior research that may impede progress in this field. These findings provide critical insights to guide future research directions. \bullet We introduce Open-Rag , an end-to-end optimized RAG framework that learns in-context retrieval for various downstream tasks without requiring query-document human annotations, facilitating broader real-world deployment and applications. \bullet Extensive experiments show that Open-Rag achieves superior performance across diverse tasks compared to RAG systems using SOTA retrievers or fine-tuned LLMs, underscoring its effectiveness as a reliable and versatile solution for improving RAG systems. 2 Preliminary 2.1 Transferring from IR to RAG Scenarios In Table 1 , we examine the performance of off-the-shelf retrievers across different datasets in IR and RAG scenarios. Details about the datasets and retrievers can be found in Appendix A and B , while the evaluation metric is described in Section 4.1 . Key findings are summarized below. Table 1 : Accuracy in IR and RAG scenarios using Llama3-8b with top-1 retrieved document in-context; Bold : best performance; \Delta : improvement or decline compared to \textsc{SiDR}_{\textsc{MS}} ; \lx@sectionsign : has accessed the training split of the dataset. Dataset ( \rightarrow ) NQ TriviaQA PubHealth ARC-C Retriever ( \downarrow ) IR \Delta RAG \Delta IR \Delta RAG \Delta RAG \Delta RAG \Delta Unsupervised Pre-training Contriever 23.6 -15.5 30.9 -3.5 37.2 -18.9 56.6 -5.4 61.8 -1.7 58.6 +1.7 E5-unsup 30.8 -8.3 33.4 -1.0 39.5 -16.6 54.3 -7.7 62.9 -0.6 58.3 +1.4 Supervised on MSMARCO DPR {}_{\text{MS}} 38.9 -0.2 34.9 +0.5 43.7 -12.4 55.2 -6.8 64.5 +1.0 56.3 -0.6 SiDR {}_{\text{MS}} 39.1 – 34.4 – 56.1 – 62.0 – 63.5 – 56.9 – Supervised on NQ DPR {}_{\text{NQ}} \ddagger 43.5 +4.4 \ddagger 38.5 +4.1 39.4 -16.7 55.9 -6.1 62.9 -0.6 56.6 -0.3 SiDR {}_{\text{NQ}} \ddagger 49.5 +10.4 \ddagger 42.7 +8.3 47.4 -8.7 59.8 -2.2 63.5 – 57.1 +0.2 Supervised on TQA DPR {}_{\text{TQA}} 32.1 -7.0 32.9 -1.5 \ddagger 55.4 -0.7 \ddagger 61.1 -0.9 63.1 -0.4 56.7 -0.2 SiDR {}_{\text{TQA}} 30.6 -8.5 32.9 -1.5 \ddagger 56.9 +0.8 \ddagger 63.6 +1.6 61.1 -2.4 58.6 +1.7 Pre-training + Supervised on Multiple