Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models

Alex Laitenberger, Christopher D. Manning, Nelson F. Liu · 2025

Read Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models by Alex Laitenberger, Christopher D. Manning, Nelson F. Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03989.txt.

Introduction

Alex Laitenberger, Christopher Manning, and Nelson Liu ask in 2025 if long window chat models still gain from staged fetch write chains. Long windows now hold tens of thou sands of words at once. That gain puts the worth of chain steps in real doubt.

Two staged chains face three plain picks, one a fetch then read form that keeps the source's own page order. All run at matched word caps. The plain pick tied or beat the chains on long ask tests time and time again. The team lays the win on whole source shape, full recall in the window, and plain form. They back it as the stock test base.

Catalog record

CREATORAlex Laitenberger, Christopher D. Manning, Nelson F. Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03989

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models Alex Laitenberger, Christopher D. Manning, Nelson F. Liu Abstract With the rise of long-context language models (LMs) capable of processing tens of thousands of tokens in a single pass, do multi-stage retrieval-augmented generation (RAG) pipelines still offer measurable benefits over simpler, single-stage approaches? To assess this question, we conduct a controlled evaluation for QA tasks under systematically scaled token budgets, comparing two recent multi-stage pipelines, ReadAgent and RAPTOR, against three baselines, including DOS RAG ( D ocument’s O riginal S tructure RAG ), a simple retrieve-then-read method that preserves original passage order. Despite its straightforward design, DOS RAG consistently matches or outperforms more intricate methods on multiple long-context QA benchmarks. We recommend establishing DOS RAG as a simple yet strong baseline for future RAG evaluations, pairing it with emerging embedding and language models to assess trade-offs between complexity and effectiveness as model capabilities evolve. 1 1 1 We release our code at https://github.com/alex-laitenberger/stronger-baselines-rag . Figure 1: \infty Bench En.MC performance of various multi-stage RAG systems and long-context baselines (mean \pm standard deviation over five runs). All methods use GPT-4o as the underlying reader. For token budgets greater than 5K, DOS RAG outperforms the complex multi-stage retrieval methods (ReadAgent and RAPTOR) by 2-8 points. 1 Introduction Recent advances in long-context language models (LMs), such as GPT-4o, have expanded the token processing capabilities of LMs, enabling them to handle tens of thousands of tokens in a single pass. This raises a pivotal question: Are complex, multi-stage retrieval-augmented generation (RAG) pipelines still necessary when simpler, single-stage methods can now leverage these extended contexts effectively? Retrieval-augmented generation (RAG) systems traditionally combine a retriever , which selects passages from a large corpus relevant to a given query, and a reader , typically a language model, to generate a final answer. ( Lewis et al. 2020 ) . Prior work has proposed a variety of retrieval strategies to circumvent the limited long-context reasoning ability of earlier reader LMs. For example, abstractive preprocessing, iterative chunk summarization, and agent-based retrieval loops have been used to compress or reason over texts that might otherwise exceed the input limits of early LMs ( Chen et al. 2023 ; Sarthi et al. 2024 ; Lee et al. 2024 ; Sun et al. 2024 , inter alia ) . While effective, these pipelines often introduce significant complexity and computational overhead. In contrast, modern long-context LMs can now process substantial input sizes directly, suggesting that simpler retrieve-then-read strategies might suffice in certain settings. To assess the relevance of multi-stage pipelines under these new conditions, we conduct a controlled evaluation in which we systematically increase token budgets, analyzing how effectively each approach leverages extended contexts using a modern long-context LM (GPT-4o) as the downstream reader. We compare two recent multi-stage pipelines (ReadAgent and RAPTOR; Lee et al. 2024 ; Sarthi et al. 2024 ) against three baselines, including DOS RAG ( D ocument’s O riginal S tructure RAG ). DOS RAG maintains a simple retrieve-then-read strategy and presents retrieved passages in their original document order. Despite its simplicity, our findings across three QA benchmarks ( \infty Bench, QuALITY, NarrativeQA) indicate that DOS RAG can match or even outperform more complex multi-stage pipelines on all evaluated retrieval token budgets (Figure 1 , Figure 2 , Figure 3 ). This work advocates for establishing DOS RAG as a simple yet strong baseline for RAG evaluations, pairing it with emerging embedding and language models to benchmark trade-offs between complexity and effectiveness as model capabilities continue to evolve. 2 Experimental Setup We compare the performance of two recent multi-stage RAG pipelines (ReadAgent and RAPTOR) against three baselines (Vanilla RAG, the full-document baseline, and DOS RAG) on three long-context question-answering tasks ( \infty Bench, QuALITY and NarrativeQA). See Appendix A for further details about experimental setup, implementation, and used prompts. 2.1 Benchmarks \infty Bench. We evaluate systems on the English multiple-choice (En.MC) subset of \infty Bench ( Zhang et al. 2024 ) . The benchmark contains 229 multiple-choice questions on 58 documents (average length of 184K tokens). QuALITY. We use the QuALITY benchmark ( Pang et al. 2022 ) , a multiple-choice question-answering dataset over English context passages containing between 2K to 8K tokens. We evaluate systems on the development set, which contains 115 documents and 2,086 questions. NarrativeQA. The NarrativeQA benchmark is a long-document question-answering dataset in English that challenges models to answer questions about stories by reading entire books or movie scripts ( Kočiský et al. 2018 ) . We evaluate on the test set, which contains 355 stories (average length of 57K tokens) and 10,557 questions. Each story’s questions are constructed such that high performance requires understanding the underlying narrative, versus relying on shallow pattern matching. 2.2 Multi-Stage RAG Pipelines ReadAgent. ReadAgent handles long input contexts with a method inspired by human reading strategies ( Lee et al. 2024 ) . Concretely, ReadAgent prompts the LM with three steps: (1) episode pagination, where the LM forms a sequence of pages by identifying natural breakpoints in the text; (2) memory gisting, which compresses the content of each page into shorter “gist” summaries; and (3) interactive look-up, where the LM uses the query and the gists to identify pages to re-read and use to solve the final query. This approach extends the language model’s context window by offloading the document’s full detail into a page-wise gisted memory, retrieving original text only when needed. RAPTOR. RAPTOR handles long documents by recursively organizing the text into a tree of hierarchical summaries ( Sarthi et al. 2024 ) . Concretely, it partitions the text into sentence-level chunks, clusters related chunks, and uses a language model to summarize each cluster. This process repeats, generating higher-level summaries until a final set of root nodes represents the entire document. At inference time, RAPTOR retrieves from different levels of the summary tree, balancing broad coverage against local detail. 2.3 Baselines Our three baselines are designed to benefit from and scale with stronger language models with improved long-context reasoning abilities. Vanilla RAG. In our implementation the corpus is first split into chunks capped at 100 tokens, while preserving sentence boundaries where possible. We use neural retrieval with a sentence embedding model (Snowflake Arctic-embed-m 1.5 by Merrick 2024 ) to encode both the query and the resulting text chunks into a shared embedding space. At inference time, chunks are ranked by cosine similarity to the query embedding, with the top-ranked chunks retrieved until a fixed input token budget (e.g., 10K tokens) is reached. The selected chunks, ordered by decreasing similarity, are then concatenated with the query to construct the input to the language model. Full-Document Baseline. The full-document baseline prompts the model with all available text, preserving passage continuity and reducing the risk of missing key information through retrieval errors. We evaluate it only on the QuALITY benchmark, where document lengths fit within the model’s context window. Using the D ocument’s O riginal S tructure (DOS RAG). DOS RAG follows the same retrieval and embedding process as Vanilla RAG, but with one key difference: retrieved chunks are reordered to match their original document order, not sorted by similarity score. This reordering, achieved by tracking chunk positions, preserves passage

Read the complete text