Does RAG Really Perform Bad In Long-Context Processing?

Kun Luo, Zheng Liu, Peitian Zhang, Hongjin Qian, Jun Zhao, Kang Liu · 2025

Read Does RAG Really Perform Bad In Long-Context Processing? by Kun Luo, Zheng Liu, Peitian Zhang, Hongjin Qian, Jun Zhao, Kang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11444.txt.

Introduction

Kun Luo and co-authors challenge the view that retrieval-based methods handle long documents badly. Their 2025 paper concerns a bottleneck for language models: very long inputs are slow and costly to process whole. Retrieval-augmented generation answers by fetching only the useful parts, but weak fetching and chopped context have held it back.

Their system, RetroLM, retrieves at the level of the model's memory cache, split into pages. The most vital pages are pulled forward for the next step of writing. Working at that level makes the method tougher against fetching errors, kinder to scattered context, and cheaper than recomputing from scratch. The authors argue retrieval, done deep, rivals full-context reading.

Catalog record

CREATORKun Luo, Zheng Liu, Peitian Zhang, Hongjin Qian, Jun Zhao, Kang Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11444

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Does RAG Really Perform Bad In Long-Context Processing? Kun Luo, Zheng Liu, Peitian Zhang, Hongjin Qian, Jun Zhao, Kang Liu Abstract The efficient processing of long context poses a serious challenge for large language models (LLMs). Recently, retrieval-augmented generation (RAG) has emerged as a promising strategy for this problem, as it enables LLMs to make selective use of the long context for efficient computation. However, existing RAG approaches lag behind other long-context processing methods due to inherent limitations on inaccurate retrieval and fragmented contexts. To address these challenges, we introduce RetroLM , a novel RAG framework for long-context processing. Unlike traditional methods, RetroLM employs KV-level retrieval augmentation , where it partitions the LLM’s KV cache into contiguous pages and retrieves the most crucial ones for efficient computation. This approach enhances robustness to retrieval inaccuracy, facilitates effective utilization of fragmented contexts, and saves the cost from repeated computation. Building on this framework, we further develop a specialized retriever for precise retrieval of critical pages and conduct unsupervised post-training to optimize the model’s ability to leverage retrieved information. We conduct comprehensive evaluations with a variety of benchmarks, including LongBench, InfiniteBench, and RULER, where RetroLM significantly outperforms existing long-context LLMs and efficient long-context processing methods, particularly in tasks requiring intensive reasoning or extremely long-context comprehension. 1 Introduction The processing of long contexts has emerged as a critical issue in the development and application of Large Language Models (LLMs). Numerous applications necessitate the ability to handle extended sequences of information, including understanding lengthy documents ( Bai et al. 2023 ; Caciularu et al. 2023 ) , supporting sophisticated AI agent systems ( Jin et al. 2024 ) , and generating long-form reasoning chains for complex tasks, such as mathematical proofs ( OpenAI 2024 ) or computer programming ( Gur et al. 2023 ) . To address this crucial requirement, substantial efforts have been devoted to extending the maximum context lengths accommodated by LLMs. For example, GPT-4 ( Achiam et al. 2023 ) and LLaMA-3.1 ( Dubey et al. 2024 ) , both of which support a 128K token context window. Moreover, the recent Gemini-1.5-pro ( Team et al. 2024 ) makes a dramatic extension, enabling a context window of over 10M input tokens. Despite these advancements, the naive extension of context lengths remains constrained in several aspects. One significant challenge is the dramatic rise in computation when processing long contexts. As such, efficient long-context processing techniques have attracted growing interest. For example, StreamingLLM and LM-Infinite ( Xiao et al. 2023a ; Han et al. 2023 ) maintain the most recent KVs within a sliding window alongside initial attention sinks; while SnapKV and InfLLM Li et al. 2024b ; Xiao et al. 2024 identify critical attention features for KV compression. Recently, retrieval-augmented generation ( RAG ) has emerged as a promising strategy for this problem Xu et al. 2023 ; Li et al. 2024a . These approaches leverage retrievers to extract useful context fragments from very long inputs, which effectively overcomes the limits of LLMs’ context lengths. By making selective use of the retrieved fragments, RAG further enables more efficient computation for long-context tasks. However, RAG-based methods are subject to the following inherent limitations while handling long-context tasks. 1) Retrieval Inaccuracy . Many long-context processing tasks provide no explicit queries at all, like document summarization, code completion, and in-context learning Bai et al. 2023 . As a result, traditional retrievers become inapplicable to handle corresponding problems. Besides, it’s non-trivial to properly chunk long contexts for retrieval Qian et al. 2024 , and it’s hard for retrievers to deal with zero-shot settings. Without precise and complete acquisition of useful information, LLMs will be unable to produce correct outputs through RAG. 2) Fragmented Contexts . The retrieval operation introduces fragmented token spans from the input data, which are incoherent and prone to incompleteness. This significantly prevents LLMs from making effective use of the contextual information. 3) Repeated Computation . The pre-filling operation needs to be re-conducted for the retrieved tokens of each task, both for retriever and generator, resulting in a huge waste of computation. Because of the above problems, existing RAG-based methods fall behind long-context LLMs and other efficient long-context processing approaches in many popular evaluation benchmarks Bai et al. 2023 ; Xu et al. 2023 ; Li et al. 2024a . In this paper, we propose RetroLM , a novel RAG framework designed for efficient long-context processing. RetroLM partitions the LLM’s KV cache into contiguous pages and offloads them to external storage. During both pre-filling and decoding stages, it retrieves only the most crucial pages for the current context window, enabling efficient long-context processing. Unlike traditional RAG approaches which operate on raw tokens, retrieval augmentation at the KV cache level offers several advantages. First, it is robust to retrieval inaccuracy, as useful information within a certain token span can be captured by all succeeding KVs. Second, LLMs can naturally accommodate fragmented KVs due to the inherent sparsity of LLMs’ attention patterns Jiang et al. 2024 . Third, the KV cache is computed once and reused, thus eliminating repeated computation Pope et al. 2023 . We introduce a couple of key operations to optimize the performance of RetroLM. For precise retrieval of crucial pages, we design a specialized page retriever . It estimates the pages’ importance using fine-grained KV interactions; and by fine-tuning over well-curated datasets, it achieves strong generality across various downstream tasks and a broad scope of context lengths. To make better use of fragmented KVs, we perform post-training based on unlabeled data. This further contributes to the end-to-end performance of RetroLM. We perform comprehensive evaluations using several standard benchmarks in this field, including LongBench Bai et al. 2023 , InfiniteBench Zhang et al. 2024b , and RULER Hsieh et al. 2024 . In our experiment, RetroLM outperforms popular efficient long-context processing methods with notable advantages. In majority of the tasks, it achieves an equivalent performance as the expensive full-attention methods; while for certain scenarios like long-doc QA, it even surpasses full-attention by effectively filtering out background noise and focusing on the most useful KV entries. Our well-trained models and source code will be made publicly available to facilitate future research. 2 Related Work In this section, we make discussions on the following related works: 1) context extension of LLMs, 2) efficient long-context processing, 3) RAG approaches for long-context processing. First of all, a substantial body of research has focused on extending the context length of LLMs directly. One common approach involves modifying positional encoding mechanisms to enable LLMs trained on short texts to process longer inputs directly during inference ( Chen et al. 2023a ; Peng et al. 2023 ; Ding et al. 2024 ) . While straightforward, these methods often yield suboptimal performance without additional fine-tuning. Another widely adopted strategy is continual training, where existing LLMs are fine-tuned on long-sequence data to expand their context windows ( Li et al. 2023 ; Chen et al. 2023b ; Mohtashami and Jaggi 2023 ; Xiong et al. 2023 ) . However, fine-tuning approaches typically require training from extremely long-sequence data, which is challenging due to the scarcity of native human-annotation data and the high expenses resulted from the training operations ( Fu et al.

Read the complete text