Predictive Prefetching for Retrieval-Augmented Generation

Wuyang Zhang, Shichao Pei · 2026

Read Predictive Prefetching for Retrieval-Augmented Generation by Wuyang Zhang, Shichao Pei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17989.txt.

Introduction

Wuyang Zhang and Shichao Pei propose predictive prefetching for retrieval-augmented generation. That grounding helps language models but suffers latency from sync retrieval. Async methods rely on heuristic coordination and assume stable information demands during decoding. Their framework predicts when to retrieve and what to retrieve, using a predictor, a monitor, and a query maker. It exploits early signs that appear several tokens before uncertainty turns critical. Tests show up to 43.5% lower overall latency and 62.4% better first-token time, with quality matching baselines.

Catalog record

CREATORWuyang Zhang, Shichao Pei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99151
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17989

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Predictive Prefetching for Retrieval-Augmented Generation Wuyang Zhang, Shichao Pei Abstract Retrieval-Augmented Generation (RAG) improves factual grounding in large language models but suffers from substantial latency due to synchronous retrieval. While recent work explores asynchronous retrieval, existing approaches rely on heuristic coordination between retrieval and generation and assume stable information demands during decoding that often break in complex, multi-domain settings. In this paper, we propose an advanced asynchronous retrieval framework that enables predictive prefetching aligned with evolving information needs. The framework explicitly predicts when retrieval should be triggered and what information should be retrieved using three components, a retrieval predictor, a context monitor, and a query generator, by exploiting semantic precursors in generation dynamics that emerge several tokens before uncertainty becomes critical. Experiments on multiple benchmarks demonstrate up to 43.5% end-to-end latency reduction and 62.4% improvement in time-to-first-token, while maintaining answer quality comparable to synchronous RAG baselines. 1 Introduction Retrieval-Augmented Generation (RAG) has become the dominant approach for grounding large language model (LLM) outputs in factual and up-to-date information ( 21 ; 8 ; 3 ) . By augmenting static model parameters with external knowledge, RAG effectively mitigates hallucination and knowledge staleness. However, the latency overhead introduced by retrieval operations remains a major barrier to production deployment and significantly degrades user experience. Complex queries often require multiple sequential retrieval steps ( 1 ; 13 ) , particularly deep research workloads ( 43 ; 36 ) may trigger hundreds of retrievals within a single query. Moreover, in high-stakes applications such as financial analysis, news summarization, and emergency response, retrieval frequently relies on external data sources rather than pre-indexed local collections. These external calls typically incur 100–500 ms latency per retrieval due to network round trips and API rate limits, causing the cumulative overhead to become prohibitive. The core architectural limitation stems from the synchronous nature of current RAG designs. When uncertainty triggers a retrieval, token generation is fully suspended until the retrieval completes ( 20 ) . This reactive behavior creates a fundamental conflict between generation quality and system performance. Applications that demand high factual accuracy must tolerate multiple retrieval rounds and cumulative delays, whereas latency-sensitive deployments are forced to limit retrieval depth, inevitably sacrificing the completeness and reliability of retrieved evidence. Despite the growing number of advanced retrieval mechanisms, little attention has been devoted to addressing this fundamental architectural limitation. Recent concurrent work has begun to explore asynchronous retrieval strategies ( 22 ; 37 ; 12 ) . However, these approaches largely depend on manually designed heuristics to coordinate retrieval and generation, such as triggering retrieval at fixed time intervals or using stale tokens as queries, assuming relatively stable information demands during generation. While this assumption holds in scenarios with high query–document similarity, it becomes fragile in complex multi-domain settings, where topic boundaries blur, and entity references evolve mid-generation. As a result, prefetching may introduce irrelevant context, undermining both generation efficiency and factual reliability. In such cases, current asynchronous architectures primarily mask retrieval latency but cannot correct mismatches between retrieved content and actual information needs, potentially leading to increased computation and delayed responses. Figure 1 : Comparison of RAG architectures. (a) Synchronous RAG : generation blocks during each retrieval (287ms). (b) PipeRAG : retrieval runs in parallel but suffers from query staleness, since queries use outdated context (e.g., RET 2 at token 12 is applied at token 16 when context has shifted). (c) Ours : predictive prefetching anticipates future needs. At token 2, we predict token 5 will require retrieval and issue a query for token 5’s anticipated context. Retrieved content arrives aligned with actual needs, reducing end-to-end latency by 43.5%. Motivated by the limitation of existing asynchronous retrieval strategies, in this paper, we propose an advanced asynchronous retrieval framework that explicitly addresses three key questions: when retrieval should be triggered to initiate prefetching, whether the current context is sufficient to support retrieval, and what information should be retrieved to assist generation . Our key insight is twofold. First, retrieval needs are preceded by identifiable semantic precursors in generation dynamics, such as characteristic patterns in entropy trajectories, attention allocation, and value representation dynamics, which emerge approximately 8–16 tokens before uncertainty becomes critical. Second, these same signals encode retrieval intent and can be leveraged to infer the retrieval query itself. Building on these insights, to determine retrieval timing, we introduce a retrieval predictor that forecasts impending information needs by monitoring generation signals, including token distributions, attention patterns, and discourse markers. Then, to assess whether the accumulated generation context provides adequate semantic information for reliable query construction, we design a context monitor to determine the optimal waiting horizon before initiating retrieval. Lastly, a query generator is developed to construct queries aligned with anticipated information requirements rather than merely echoing recent context. Together, these components enable informed decisions about both when to initiate retrieval and what information to retrieve. This predictive capability supports asynchronous prefetching, allowing retrieval to proceed concurrently while generation continues uninterrupted, such that the prefetched content aligns with the model’s actual information needs when uncertainty arises. Specifically, all three components (the retrieval predictor, context monitor, and query generator) are pretrained jointly on collected historical generation traces and further adapted online via policy gradient with action-specific feedback. Through this process, the predictor learns to make prefetching decisions that improve generation quality, balancing prediction confidence with resource constraints. Our main contributions are summarized as follows: • We propose an asynchronous retrieval strategy that explicitly predicts when retrieval should be triggered and what information should be retrieved to support generation. • We design three key components, namely a retrieval predictor, a context monitor, and a query generator, that enable asynchronous prefetching and align retrieved content with the model’s actual information needs. • A comprehensive evaluation on HotpotQA, 2WikiMultiHopQA, Natural Questions, and TriviaQA, demonstrates 43.5% end-to-end latency reduction, 62.4% time-to-first-token improvement, and 31% fewer retrievals per 1K tokens, while maintaining answer quality within 1% of synchronous RAG baselines. 2 Related Work Retrieval-Augmented Generation (RAG) enhances language models by incorporating external knowledge, but each retrieval introduces additional latency that directly constrains generation throughput. Prior work has addressed this bottleneck along two main directions: system-level optimization and adaptive retrieval strategy. We situate our predictive prefetching approach with respect to both lines, with detailed technical discussion provided in Appendix H . System Optimization. System-level optimizations reduce retrieval overhead through advances such as efficient vector databases ( 14 ) , hybrid retrieval that combines dense and sparse signals, knowledge graph–based methods ( 6 ) , and cache-augmented strategies ( 4 ) . Despite shortening per-query latencies, these approaches still maintain a dependency between retrieval and generation, leaving generation fundamentally constrained by retrieval latency. Our predictive prefetching approach is orthogonal to such optimizations: by forecasting retrieval needs in advance, it overlaps retrieval with ongoing generation, effectively hiding retrieval latency. Adaptive Retrieval Strategy. Adaptive methods such as FLARE ( 13 ) , Self-RAG ( 1 ) , and DRAGIN ( 33 ) dynamically trigger retrieval based on uncertainty signals, but do so reactively, first detecting uncertainty and then blocking generation to perform retrieval. TeleRAG ( 22 ) and PipeRAG ( 12 ) are concurrent efforts that explore asynchronous retrieval via pipeline parallelism. TeleRAG relies on a fixed lookahead window, triggering retrieval

Read the complete text