QCFuse: Query-Centric Cache Fusion for Efficient RAG Inference
Jianxin Yan, Zeheng Qian, Wangze Ni, Zhitao Shen, Zhiping Wang, Haoyang Li, Jia Zhu, Lei Chen, Kui Ren · 2026
Read QCFuse: Query-Centric Cache Fusion for Efficient RAG Inference by Jianxin Yan, Zeheng Qian, Wangze Ni, Zhitao Shen, Zhiping Wang, Haoyang Li, Jia Zhu, Lei Chen, Kui Ren (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08585.txt.
Introduction
Yan and eight colleagues speed retrieval aided generation with QCFuse, a cache fusion system built around the user query. Short summary anchors sharpen query forms, and only query related tokens get recomputed, guided by the most critical transformer's attention pattern. Tests on real datasets lift response speed by 40 percent at equal accuracy, and in some settings attention denoising even raises accuracy above current methods.
Catalog record
| CREATOR | Jianxin Yan, Zeheng Qian, Wangze Ni, Zhitao Shen, Zhiping Wang, Haoyang Li, Jia Zhu, Lei Chen, Kui Ren |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17372 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08585
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
QCFuse: Query-Centric Cache Fusion for Efficient RAG Inference Jianxin Yan, Zeheng Qian, Wangze Ni, Zhitao Shen, Zhiping Wang, Haoyang Li, Jia Zhu, Lei Chen, Kui Ren Abstract . Cache fusion accelerates generation process of LLMs equipped with RAG through KV caching and selective token recomputation, thereby reducing computational costs and improving efficiency. However, existing methods primarily rely on local perspectives for token selection and lack global awareness from the user query. Utilizing this global awareness is challenging due to the high cost of obtaining context-aware query representations and the strict pipeline constraints required for efficient attention analysis. Thus, this demonstration introduces QCFuse, an innovative KV cache fusion system centered on the user query. QCFuse leverages semantic summary anchors to enhance query representations and selectively recomputes query-related tokens to improve accuracy, updating tokens based on the attention distribution of the most critical Transformer layer to preserve the high efficiency of the pipeline structure. Evaluations on real-world datasets demonstrate that QCFuse significantly improves the response efficiency of LLMs by 40% while maintaining equivalent accuracy compared to current methods. Additionally, in certain scenarios, QCFuse achieves an attention denoising effect that yields higher response accuracy, demonstrating substantial potential in the optimization of LLM inference. † † authors: . PVLDB Reference Format: PVLDB, 14(1): XXX-XXX, 2020. doi:XX.XX/XXX.XX † † This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected] . Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 14, No. 1 ISSN 2150-8097. doi:XX.XX/XXX.XX PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/uYanJX/QCFuse-Demo . 1. Introduction LLMs equipped with RAG are standard for enterprise knowledge-base question answering and content generation, as they mitigate hallucinations and support real-time knowledge updates. In high-concurrency production environments, however, RAG generation remains heavily bottlenecked. Although context chunks retrieved for different queries can overlap by over 70%, strict prefix-matching policies prevent traditional prefix caching from reusing them. LLMs are thus forced to fully prefill redundant contexts. As a result, time to first token (TTFT) grows quadratically with context length, wasting immense computational resources. Figure 1 . Comparison among Full Computation, Full Reuse, and Cache Fusion. As illustrated in Figure 1 , cache fusion has emerged as a primary optimization strategy. These approaches merge historical KV caches and selectively recompute a subset of tokens. This reduces costs while preserving accuracy comparable to full computation. Prior work determines token recomputation targets in various ways: CacheBlend ( 5 ) uses numerical deviations of KV tokens, whereas EPIC ( 1 ) statically recomputes a fixed ratio. Both methods effectively reduce computational overhead. Figure 2 . Architecture of the QCFuse System. A central limitation of existing methods is their lack of global awareness regarding the user query. By relying primarily on local cues, such as static positional heuristics or first-layer KV deviations, they overlook the query’s role as the primary driver of the generation process. Operating on intermediate representations without considering the original request leads to suboptimal resource allocation. Computational budget is often spent on irrelevant tokens while critical ones are ignored, causing significant accuracy drops under aggressive acceleration. Using the query’s attention distribution over context tokens as a selection criterion is an intuitive alternative. Tokens with high query attention typically exert the greatest influence on generation quality. Realizing this within cache fusion systems, however, presents two primary challenges. • The first challenge concerns obtaining context-aware query representations at minimal cost . A naive query-only forwarding process, which omits the contextual KV cache, yields ungrounded representations and unreliable attention distributions. Conversely, injecting the complete context KV cache during query forwarding disrupts the pipelined execution of cache fusion systems. To handle very long contexts, systems store full KV caches on SSDs and process them layer by layer. While the GPU recomputes one layer, the pipeline prefetches the next layer’s KV from the SSD. Requiring the full context KV to be materialized beforehand forces the system to wait for all data to load, reducing the pipeline to sequential execution and eliminating efficiency gains. Acquiring context-enhanced query representations without disrupting this pipeline is thus essential. • The second challenge involves efficient attention analysis within these pipeline constraints . Computing query attention across all Transformer layers, as done in ProphetKV ( 3 ) , blocks the pipeline due to cross-layer dependencies. Relying solely on the final layer, as in FusionRAG ( 2 ) , provides an incomplete semantic view. The system must instead identify a single pipeline-friendly layer whose attention distribution serves as a reliable proxy for global token importance. We address these challenges through two main technical contributions. First, we propose anchor-based lightweight query probing. By analyzing token key-norm magnitudes, we extract anchor tokens from each precomputed context chunk to serve as compressed semantic summaries. These tokens are injected as lightweight prefixes during query forwarding to produce context-enhanced query representations. Unlike previous context-free forwarding methods, this approach maintains pipeline efficiency. Second, we achieve semantic localization via critical-layer attention profiling. Empirical findings suggest that middle layers offer superior semantic localization. We therefore analyze query attention at a single critical middle layer. This mechanism avoids the pipeline stalls associated with cross-layer dependencies and the incomplete views of last-layer approaches, successfully balancing accuracy and system efficiency. Building on these mechanisms, this demonstration paper presents QCFuse, an efficient query-centric cache fusion system implemented on SGLang. The system features a high-performance sparse-attention Triton kernel for discrete token recomputation. Evaluations across multiple LLMs and multi-hop QA benchmarks demonstrate up to a 2 \times speedup of TTFT over full computation and a 40% latency reduction compared to existing cache fusion baselines, with matching or improved generation quality. We demonstrate how QCFuse integrates seamlessly into enterprise knowledge assistants to deliver near-real-time answers over massive document collections. 2. Related Work KV and Prefix Caching. KV and prefix caching ( 6 ) accelerates large language model inference by reusing historical Key/Value matrices, avoiding redundant computation and reducing latency. Frameworks such as SGLang ( 6 ) implement this via prefix-tree matching. While highly effective for constant prefixes, these mechanisms struggle with the dynamic context assemblies typical of RAG applications, where retrieval chunks are frequently reordered or inserted into the middle of the prompt. Location-independent Caching and Selective Recomputation. To enable cache reuse in dynamic contexts, fusion methods concatenate discrete KV chunks and selectively recompute tokens that experience semantic shifts. CacheBlend ( 5 ) identifies these tokens using first-layer KV deviations, EPIC ( 1 ) defaults to a static ratio at the sequence start, and KVShare ( 4 ) scales KV deviation by initial attention weights. More recently, ProphetKV ( 3 ) and FusionRAG ( 2 ) proposed query-guided token selection, yet both encounter pipeline synchronization issues. ProphetKV evaluates all layers concurrently, ignoring stage-wise scheduling constraints. FusionRAG computes an initial query pass without context, relying on last-layer attention that provides insufficient semantic grounding. We emulate these two query-centric baselines in our evaluations as QCAll and QCLast. 3. System 3.1. System Workflow As shown in Figure 2 , the workflow of QCFuse consists of four highly optimized phases: Phase