Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?

Syed Huma Shah · 2026

Read Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer? by Syed Huma Shah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27494.txt.

Introduction

Syed Huma Shah asks when reusing a cached answer is safe in retrieval-augmented generation. His GroundedCache router admits a cached answer only when four gates hold together: query similarity, evidence overlap, source-version validity, and lexical support from fresh evidence. The router drives unsafe servings to zero on HotpotQA regimes and under 2 percent on document drift, at near-baseline latency.

Catalog record

CREATORSyed Huma Shah
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36750
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27494

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer? Syed Huma Shah Abstract Modern retrieval-augmented generation (RAG) deployments increasingly rely on caching to reduce token cost and time-to-first-token (TTFT). Prefix-level KV reuse is now standard in serving stacks such as vLLM, and chunk-level and position-independent reuse have been pushed further by recent systems (RAGCache, TurboRAG, CacheBlend, EPIC, ContextPilot, PCR, LMCache). Output-level semantic answer caches , by contrast, remain fragile: similar prompts can map to different correct answers, retrieved evidence drifts as the corpus is updated, and adversarial collision attacks have been shown to hijack cached responses. We argue that the right framing for cached answer reuse is not how to reuse faster but when reuse is safe . We propose GroundedCache , an evidence-validated cache router that admits a cached answer only when four cheap gates simultaneously hold: query similarity, retrieved-evidence overlap, source-version validity, and lexical (or judge-based) support of the cached answer by the freshly retrieved evidence. We build a six-regime workload (exact-repeat, paraphrase, near-miss, document-drift, long shared-document, bounded-KB CAG) that explicitly stress-tests cache safety rather than only hit rate, and an operator-facing metric, the unsafe-served rate (USR), defined as the fraction of all queries that received a wrong cached answer, alongside TTFT and token cost. Across two datasets and 12,000 real-LLM generations (Qwen2.5-7B-Instruct served by vLLM with Automatic Prefix Caching), GroundedCache drives USR to 0.0% on every HotpotQA regime (vs. 15–35% under naive semantic caching) and to 1.5% on mtRAG document drift (vs. 51.5% under naive), a 34 × \times reduction in wrong cached answers per query on the system’s design-point adversarial regime, and 3–10 × \times reductions across the other mtRAG regimes. End-to-end p50 latency under GroundedCache stays within 1.04 1.04 – 1.07 × 1.07\times of a no-cache RAG baseline; the no-support variant trades USR for 1.4 1.4 – 1.5 × 1.5\times speedup. A per-gate marginal ablation isolates the lexical support gate as the load-bearing safety mechanism on both datasets ( + 0.125 +0.125 USR on HotpotQA, + 0.118 +0.118 on mtRAG when removed), with the remaining gates providing defense-in-depth at near-zero cost. We release the end-to-end implementation, traffic-synthesis tooling, and evaluation harness so the framework can be ported to richer datasets and stronger compressors. 1 Introduction A typical RAG pipeline repeatedly does the same expensive work for queries that are functionally indistinguishable to the user: it embeds the query, runs nearest-neighbor search over a corpus, concatenates the top- k chunks into a prompt, and decodes a long-context answer. Three layers of caching can in principle eliminate that work: 1. Prefix / KV cache reuse at the serving layer (vLLM APC [ 24 ] , SGLang RadixAttention [ 31 ] ), and its RAG-aware extensions (RAGCache [ 13 ] , TurboRAG [ 19 ] , CacheBlend [ 29 ] , EPIC [ 11 ] , ContextPilot [ 7 ] , PCR [ 25 ] , LMCache [ 16 ] ). 2. Retrieval-result reuse , where the top- k document set itself is cached for similar queries (Proximity [ 4 ] , QVCache [ 10 ] ). 3. Output / answer reuse , where the entire generated answer is returned for a semantically similar prior query (GPTCache [ 3 ] , ContextCache [ 28 ] ). The first two have strong correctness guarantees by construction: KV reuse preserves token outputs exactly when prefixes match, and retrieval reuse is safe as long as the underlying corpus is stable. Output-level semantic caching does not. A cosine-similar prior query can map to a cached answer whose underlying evidence is no longer valid, no longer the same evidence, or never supported the question in the first place. Recent work has shown that naive semantic caches are even adversarially exploitable, with reported 86% response-hijacking rates in evaluated scenarios [ 30 ] . We make three contributions: 1. An evidence-validated cache router ( GroundedCache ). A cached answer is admitted only when (G1) the new query is semantically close to the cached query, (G2) the new retrieved evidence shares a high Jaccard with the cached evidence signature, (G3) shared chunks carry the same source version, and (G4) the cached answer’s content tokens are covered by the new evidence (or, optionally, verified by a lightweight judge call). The router falls back to query-conditioned compression and generation when any gate fails. 2. A six-regime workload for cache-safety stress testing. We provide deterministic synthesis for exact-repeat, paraphrase, near-miss, document-drift, long shared-document, and bounded-KB CAG traffic over arbitrary RAG corpora, plus a HotpotQA adapter. The document-drift regime mutates numeric tokens in the gold documents so that a cached answer becomes verifiably wrong unless rejected. 3. Operator-facing metric stack. Beyond TTFT, latency, tokens, and answer-cache hit rate we report the unsafe-served rate USR (fraction of all queries that received a wrong cached answer), the conditional false-hit rate FH (per served cached answer), and stale-/unsupported-hit decompositions, so the safety/reuse trade-off is auditable at both the system and the per-query level. Our framing is complementary to, rather than competitive with, prefix-level work: GroundedCache treats vLLM APC and chunk-level KV reuse as orthogonal serving primitives operating below the router. The contribution is a policy layer on top. 2 Related Work Prefix and KV caching are now standard in open-source serving stacks. vLLM’s Automatic Prefix Caching [ 24 ] and SGLang’s RadixAttention [ 31 ] reuse KV blocks when prompt prefixes align. The line of work on RAG-specific KV reuse moves beyond exact prefix matching: RAGCache [ 13 ] caches intermediate states of retrieved documents and reports up to 4\times TTFT and 2.1\times throughput gains; TurboRAG [ 19 ] pre-computes document KV caches offline for up to 9.4\times TTFT reduction; CacheBlend [ 29 ] corrects reused non-prefix chunks via selective recomputation. EPIC [ 11 ] formalizes position-independent caching, and ContextPilot [ 7 ] adds context indexing, ordering, and de-duplication. PCR [ 25 ] adds prefetch-enhanced reuse, and LMCache [ 16 ] provides multi-tier KV storage and cross-engine reuse. All of these target the how of reuse and largely assume the when is safe. Semantic answer caching has a separate failure-mode literature. GPTCache [ 3 ] is the canonical open-source baseline. ContextCache [ 28 ] extends the cache key with multi-turn dialogue context, demonstrating that turn-level similarity is insufficient when the referent has shifted. Proximity [ 4 ] caches retrieval results rather than outputs, validating that retrieval-side caching is safer. QVCache [ 10 ] learns region-specific thresholds for safe ANN-level reuse, cutting query latency 40 – 1000\times on hits without compromising recall. Recent work shows naive semantic caching is vulnerable to key-collision attacks with up to 86% response-hijacking rate [ 30 ] , and agentic-plan caching suffers high false positives when outputs depend on external state. These three failure modes, referent shift, retrieval drift, and adversarial collision, motivate the four gates in our validator. The compression literature has moved from hard pruning (RECOMP [ 27 ] , LongLLMLingua [ 12 ] , Provence [ 6 ] , PISCO [ 18 ] ) to adaptive, query-conditioned, or representation-level compression (OSCAR [ 17 ] , SeleCom [ 2 ] , REFRAG [ 20 ] ). We use a sentence-level selector as a practical baseline; learned compressors plug into the same compress(query,

Read the complete text