VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
Jiayi Yao 1 , Samuel Shen 2 , Kuntai Du 2 , Shaoting Feng 1 , Dongjoo Seo 3 , Rui Zhang, Yuyang Huang 1 , Yuhan Liu 1 , Shan Lu 4,1 , Junchen Jiang 2,1 · 2026
Read VeriCache: Turning Lossy KV Cache into Lossless LLM Inference by Jiayi Yao 1 , Samuel Shen 2 , Kuntai Du 2 , Shaoting Feng 1 , Dongjoo Seo 3 , Rui Zhang, Yuyang Huang 1 , Yuhan Liu 1 , Shan Lu 4,1 , Junchen Jiang 2,1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17613.txt.
Introduction
Jiayi Yao and colleagues present VeriCache, which makes lossy KV-cache compression match full-cache output at near full decoding speed. Compressed caches draft tokens for verification against the full cache, kept off GPU memory. Compressed-cache decoding runs in parallel with full-cache swap, and similar drafts allow a long horizon to amortize each swap. VeriCache covers long-context decoding and remote prefix caching, handles token-dropping and quantization methods, and composes with speculative decoding. Tests show up to four times full-cache throughput with identical outputs.
Catalog record
| CREATOR | Jiayi Yao 1 , Samuel Shen 2 , Kuntai Du 2 , Shaoting Feng 1 , Dongjoo Seo 3 , Rui Zhang, Yuyang Huang 1 , Yuhan Liu 1 , Shan Lu 4,1 , Junchen Jiang 2,1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68939 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17613
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VeriCache: Turning Lossy KV Cache into Lossless LLM Inference Jiayi Yao 1 , Samuel Shen 2 , Kuntai Du 2 , Shaoting Feng 1 , Dongjoo Seo 3 , Rui Zhang, Yuyang Huang 1 , Yuhan Liu 1 , Shan Lu 4,1 , Junchen Jiang 2,1 Abstract . The large size of the KV cache has become a major bottleneck for serving LLMs with increasing context lengths. In response, many KV cache compression methods, such as token dropping and quantization, have been proposed. However, almost all of these methods are inherently lossy—despite minimal accuracy degradation for short outputs, their outputs increasingly diverge from full-KV-cache outputs as more tokens are decoded, which leads to catastrophic failures in code generation and tool calling. We present VeriCache, the first inference framework that ensures the same output as full-KV-cache decoding but largely preserves the high decoding throughput of a range of KV cache compression algorithms. VeriCache uses the compressed KV cache to draft tokens, then verifies them against the full KV cache. While it may seem like just speculative decoding, VeriCache requires addressing a key system challenge to work—keeping the full KV cache out of GPU memory and minimizing the overhead of swapping it in for verification. The insight is two-fold: (1) compressed-KV decoding can be parallelized with full-KV swap, because one is HBM-bandwidth-bound and the other is PCIe/network-bound, and (2) the compressed KV cache often produces output similar to the full KV cache, allowing a long drafting horizon to amortize each full-KV swap. VeriCache applies to both long-context decoding and remote prefix caching, supports a broad family of token-dropping and quantization methods through a uniform compressor interface, and composes with traditional speculative decoding. Experimental results show that VeriCache achieves up to 4 × 4\times higher throughput than full-KV inference while producing identical outputs. 1. Introduction Figure 1. The accuracy–throughput dichotomy. VeriCache attains the throughput of compression with the same output as full KV. The context lengths of state-of-the-art LLMs have grown to more than one million tokens. This growth has powered many applications, from repository-level code generation ( 12 ; 40 ; 54 ; 33 ; 32 ) , multi-document reasoning ( 47 ; 67 ; 23 ; 59 ) , to agentic workflows with long interaction histories ( 56 ; 42 ; 43 ; 75 ) . The performance impact of KV cache manifests at both single-request and multi-request LLM serving. Within a single request, every decoding step must read the entire KV cache from GPU memory (HBM) to on-chip memory for attention. Furthermore, the size of the KV cache degrades request throughput, as it reduces the number of requests that can fit in GPU memory ( 58 ; 36 ) . Across requests, KV cache reuse is a common practice to avoid repetitive KV cache generation among requests that share long common prefixes (e.g., system prompts, shared documents, prior conversation turns). Then large KV caches on storage need to be quickly loaded onto GPU memory when a request arrives, which can dominate request latency ( 37 ; 36 ; 64 ) . A growing line of work tackles these problems by compressing the KV cache—either by dropping tokens at selected layers and attention heads ( 65 ; 20 ; 17 ; 8 ; 60 ) or by reducing precision through quantization ( 37 ; 15 ; 38 ; 74 ; 68 ) . Both deliver substantial efficiency gains—with 2–5 \times reductions in memory or transfer size. Although effective at improving throughput, compressing KV cache changes its content , causing inference output to diverge from the distribution of decoding outputs using the full-size KV cache. As we will show in Section 3 , the probability of divergence accumulates with more output tokens. When an application needs long output, the output divergence often violates functional correctness or structural requirements, even if the output is still smooth natural language. On coding and tool-calling benchmarks, functional accuracy (e.g., syntax validity, exact argument matching) degrades sharply even at moderate compression ratios. This creates a dichotomy (Figure 1 ): accept lossy KV and risk output quality, or use full KV at the cost of much lower throughput. We pose the following question: Can we exploit the throughput benefits of KV cache compression without affecting LLM output? In this paper, we present VeriCache, a new inference scheme inspired by speculative decoding ( 28 ; 27 ; 44 ; 4 ; 6 ) . Instead of directly serving tokens from compressed KV cache, VeriCache uses the compressed KV cache to draft tokens, then verifies them against the full KV cache. Wrong tokens are corrected, so the final output is identical 1 1 1 In this paper, we define identical results as identical under greedy decoding (i.e., zero temperature) except for randomness caused by hardware to full-KV inference. Directly applying existing speculative decoding techniques is insufficient as token verification overhead can cause throughput degradation (see § 4 ), negating the benefits of KV cache compression. VeriCache uses two key designs that take advantage of the fact that drafting and verification in VeriCache use exactly the same model and model weights, a property different from traditional speculative decoding. First, VeriCache schedules cross-resource staggering . By sharing exactly the same model (weights), token drafting and token verification become feasible to execute in the same batch, offering a huge performance benefit as drafting and verification have complementary resource bottlenecks . Drafting decodes one token at a time using compressed KV located in GPU memory. Its sequential vector-matrix multiplication under-utilizes GPU FLOPs and makes it a GPU memory bandwidth bound operation. Verification, in contrast, requires loading the full KV cache from secondary storage into GPU and verifying multiple tokens in parallel, putting its bottleneck on inter-connect bandwidth and GPU FLOPs. Consequently, staggering drafting and verification across these distinct hardware resources offers much better utilization than running only drafting or only verification in each batch—the lock-step playbook in traditional speculative decoding. Cross-resource staggering requires much more sophisticated scheduling, which we will explain in § 5 . Second, VeriCache uses an extended verification period that maximizes the parallelism of verification and minimizes the loading frequency of full KV cache. How often verification happens (i.e., every how many output tokens per verification) depends on how likely the drafted tokens would be accepted by the verification process. The more likely they are, the less frequent the verification can be. Different from traditional speculative decoding, compressed KV caches retain the same model weights and dominant attention patterns as the full model, so the compressed-KV drafter sustains much longer accepted runs than a traditional small-model drafter—25–40 accepted tokens per verification round for VeriCache vs. only 2–3 for typical small-model drafters (Section 4.3 ). VeriCache handles both long-context decoding and remote prefix caching, with a runtime scheduler that adapts verification frequency and batch composition to hardware and workload conditions. The idea of driving a speculative drafter from a sparser or compressed KV cache is not new. MagicDec ( 53 ) pairs a small-model drafter with a sparse KV cache; QuantSpec ( 62 ) and SparseSpec ( 77 ) extend it to self-speculation with hierarchical quantization and dynamic sparse attention, respectively. However, there are two key differences. First, prior systems all keep