Efficient Remote Prefix Fetching with GPU-native Media ASICs
Liang Mi 1 Weijun Wang 2 Jinghan Chen 1 Ting Cao 2 Haipeng Dai 1 Yunxin Liu 2 1 Nanjing University 2 Institute for AI Industry Research (AIR), Tsinghua University · 2026
Read Efficient Remote Prefix Fetching with GPU-native Media ASICs by Liang Mi 1 Weijun Wang 2 Jinghan Chen 1 Ting Cao 2 Haipeng Dai 1 Yunxin Liu 2 1 Nanjing University 2 Institute for AI Industry Research (AIR), Tsinghua University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09725.txt.
Introduction
Remote key-value cache reuse fetches cached key-value data for identical contexts from remote storage, avoiding recomputation and speeding language model inference. Liang Mi and five co-authors note that it shines in high-speed networks but fades in bandwidth-limited cases. Recent compressed-transmission studies help, yet heavy decompression eats the reuse gains. Their 2026 paper offers KVCodec, an efficient, widely deployable fix using GPU-native video codecs.
A codec-friendly tensor layout packs the key-value cache into a highly compact video format for fast sending. An efficient fetcher runs sending, decoding, and restoring of packed caches in a pipelined way, cutting resource contention, masking network swings, and reaching minimum time-to-first-token. Tested across diverse GPUs from high- to low-end, KVCodec cuts time-to-first-token by up to three and a half times while keeping lossless accuracy against state-of-the-art methods.
Catalog record
| CREATOR | Liang Mi 1 Weijun Wang 2 Jinghan Chen 1 Ting Cao 2 Haipeng Dai 1 Yunxin Liu 2 1 Nanjing University 2 Institute for AI Industry Research (AIR), Tsinghua University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-09725
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Remote Prefix Fetching with GPU-native Media ASICs Liang Mi 1 Weijun Wang 2 Jinghan Chen 1 Ting Cao 2 Haipeng Dai 1 Yunxin Liu 2 1 Nanjing University 2 Institute for AI Industry Research (AIR), Tsinghua University Abstract . Remote KV cache reuse fetches KV cache for identical contexts from remote storage, avoiding recomputation, accelerating LLM inference. While it excels in high-speed networks, its performance degrades significantly in bandwidth-limited scenarios. Recent studies address this by transmitting KV caches in compressed form, but the associated heavyweight decompression counteracts the KV reuse benefits. In this paper, we propose an efficient and widely deployable remote KV cache reuse solution that leverages GPU-native video codecs . Our system, KVFetcher, enables effective KV cache coding with two techniques. The codec-friendly tensor layout compresses the KV cache in a highly compact video format, enabling fast transmission. The efficient KV fetcher orchestrates the transmission, decoding, and restoration of compressed KV caches in an efficient pipelined manner, eliminating resource contention, masking network fluctuations, and achieving minimum time-to-first-token (TTFT). We prototype KVFetcher on diverse GPUs from high- to low-end. Experiments reveal that it reduces TTFT by up to 3.51 × \times while maintaining lossless accuracy, compared to SOTA methods. 1. Introduction With context window trending to millions of tokens, Large Language Models (LLMs) have demonstrated superiority in many modern services. By ingesting informative contexts, including system prompts ( 45 ) , retrieved documents ( 13 ; 30 ) , and interaction histories ( 8 ; 9 ; 81 ; 67 ) , together with request queries, LLMs can generate high-quality outputs powering diverse applications, from code generation ( 12 ; 25 ; 55 ) to agentic workflows ( 26 ; 63 ; 72 ) . For efficient serving, storing the intermediate states, KV cache , during LLM inference has become a de facto choice. KV cache reuse further amortizes the computational costs across multiple requests. It persists KV caches in the first inference and reuses them when future requests share common prefixes. Most LLM inference systems ( 73 ; 22 ; 35 ; 79 ; 34 ; 54 ; 44 ; 3 ) today integrate KV cache reuse as their key feature. Their consensus is that KV caches are frequently reused. Mooncake ( 54 ) claims that 50% of KV caches will be reused in KiMi’s real-world workloads, while LMCache ( 44 ) believes this number is higher. Despite this benefit, the storage cost of KV caches can be huge ( 35 ; 68 ) . Storing 80K-token KV caches ( e.g. , Amazon’s annual report) of a medium-level 34B model can consume up to 19GB of storage ( 42 ) . For this reason, hosting all reusable KV caches locally is impractical. Figure 1. Solutions of remote KV cache reuse. Our KVFetcher exploits GPU-native video codecs, delivering the best cost-efficiency and system performance. To tackle this, KV cache reuse uses external storage and fetches them on demand from remote resources. Mooncake abstracts all GPU memory, host memory, and disks within GPU clusters into a disaggregated KV cache pool, and LMCache leverages dedicated storage servers. In such distributed setups, KV caches must be transmitted over the network. Although high-speed interconnections ( e.g. , RDMA) render transmission latency negligible ( 54 ; 80 ; 58 ; 39 ) , economic considerations drive modern LLM services to deploy on mid-range GPUs typically paired with only constrained bandwidths, tens of Gbps or less ( 46 ; 49 ; 68 ; 47 ; 14 ; 33 ) . Recent studies ( 42 ; 68 ) mitigate this networking bottleneck by transmitting KV caches in compressed form rather than raw tensors. It indeed yields bandwidth savings, but the associated decompression latency lies on the critical path of LLM inference, counteracting the network benefits. CacheGen ( 42 ) customizes a CUDA kernel to accelerate decompression, yet it competes for GPU resources with LLM inference engines, as the blue box shown in Fig. 1 (a), severely degrading LLM inference performance (§ 2.2 ). ShadowServe ( 68 ) achieves interference-free inference by offloading decompression to SmartNIC, as shown in Fig. 1 (b); but, the prohibitive hardware cost hinders its widespread adoption. We believe KV cache compression is promising, but requires a cost-efficient redesign to be widely used. Modern GPUs are typically equipped with dedicated hardware for efficient video coding. Although initially designed for media processing, their independent on-chip computing and storage units offer a great opportunity to address the above limitations (§ 2.3 ). More importantly, as dashed boxes shown in Fig. 1 , they are completely idle throughout the LLM inference. Motivated by this, we ask: Can we implement a KV cache codec with GPU-native video codecs, enabling efficient remote KV cache reuse? A concurrent study, llm.265 ( 71 ) , attempts to compress KV caches by video coding, but cannot be the answer to this question. It provides very limited compression gain and lacks system co-design with inference engines (§ 2.4 ). To be the answer, we must tackle two challenges. First, to minimize transmission delay, efficient remote KV cache reuse necessitates highly compact compression. During compression, it must map KV tensors to a video format and encode them into bitstreams using video encoders. However, this is non-trivial. A successful compression strategy must comprehensively explore and judiciously exploit both the distribution of K and V values and the characteristics of the video coding, to maximize the compression ratio without compromising LLM generation quality. Blindly representing the KV tensor as individual video frames and directly encoding them into the bitstream, like llm.265 ( 71 ) , results in both suboptimal compression ratios and generation quality (§ 2.4 ). Second, to minimize the Time to First Token, TTFT, only compact compression is not enough. It just accelerates the transmission of single video chunks, but efficient remote KV reuse still requires considerate scheduling as well as rapid and interference-free decoding and restoration * * * This paper follows the same setting as prior studies ( 42 ; 68 ; 54 ) , where the KV caches are chunked and compressed in advance, and stored at remote storage nodes. So remote KV reuse (a.k.a., KV fetching) consists of KV transmission, decoding, and restoration. . Existing KV cache reuse systems ( 44 ; 54 ) hardly meet these requirements. First, their schedulers indiscriminately batch all arriving requests, leading to non-reuse requests (requests without remote KV cache reuse) being blocked by the fetching stage of requests with remote reuse. Second, their bulk KV restoration consumes excessive memory from LLM inference engines, delaying the TTFT. And last, their ill-considered design hardly copes with the networking jitter (§ 2.4 ). In this paper, we present KVFetcher, an efficient remote KV cache reuse module powered by GPU-native video codecs, to address the challenges outlined above. To address the first challenge, we propose a codec-friendly tensor layout that enables a tenfold compression ratio without a drop in LLM inference accuracy. Its core idea is to skip the lossy Discrete Cosine Transform (DCT) and quantization steps of video encoding, and fully utilize the lossless intra- and inter-frame redundancy elimination capability. To achieve this, we conduct an in-depth analysis of the characteristics of video coding, the