RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, Alexey Tumanov · 2025
Read RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression by Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, Alexey Tumanov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14051.txt.
Introduction
Payman Behnam and co-authors present RocketKV, a method that shrinks the memory cache used by large language models on long inputs, in work from 2025. The cache stores past token data so the model need not recompute it, but it grows with input length and strains memory. Long documents make the problem severe.
RocketKV compresses in two stages. First it permanently drops less useful cached entries from the input. Then it uses a sparse attention scheme that estimates scores cheaply along two axes at once. The method needs no retraining. It reached compression up to four hundred times and end-to-end speedups near four times, with peak memory cut by about a third and little accuracy loss on long-context tasks. A variant handles multi-turn chat.
Catalog record
| CREATOR | Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, Alexey Tumanov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 107 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-14051
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, Alexey Tumanov Abstract Transformer-based Large Language Models rely critically on the KV cache to efficiently handle extended contexts during the decode phase. Yet, the size of the KV cache grows proportionally with the input length, burdening both memory bandwidth and capacity as decoding progresses. To address this challenge, we present RocketKV, a training-free KV cache compression strategy containing two consecutive stages. In the first stage, it performs coarse-grain permanent KV cache eviction on the input sequence tokens. In the second stage, it adopts a hybrid sparse attention method to conduct fine-grain top-k sparse attention, approximating the attention scores by leveraging both head and sequence dimensionality reductions. We show that RocketKV provides a compression ratio of up to 400 × \times , end-to-end speedup of up to 3.7 × \times as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks. We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme. The source code is available here: https://github.com/NVlabs/RocketKV . † † affiliationnotice: Equal contribution 1 Introduction In Transformer-based LLM inference ( Vaswani et al., 2017 ; Pawar et al., 2024 ) , the key-value cache (KV cache)—which stores past attention keys and values to avoid recomputation—becomes a major bottleneck during the decode phase, as its size scales linearly with both the sequence length and batch size. For example, the Llama3.1-70B-Instruct ( AI@Meta, 2024 ) model with a batch size of 32, and a context length of 32K requires around 320GB of KV cache storage at FP16 precision, which even advanced hardware ( NVIDIA, 2024 ; Jouppi et al., 2023 ; AMD, 2024 ) can hardly handle. Fortunately, previous work has shown that only a small subset of KV tokens is required at each decode step to maintain accuracy ( Zhang et al., 2023 ; Tang et al., 2024 ; Ribar et al., 2024 ; Li et al., 2024 ; Singhania et al., 2024 ; Ge et al., 2024 ) . Therefore, if those KV tokens can be accurately predicted in advance, dense attention operations can be replaced with sparse attention operations with significant memory bandwidth and capacity improvement. These methods often fall into two categories: 1) permanent KV token eviction, and 2) dynamic KV token selection. The former results in both memory bandwidth and storage savings, but could lead to noticeable accuracy loss if KV tokens dropped earlier are needed by later decode steps. The latter avoids this shortcoming by keeping all KV tokens in memory and dynamically selecting a subset each time. Hence, it only results in memory bandwidth savings, but often requires extra memory storage overhead for auxiliary data. Figure 1: Existing KV token dropping methods fail to match the accuracy scores of oracle top-k attention (Exact-TopK) on Mistral-7B-Ins-v0.2 in the qasper benchmark. To understand the effectiveness of existing KV token dropping methods, Figure 1 presents the accuracy comparison of four different methods (two in each category) for qasper benchmark in LongBench ( Bai et al., 2024 ) on the Mistral-7B-Instruct-v0.2 model ( Jiang et al., 2023 ) . We observe that the accuracy of all four methods drops significantly as the token budget becomes lower than 1024 while an oracle top-k attention scheme achieves negligible accuracy drop even with a token budget of 256. This indicates that existing practical methods fail to accurately predict top-k KV tokens under low token budgets. To improve the prediction accuracy, we propose RocketKV, a two-stage KV cache compression method that combines permanent KV token eviction with dynamic KV token selection to accelerate the decode phase of LLM inference. The RocketKV framework enables flexible integration of a wide range of KV cache compression techniques at each stage. To achieve optimal performance, we directly adopt an existing method SnapKV ( Li et al., 2024 ) for coarse-grain permanent KV token eviction in the first stage. In the second stage, we propose hybrid sparse attention (HSA) to perform fine-grain dynamic KV token selection, which estimates KV token indices with top-k attention scores via two-dimensional reductions. To balance the compression between two stages, RocketKV introduces an adaptive compression decomposition mechanism that intelligently splits any given target compression ratio across two stages. Combining these two stages together, RocketKV achieves significant memory bandwidth and storage savings with negligible accuracy loss across a wide variety of models and downstream tasks. Moreover, recent work ( Li et al., 2025 ) demonstrates that permanent KV token eviction suffers in multi-turn decoding because important KV tokens vary significantly across queries. To address this challenge, we propose a variant of RocketKV called RocketKV-MT for multi-turn scenarios where we do not evict the unselected KV tokens in the first stage but keep them all for later turns. Yet, the decode phase is still restricted to perform dynamic selection on the filtered KV tokens from the first stage in each turn. By doing this, RocketKV-MT achieves the same memory traffic savings as RocketKV but does not introduce memory storage savings. In summary, we make the following contributions: • We analyze and identify the limitations of existing KV token dropping methods and then propose RocketKV as a two-stage KV cache compression scheme. We further introduce a variant RocketKV-MT for multi-turn decoding scenarios. • We propose a hybrid sparse attention (HSA) for dynamic KV token selection with two-dimensional reductions in the second stage while directly adopting SnapKV in the first stage. We design an adaptive compression decomposition mechanism to balance the compression between the two stages. • We conduct a comprehensive evaluation of RocketKV and RocketKV-MT on a wide variety of models and downstream tasks. RocketKV consistently demonstrates comparable accuracy with full KV attention at up to 400 \times compression ratio, while achieving up to 3.7 \times end-to-end speedup and 32.6% peak memory saving at the decode phase on an NVIDIA A100 GPU. Meanwhile, RocketKV-MT is more suitable for multi-turn decoding and performs on par with oracle top-k attention. 2 Related Work A number of recent approaches have focused on improving the efficiency of attention mechanisms in LLMs, particularly when dealing with long contexts. One feasible solution is KV cache sharing across multiple layers ( Brandon et al., 2024 ) or selectively dropping attention for some layers ( Ling et al., 2024 ) . Others ( Sun et al., 2024 ; Ho et al., 2024 ) propose mixed attention designs where some layers use global attention while others use local attention. Multi-Query Attention (MQA) ( Shazeer, 2019 ) , Grouped-Query Attention (GQA) ( Ainslie et al., 2023 ) , and Multi-head Latent Attention(MLA) ( Liu et al., 2024a ) are widely adopted by many recent LLMs ( AI@Meta, 2024 ; Jiang et al., 2023 ; Team et al., 2024 ; Liu et al., 2024b ) , which reduce the KV cache by sharing key-value pairs across multiple attention heads. All above techniques directly modify the attention architecture and require integration since pre-training. Another