Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query

Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Qingfu Zhu, Wanxiang Che · 2025

Read Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query by Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Qingfu Zhu, Wanxiang Che (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20334.txt.

Introduction

Yixuan Wang and co-authors present a smarter way to trim the memory of large language models, in a 2025 paper. These models store past word pairs in a key-value cache to speed up writing. The cache grows fast on long texts and strains memory. Older trimming methods judge each word's worth using early-stage attention, which can mismatch the real later queries.

Their method, Lookahead Q-Cache, invents cheap fake future queries. Those stand-ins form a truer window for judging which cached words matter. Eviction then tracks real use more closely, even on tight budgets. Tests on long-text benchmarks beat prior methods at many budget levels. The trick can also pair with older methods for further gains.

Catalog record

CREATORYixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Qingfu Zhu, Wanxiang Che
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20334

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Qingfu Zhu, Wanxiang Che Abstract Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially with longer text sequences, posing challenges for efficient deployment. Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries, especially under tight memory budgets. In this paper, we propose Lookahead Q-Cache (LAQ), a novel eviction framework that generates low-cost pseudo lookahead queries to better approximate the true decoding-stage queries. By using these lookahead queries as the observation window for importance estimation, LAQ achieves more consistent and accurate KV cache eviction aligned with real inference scenarios. Experimental results on LongBench and Needle-in-a-Haystack benchmarks show that LAQ outperforms existing methods across various budget levels, achieving a 1 ∼ \sim 4 point improvement on LongBench under limited cache budget. Moreover, LAQ is complementary to existing approaches and can be flexibly combined to yield further improvements. 1 Introduction Large language models (LLMs) have demonstrated strong capabilities in long-sequence text modeling tasks ( Liu et al. 2025a ; Liu et al. 2025b ) such as code generation ( Guo et al. 2024a ; Hui et al. 2024 ) , document summarization ( Liu et al. 2024c ) , and mathematical reasoning ( Chen et al. 2025 ) . To improve inference efficiency during the decoding stage, LLMs leverage key-value cache (KV-Cache) to reduce redundant computations and accelerate inference. However, increasing text lengths lead to a substantial rise in KV cache memory usage, introducing considerable obstacles ( Shi et al. 2024 ; Li et al. 2024a ; Yuan et al. 2024 ) to efficient model deployment. Recent work ( Liu et al. 2024d ; Sun et al. 2024 ) has focused on lightweighting the KV cache in long-context scenarios. Figure 1: Illustration of the differences between the proposed Lookahead approach and existing methods. Pseudo queries are introduced to probe the importance of cached keys and values. As a straightforward and effective compression method, KV cache eviction has attracted widespread attention ( Ge et al. 2024 ; Liu et al. 2023 ; Tang et al. 2024 ) from researchers. The method improves decoding efficiency by performing token-level pruning on the prefilling-stage KV cache. Following the observations of Liu et al. 2023 ; Zhang et al. 2023 , the majority of studies adopt cumulative attention scores as the criterion for token pruning. Compared to direct accumulation, SnapKV ( Li et al. 2024b ) achieves more accurate KV cache importance estimation by leveraging an observation window. In addition, some studies ( Cai et al. 2024 ; Feng et al. 2024 ) aim to allocate finer-grained budgets across layers or attention heads to achieve higher compression rates. Although existing methods have partially alleviated the KV cache overhead in long-context scenarios, several challenges remain to be addressed. One key issue is the inconsistency between compression and inference . Under a constrained budget, KV cache eviction seeks to maintain the key-value pairs most likely to be accessed by future queries during decoding. However, as shown in Figure 1 , current approaches rely on prefilling-stage queries to approximate those in the decoding stage when selecting key-value pairs for KV cache retention. This inconsistency significantly reduces the accuracy of cache selection under a low budget, thereby degrading the performance of eviction algorithms. In this paper, we observe that employing the prefix of the generated response as the observation window leads to more consistent KV cache selection. Notably, this phenomenon is insensitive to the quanlity of the generated response: even incorrect answers are often able to recall the key cache entries required for generating the correct one. Motivated by these findings, we propose a KV cache eviction framework, LookAhead Q-Cache (LAQ) , which aligns more closely with the actual inference situation. Specifically, unlike prior works that select observation windows from the input text to compute the importance of KV cache, our method generates lookahead queries in a low-cost manner that are more consistent with the actual response queries. Once the lookahead queries are cached, we use them as the observation window for importance estimation, and then proceed with standard KV cache eviction. Experimental results on LongBench and Needle-in-a-Haystack tasks demonstrate that LAQ consistently outperforms existing methods across various budget settings. By leveraging Q-Cache, which better aligns with inference scenarios, the proposed method even outperforms some dynamically budgeted approaches under low-budget settings. Furthermore, experiments show that the proposed method can be integrated with existing methods to yield orthogonal improvements. The main contributions of this paper are as follows: • We propose the Lookahead Q-Cache, a novel framework that mitigates the inconsistency issue during both compression and inference phases by leveraging pre-generated low-quality pseudo queries. • The proposed method offers orthogonal improvements when combined with existing approaches, and its flexible configuration enables broad applicability. • Experiments across diverse benchmarks indicate that the proposed method significantly outperforms existing strategies, consistently achieving performance improvements of 1 \sim 4 percentage points on LongBench. 2 Related Works 2.1 KV Cache Eviction KV Cache eviction aims to prune redundant KV cache entries from the prefill stage at the token level, improving decoding speed and alleviating memory usage without compromising decoding performance. Previous work primarily builds on Liu et al. 2023 ; Zhang et al. 2023 ’s ( Liu et al. 2023 ; Zhang et al. 2023 ) findings, utilizing the attention mechanism as an inherent evaluation criterion to assess the importance of KVCache entries and guide their eviction. However, using global attention scores often lacks specificity. Li et al. 2024b discover that the suffix of the input window exhibits behavior more consistent with the generation phase. By maintaining a local window to guide KVCache eviction, lower performance degradation is achieved. Due to conflicts between obtaining attention scores and existing acceleration techniques (such as FlashAttention ( Dao et al. 2022 ; Dao 2024 ) ), some methods focus on identifying alternative importance evaluation metrics. Devoto et al. 2024 identify a correlation between the norm of the key values and their importance, and propose an L2 norm-based KV cache eviction method that is compatible with existing acceleration frameworks. Building on this, Guo et al. 2024b also explore the norm of the value vectors and achieve more precise KV Cache eviction by combining the attention mechanism with the L2 norm of Value caches. Despite alleviating performance degradation, existing KV Cache eviction strategies continue to face the inconsistency problem in compressed inference illustrated in Figure 1 , thereby constraining the upper bound of eviction performance. 2.2 KV Cache Compression In addition to direct eviction, some studies ( Wan et al. 2024 ) focus on merging similar KV cache entries to improve efficiency. To preserve as much information from the prefill stage as possible, some methods ( Bolya et al. 2023 ; Zhang et al. 2024 ) build upon KV cache eviction and further explore token-level cache fusion, leading to improved task performance after eviction. Going further, with appropriate processing and training ( Sun et al. 2024 ; Liu et al. 2024b ; Zuhri et al. 2024 ; Ma et al.

Read the complete text