IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference

Xintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Sirui Han, Yike Guo · 2026

Read IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference by Xintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Sirui Han, Yike Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25475.txt.

Introduction

Xintong Yang and eight colleagues make long-context language-model inference cheaper by learning which key-value entries to keep. Standard attention grows its key-value cache in line with sequence length, quickly becoming the bottleneck. Their learnable indexer predicts token value for eviction. A lightweight latent memory module compresses evicted tokens into a compact online state with residual readouts. The combination holds accuracy under a bounded cache, improving RULER scores by up to 25 points under aggressive eviction and fixing needle-in-a-haystack retrieval.

Catalog record

CREATORXintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Sirui Han, Yike Guo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58624
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25475

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference Xintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Sirui Han, Yike Guo Abstract Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference. A practical remedy is to evict less important KV entries; however, existing eviction policies are largely heuristic and struggle to capture the rich, input-dependent distribution of token importance. In this work, we introduce a learnable indexer that predicts KV importance, enabling more accurate retention of critical tokens. Meanwhile, naively evicting tokens permanently discards their information, leading to irreversible forgetting and degraded retrieval over long ranges. To address this, we propose a lightweight latent memory module that compresses evicted tokens into a compact, online-updated state and provides residual readouts to compensate for the attention contributions lost through KV eviction. Collectively, our method enables accurate long-context inference under a bounded KV budget, delivering consistent improvements on RULER (4K/16K) across Qwen, Mistral, and Llama models (up to 25 points under aggressive eviction), markedly more stable Needle-in-a-Haystack retrieval, and superior LongBench scores and compression curves compared to existing eviction policies. † † affiliationnotice: * Equal contribution. † Corresponding authors. Contact emails: Xintong Yang <[email protected]> , Hao Gu <[email protected]> . 1 Introduction Large Language Models (LLMs) have exhibited remarkable prowess in long context understanding, powering sophisticated applications such as complex reasoning over mathematical ( 43 ) and coding problems ( 21 ) , as well as agentic workflows like making slides ( 32 ) which demands large scale multimodal input processing ( 41 ; 9 ) . Leading models like Gemini3 ( 36 ) now support context windows of up to 1-million tokens, a scale readily encountered when ingesting video streams or extensive codebases. However, autoregressive generation poses a critical resource bottleneck for long context inference. To avoid the quadratic complexity of recomputing attention at each decoding step, modern inference engines leverage Key-Value (KV) caching to preserve intermediate states of previously processed tokens. While this strategy substantially reduces computational overhead, it displaces the bottleneck from computation to memory: the KV cache footprint scales linearly with both sequence length and batch size. For instance, maintaining a KV cache for a 1-million-token context in a 70-billion parameter model consumes approximately 320 GB of GPU memory ( 10 ) , far exceeding the capacity of most commodity accelerators. This ”memory wall” constrains not only the maximum deployable context length but also exacerbates decoding latency, as massive KV data movement saturates memory bandwidth. Recent advances in KV cache compression, such as quantization ( 30 ; 16 ) and low-rank approximation ( 4 ) , primarily reduce the per-token memory footprint. However, they do not address the cache’s inherent linear growth with context length and quadratic complexity of attention, and thus remain insufficient for truly long-context inference. We therefore focus on KV eviction, which directly bounds memory by retaining only the most valuable tokens. KV eviction faces two key challenges. First, it requires accurately predicting which tokens will remain important for future decoding. Most existing strategies are training free, relying on hand crafted proxy scores derived from simple static statistics or strong modeling assumptions, an approach that often fails to capture the richer, context dependent patterns governing future token usage. For example, SnapKV ( 28 ) estimates importance from a local attention window, which can bias retention toward nearby tokens; KeyDiff ( 35 ) uses inter-key dissimilarity as a proxy for informativeness; and Expected Attention ( 7 ) scores keys using “average” queries computed from corpus-level query statistics, an assumption that may not adapt to the specific context and query distribution encountered at inference time. Second, existing methods permanently discard evicted tokens, which cannot be retrieved if they become relevant later, incurring irreversible information loss. To address these challenges, we propose a learnable indexer that more accurately estimates token importance, coupled with a dedicated memory module that preserves information from evicted tokens for later retrieval. To summarize, our contributions are as follows: 1. We propose a learnable indexer that more accurately predicts the importance of KV tokens and enables adaptive KV eviction. 2. To mitigate irreversible forgetting caused by permanently discarding evicted tokens, we introduce a memory module that compresses evicted tokens into a fixed-size, compact latent memory, which is updated online during inference. 3. We conduct comprehensive experiments demonstrating that our method consistently improves long context performance, achieving strong gains on RULER across Qwen/Mistral/Llama (up to +25 points), more robust NIAH retrieval, and better LongBench score. 2 Background 2.1 Attention and KV Caching The Transformer architecture ( 42 ) serves as the foundational component of modern LLMs. It processes an input sequence X=(x_{1},x_{2},\ldots,x_{T})\in\mathbb{R}^{T\times d_{\text{model}}} through stacked Transformer blocks. Each block f(\cdot) sequentially applies causal self-attention and a feed-forward network (FFN) to produce the output sequence X^{\prime}=(x^{\prime}_{1},x^{\prime}_{2},\ldots,x^{\prime}_{T})\in\mathbb{R}^{T\times d_{\text{model}}} : X^{\prime}=f(X)=\mathrm{FFN}(\mathrm{Attention}(X)). (1) Causal self-attention. Given hidden states X , self-attention projects each token into query, key, and value vectors using projection matrices W_{q},W_{k},W_{v} : Q=XW_{q},\quad K=XW_{k},\quad V=XW_{v}. (2) The attention output is then computed as: O^{\mathrm{attn}}=\mathrm{Softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_{\text{model}}}}+Mask\right)V=AV, (3) where O^{\mathrm{attn}}\in\mathbb{R}^{T\times d_{\text{model}}} denotes the output matrix, A\in\mathbb{R}^{T\times T} is the attention matrix, and Mask is a causal mask with upper-triangular entries set to -\infty to prevent future token access. KV Caching. During autoregressive decoding, recomputing keys and values for all prior tokens x_{1},\ldots,x_{T} when processing a new token x_{T+1} is computationally inefficient. KV caching circumvents this redundancy by maintaining a cache \mathcal{C}=\{(k_{j},v_{j})\}_{j=1}^{T} of previously computed features. For the new token, only its query, key, and value vectors (q_{T+1},k_{T+1},v_{T+1}) are generated. The attention output for x_{T+1} is derived by aggregating the values from the updated cache using attention weights computed against the query: o^{\mathrm{attn}}_{T+1}=\sum_{j=1}^{T+1}\text{softmax}\Big(\frac{q_{T+1}K_{1:T+1}^{\top}}{\sqrt{d_{\text{model}}}}\Big)_{j}v_{j} where the Softmax is normalized over the causal context j\in\{1,\ldots,T+1\} . This formulation highlights that the output is a weighted sum of cached values. While KV caching substantially reduces decoding latency, the linear growth of \mathcal{C} with context length makes it the dominant memory consumer during long-context inference. Figure 1 : The overall pipeline of IndexMem is as follows: in the main attention stream, we use the learnable indexer to accurately select which KV tokens to save and evict the rest. The evicted tokens are then used to update a latent memory online , and the memory readout is added as a residual to compensate the main attention stream for the information lost due to eviction. 2.2 Formulation of KV cache Eviction Methods We can formulate most KV cache eviction methods in a unified scoring-and-selection framework. Given the cached keys and values K,V\in\mathbb{R}^{B\times H_{kv}\times L\times d_{\text{head}}} , an eviction policy first assigns an importance score to each cached token position: S=f(\cdot)\in\mathbb{R}^{B\times H_{kv}\times L}, where S_{b,h,t} measures the importance of token t for head h in sample b . The policy then keeps the top- L^{\prime} tokens and compacts the cache by gathering along the sequence dimension: \mathcal{I}=\mathrm{TopK}(S,L^{\prime}),\,[K^{\prime},V^{\prime}]=\mathrm{Gather}(K,V,\mathcal{I}). Here \mathcal{I} denotes the indices of retained tokens and \mathrm{Gather}(\cdot) selects entries along the token dimension. Different methods mainly differ in the definition of the score function f(\cdot) . SnapKV uses attention from the most recent window of queries

Read the complete text