DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference

Zahra Dehghanighobadi, Asja Fischer · 2026

Read DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference by Zahra Dehghanighobadi, Asja Fischer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24647.txt.

Introduction

Dehghanighobadi and Fischer address the memory bottleneck of long-context language-model inference: the key-value cache grows linearly with sequence length. Existing pruning methods cut cached tokens by attention score but apply one pruning ratio to every layer. The authors show layers differ sharply in pruning sensitivity. Their DepthKV framework allocates a fixed global cache budget across layers by sensitivity. Across models and tasks it consistently beats uniform pruning at the same global ratio.

Catalog record

CREATORZahra Dehghanighobadi, Asja Fischer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78195
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.24647

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference Zahra Dehghanighobadi, Asja Fischer Abstract Long-context reasoning is a critical capability of large language models (LLMs), enabling applications such as long-document understanding, summarization, and code generation. However, efficient autoregressive inference relies on the key–value (KV) cache, whose memory footprint grows linearly with sequence length, leading to a major memory bottleneck. To mitigate this overhead, KV cache pruning methods discard cached tokens with low attention scores during inference. Most existing methods apply a uniform pruning ratio across layers, implicitly assuming that all layers contribute equally to overall model performance. We show that this assumption is suboptimal, as layers differ significantly in their sensitivity to pruning. We propose DepthKV , a layer-dependent pruning framework that allocates a fixed global KV budget across layers based on their sensitivity, rather than using a uniform allocation. Across multiple models and tasks, DepthKV consistently outperforms uniform pruning at the same global pruning ratio, demonstrating more effective utilization of the KV cache budget through layer-dependent allocation. github.com/zahra-dehghani/depthkv 1 Introduction Figure 1: Uniform vs. layer-dependent KV allocation. Uniform allocation (left) assigns an equal KV budget across transformer layers. DepthKV (right) reallocates this budget based on sensitivity to pruning, retaining more tokens in critical layers (highlighted) and pruning less important ones more aggressively. Token rank denotes relative importance. Recent advances in large language models (LLMs) have greatly increased context window sizes, ranging from 128K to millions of tokens ( 32 ; 11 ) . Larger context windows enable applications such as long-form reasoning, agent-based workflows, and large-scale document retrieval, where relevant information is often sparsely distributed across lengthy inputs. However, in autoregressive LLMs, each generated token attends to all previous tokens via self-attention. Decoding scales linearly with context length and is repeated at every generation step, while the prefill stage processes the entire input with quadratic complexity. Therefore, as context windows grow, long-context inference becomes prohibitively expensive. To avoid redundant computation, modern LLMs cache key–value (KV) representations of previous tokens. While this eliminates recomputation during decoding, it introduces a new bottleneck: the KV cache grows with both sequence length, quickly exceeding GPU memory capacity in long-context settings ( 38 ) . As a result, the primary bottleneck shifts from computation to memory. Prior work addresses this challenge at different stages of the model lifecycle. Training-phase approaches alter the attention architecture but typically require retraining ( 27 ; 2 ; 4 ) . Deployment-stage methods focus on optimizing how the KV cache is stored and accessed at the system level, such as memory layout and hardware placement, without changing its values or the model’s computations ( 18 ; 20 ; 43 ) . In contrast, post-training approaches directly modify the KV cache representation, for example through eviction, merging, or quantization, and often introduce approximations to improve efficiency ( 44 ; 36 ; 41 ; 13 ) . Training-phase methods are difficult to apply to existing pretrained models, while deployment-stage approaches mainly optimize memory access rather than reducing KV cache size. In contrast, post-training methods directly reduce memory usage during inference, making them particularly practical for long-context settings ( 28 ) . Therefore, we focus on post-training KV cache pruning. Most existing post-training methods prune the KV cache uniformly across transformer layers, implicitly assuming that all layers are equally important. However, prior work ( 30 ) suggests that intermediate transformer layers may play a more critical role than early or late layers. To examine whether such non-uniformity persists under KV cache pruning, we conduct a layer-wise ablation study in which pruning is applied to one layer at a time while keeping others unchanged, and measure the resulting performance degradation. A permutation test consistently rejects the hypothesis of uniform layer importance across models and datasets, demonstrating that transformer layers contribute unevenly to long-context performance. We further analyze how layer removal affects generation behavior. As shown in Section 4.1 , layers that are most sensitive in the ablation study also lead to shorter and less informative outputs when pruned, indicating that their impact on performance is closely tied to their role in sustaining content generation. Motivated by these findings, we propose DepthKV , a framework for layer-dependent KV cache pruning that allocates the memory budget non-uniformly across transformer layers based on their importance for long-context performance. As illustrated in Figure 1 , uniform allocation retains the same number of tokens across all layers, whereas DepthKV assigns a larger budget to more important layers while pruning less critical ones more aggressively, thereby preserving key information under a fixed memory constraint. The framework supports multiple allocation strategies, including position-dependent protection (preserving specific regions such as middle layers), metric-guided allocation (allocating more budget to layers ranked higher by a scoring metric), and hybrid strategies that combine these rules within a unified budget. We evaluate DepthKV on long-context tasks including document summarization, question answering, and mathematical reasoning, where inputs are substantially longer than outputs, making the prefill stage the dominant source of computation and memory usage. In this setting, pruning during prefill directly targets the primary bottleneck while maintaining stable decoding. Across all tasks and models, layer-dependent allocation consistently outperforms uniform pruning under the same KV budget. In summary, our main contributions are as follows: • We show that transformer layers exhibit statistically significant variation in importance for long-context inference, challenging the implicit assumption underlying uniform KV cache pruning. • We identify content amplification layers , whose pruning suppresses content generation and strongly correlates with downstream performance degradation. • We propose DepthKV , a layer-dependent KV cache allocation framework that redistributes a fixed memory budget across transformer layers based on their importance. • We demonstrate that DepthKV consistently outperforms uniform KV cache pruning across diverse long-context tasks under the same memory budget. 2 Related Work Building on the discussion in the introduction, we focus on post-training KV cache pruning methods. Within this setting, existing approaches can be categorized along two complementary dimensions: (i) how token importance is estimated, and (ii) whether this importance depends on the current decoding query. The first distinguishes between heuristic and learned methods, while the second separates query-aware from non-query-aware strategies. Heuristic methods. Heuristic approaches estimate token importance using predefined rules, typically based on positional bias, aggregated attention statistics, or attention profiling across heads or tokens. Representative methods include H_{2}O ( 44 ) , StreamingLLM ( 40 ) , SnapKV ( 19 ) , and FastGen ( 9 ) . H_{2}O retains attention-dominant tokens alongside recent context. StreamingLLM preserves initial attention-sink tokens (i.e., tokens that consistently attract high attention, such as the first token <bos> ), together with a sliding window of recent tokens. SnapKV selects prefix tokens based on attention patterns computed from an observation window near the end of the prompt, while FastGen derives head-specific retention policies from attention profiling to determine which tokens to preserve. While these methods are computationally efficient, their reliance on heuristic rules may limit generalization under shifts in the input distribution. Learned methods. In contrast, learned approaches estimate token importance directly from data rather than predefined rules, enabling them to better capture semantic and task-specific relevance. Representative examples include DuoAttention ( 39 ) and SeerAttention ( 7 ) . DuoAttention learns per-head gates to separate retrieval heads

Read the complete text