Hard-KV : Head-Adaptive Regularization for Decoding-time KV Compression
Yuxuan Yang, Feiyang Ren, Bowen Zeng, Dalin Zhang, Jinpeng Chen, Gang Chen, Huan Li · 2026
Read Hard-KV : Head-Adaptive Regularization for Decoding-time KV Compression by Yuxuan Yang, Feiyang Ren, Bowen Zeng, Dalin Zhang, Jinpeng Chen, Gang Chen, Huan Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28831.txt.
Introduction
Yang, Ren, Zeng, Zhang, Chen, Chen and Li present Hard-KV, a method for shrinking the memory that language models keep while generating long answers. That memory, the key-value cache, grows with text length, and smart schemes that vary its size per attention head clash with inference engines that demand fixed memory layouts. Their framework manages stored tokens in three tiers, from dense to sparse to condensed, as generation proceeds.
A logits calibration step converts different importance scores into one shared probability scale, so a single top-p budget, keeping the most probable share, can be applied fairly across unlike heads. A system-level rewrite then packs the chosen scattered indices into continuous memory blocks that fast engines accept. On maths reasoning benchmarks the method doubled throughput compared with static baselines. It also kept generation quality high on inputs longer than ten thousand tokens, with code released publicly.
Catalog record
| CREATOR | Yuxuan Yang, Feiyang Ren, Bowen Zeng, Dalin Zhang, Jinpeng Chen, Gang Chen, Huan Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.28831
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hard-KV : Head-Adaptive Regularization for Decoding-time KV Compression Yuxuan Yang, Feiyang Ren, Bowen Zeng, Dalin Zhang, Jinpeng Chen, Gang Chen, Huan Li Abstract Long-context LLM inference faces a fundamental conflict: head-adaptive compression algorithms (e.g., Top- p p nucleus sampling) offer superior accuracy by dynamically fluctuating memory budgets, yet modern inference engines (e.g., vLLM) demand rigid, static memory patterns to leverage CUDA Graphs and PagedAttention. We resolve this “Static-Dynamic” mismatch with Hard-KV , a unified framework that that bridges dynamic selection with rigid system constraints. Hard-KV introduces a Cascade Cache hierarchy, managing the token lifecycle across dense, sparse, and condensed tiers. Crucially, we propose a Logits Calibration mechanism that normalizes diverse importance metrics into a unified probability space, enabling consistent Top- p p budgeting across heterogeneous heads. To bridge the efficiency gap, we offer a system-level solution, which rewrites fragmented, dynamic indices into contiguous physical layouts compatible with high-performance inference engine. Extensive experiments on math-reasoning benchmarks (AIME, U-Math) verify that Hard-KV achieves up to 2 × \times throughput improvement over static baselines while maintaining high-fidelity generation in 10k+ token scenarios. Code is available at https://github.com/SuDIS-ZJU/HARDInfer . 1 Introduction The deployment of Large Language Models (LLMs) in long-context scenarios is increasingly constrained by the linear growth of Key-Value (KV) cache memory. As sequence lengths extend into the tens of thousands, the memory footprint of the KV cache frequently exceeds that of the model weights, creating a critical bottleneck for throughput and latency. To mitigate this, decoding-time compression methods ( 28 ; 38 ; 4 ; 23 ; 49 ; 51 ) have emerged, dynamically reducing memory usage by retaining only the most critical tokens during generation. A central tension in this domain lies between algorithmic accuracy and system efficiency. Recent research ( 17 ; 39 ; 35 ) indicates that “head-adaptive” strategies—which allow cache size to fluctuate based on the specific information density of each attention head—offer superior performance. This accuracy is crucial for extending compression to decoding time, particularly for tasks like mathematical reasoning where errors accumulate over long generation trajectories ( 6 ) . 29 argue that the superiority of head-adaptive budget allocation stems from the intrinsic dynamism of attention, demonstrating that simple-yet-effective Top- p (nucleus sampling ( 20 ) ) is sufficient for selection. Similarly, other works ( 42 ; 16 ; 14 ) propose new criteria for performance-prioritized compression that rely on dynamic allocation. However, this dynamic nature is fundamentally incompatible with modern inference engines. The uneven, unpredictable, and fragmented memory indices generated by head-adaptive selection conflict with standard optimizations such as PagedAttention ( 26 ) , continuous batching ( 47 ) , and CUDA Graphs ( 32 ) , all of which rely on static or regularized memory block structures. Challenge: We characterize the core conflict as the gap between Dynamic, Context-Dependent Attention Patterns and Static, Request-Independent System Designs . To bridge this gap, we propose Hard-KV ( H ead- A daptive R egularization for D ecoding-time KV Compression), a framework designed to enable dynamic cache allocation within the constraints of modern inference architectures. To bridge this gap, Hard-KV introduces a Cascade Cache architecture that organizes KV memory into three sequence-level tiers. We further propose a Logits Calibration procedure that converts Top- k selections into Top- p -aligned counterparts, allowing diverse KV selection strategies to share a unified head-adaptive budget allocation mechanism. This calibration produces cache patterns that better follow the raw attention distribution, making them naturally compatible with the three-tier cache. Finally, Hard-KV regularizes dynamic head-wise indices through a customized inference system that combines memory rewriting and sparse loading. This design reconciles adaptive token selection with the structural constraints of modern inference engines, preserving algorithmic accuracy while improving system efficiency. On math-reasoning benchmarks, including AIME and U-MATH, Hard-KV achieves up to a 1.5 \times performance gain and a 2 \times throughput improvement. To summarize, our work unifies head-adaptive KV cache management into a robust system that is both algorithmically performant and system-efficient We summarize our technical contributions as follows: 1. Unified Cascade Cache Architecture: We introduce a multi-tier memory hierarchy that standardizes the token lifecycle. This structure accommodates various KV selection methods, unifying diverse compression techniques into a single pipeline. (Section 3.1 ) 2. Logits Calibration for Effective Top- p Budgeting: We propose a calibration method connecting Top- k and Top- p selection. By mapping distribution alterations onto the raw attention distribution, we enable Top- p budgeting to dynamically adapt to attention patterns rather than relying on fixed limits. (Section 3.2 ) 3. System-Aware Index Regularization: We provide a method to integrate dynamic compression into rigid system architectures. Our regularization for head-adaptive indices resolves compatibility issues with CUDA Graphs, ensuring high system efficiency with minimal metadata overhead.(Section 3.3 ) 2 Related Work Adaptive KV Cache Compression KV cache compression ( 27 ) optimizes LLM inference SLOs (e.g., latency, throughput) by reducing memory footprints during attention computation. Unlike Efficient Attention methods ( 44 ; 1 ; 18 ) designed for full-stack industrial deployment, KV cache compression has largely remained a research testbed, lacking robust integration with modern inference engines. Recent studies ( 17 ; 15 ; 13 ; 50 ) have pivoted from uniform compression to head-adaptive strategies , driven by the observation of attention heterogeneity. Methods like AdaKV ( 15 ) and HeadKV ( 17 ) implement head-wise allocation based on metrics like eviction loss and importance scores. Similarly, 13 use policy optimization to mix full and sparse attention, showing that heterogeneous configurations outperform uniform baselines. Concurrently, approaches like Twilight ( 29 ) and MagicPig ( 9 ) have shown that Top- p (cumulative probability) selection yields higher fidelity than rigid Top- k (fixed count) truncation. Despite the superiority of Top- p selection in ideal scenarios, robustly integrating these dynamic strategies remains a challenge: existing methods often decouple sampling from memory management, while diverse importance metrics (e.g., L2 norm vs. attention scores) defy unified probabilistic budgeting. Hard-KV bridges this gap with a logits calibration mechanism that normalizes diverse selection metrics into a unified probability space, enabling consistent Top- p budgeting across heterogeneous heads while enforcing strict, system-compatible regularity on the resulting memory layout. System-Algorithm Co-Design in LLM Inference State-of-the-art inference engines maximize throughput by enforcing rigid, predictable execution patterns. Innovations such as RadixAttention ( 55 ) , Tree-Attention ( 5 ) , and static load-balancing ( 30 ) demonstrate the efficacy of co-designing algorithms with static system constraints. Likewise, deploying head-adaptive KV compression introduces an algorithm-system conflict: the memory allocations vary at every step, violating the static assumptions of engines built for regular allocation. Although head-wise sparse attention is computationally feasible via kernel support like FlashInfer ( 45 ) , we identify three systemic barriers that prevent theoretical compression rates from actual speedups: 1) CUDA Graphs Incompatibility: CUDA Graphs rely on static tensor shapes and persistent pointers. The variable cache sizes produced by Top- p selection force the system to either frequently recapture graphs or revert to eager execution, neutralizing latency gains. 2) PagedAttention Fragmentation: Current paging mechanisms in real-time LLM serving ( 47 ; 26 ) assume uniform memory distribution. Head-adaptive eviction shatters this uniformity, causing severe fragmentation that breaks the block contiguity required for efficient virtual-to-physical mapping. 3) KV Selection Overhead: Standard KV selection