TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization

Dingyu Yao, Bowen Shen, Zheng Lin, Wei Liu, Jian Luan, Bin Wang, Weiping Wang · 2025

Read TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization by Dingyu Yao, Bowen Shen, Zheng Lin, Wei Liu, Jian Luan, Bin Wang, Weiping Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19586.txt.

Introduction

Dingyu Yao and co-authors slim the memory that language models need to chat over very long texts. Their 2025 paper targets the key-value cache, the stored context that swells as a conversation grows. Older fixes either ship the whole cache back and forth, which is slow, or squeeze it hard, which dulls the model.

The team noticed that layers differ: some must hold the whole picture and resist selective loading, while others lean on a few loud tokens that suffer most from compression. Their TailorKV method joins the two fixes, loading the loud tokens exactly and quantizing the rest. In tests it held near-lossless quality under strong compression, and it served a mid-size model with a huge context on one gaming-grade graphics card.

Catalog record

CREATORDingyu Yao, Bowen Shen, Zheng Lin, Wei Liu, Jian Luan, Bin Wang, Weiping Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization Dingyu Yao, Bowen Shen, Zheng Lin, Wei Liu, Jian Luan, Bin Wang, Weiping Wang Abstract 1 1 footnotetext: Work done during an internship at Xiaomi Inc. The Key-Value (KV) cache in generative large language models (LLMs) introduces substantial memory overhead. Existing works mitigate this burden by offloading or compressing the KV cache. However, loading the entire cache incurs significant latency due to PCIe bandwidth bottlenecks in CPU-GPU communication, while aggressive compression causes notable performance degradation. We identify that certain layers in the LLM need to maintain global information and are unsuitable for selective loading. In contrast, other layers primarily focus on a few tokens with dominant activations that potentially incur substantial quantization error. This observation leads to a key insight that loading dominant tokens and quantizing all tokens can complement each other. Building on this insight, we propose a hybrid compression method, TailorKV, which seamlessly integrates quantization and offloading. TailorKV develops an inference framework along with a hardware-friendly implementation that leverages these complementary characteristics. Extensive long-context evaluations exhibit that TailorKV achieves nearly lossless performance under aggressive compression settings, outperforming the state-of-the-art. Particularly, the Llama-3.1-8B with 128k context can be served within a single RTX 3090 GPU, reaching 82 ms per token during decoding 1 1 1 Code is available at: https://github.com/ydyhello/TailorKV . . 1 Introduction Large language models (LLMs) have demonstrated exceptional performance in tasks such as multi-turn dialogues Chiang et al. (2023) and multi-document understanding Bai et al. (2024) . In response to the growing complexity of tasks, recent LLMs have expanded their context windows to over 128k tokens, e.g., GPT-4 Achiam et al. (2023) and DeepSeek V3 Liu et al. (2024a) . Typically, the inference of LLMs is auto-regressive, with the Key-Value (KV) cache stored in memory to avoid recomputation. However, the size of KV cache grows linearly with sequence length, leading to much higher GPU memory consumption and inference latency. Recent studies have proposed sparse attention mechanisms to reduce KV cache usage. These methods fall into two categories: irreversible eviction and recallable selection. Irreversible eviction methods Li et al. (2024) ; Zhang et al. (2023) ; Xiao et al. (2024b) suffer from accuracy degradation due to permanently discarding tokens that may later become crucial, particularly in multi-turn dialogues. Recallable selection methods adopt a different approach by maintaining the entire KV cache while selecting only a subset of tokens for processing. However, methods like Quest Tang et al. (2024) and SparQ Ribar et al. (2024) encounter memory limitations when attempting to store all tokens on the GPU. Although CPU offloading mitigates GPU memory limitations, existing approaches Xiao et al. (2024a) ; Zhang et al. (2024a) still require retrieving a substantial portion of tokens (around 20%), introducing significant decoding latency overheads due to slow data transfer between CPU RAM and GPU RAM. To optimize accuracy, memory, and latency simultaneously, we first analyze the compression preferences for the KV cache based on layer characteristics. Prior researches Feng et al. (2024) ; Cai et al. (2024) applied different sparsity rates to different layers under the same compression strategy. However, our analyses demonstrate that performance degradation primarily stems from the application of unsuitable compression at the layer level (Section 3 ). Therefore, we suggest that shallow layers, which exhibit dense attention patterns and emphasize global information Wan et al. (2025) , are better suited for uniform compression like quantization. Conversely, layers with a few dominant tokens and largely redundant information are well-suited for sparsity, as performance can be maintained by retrieving only the dominant tokens. Building upon these insights, we propose a novel framework, TailorKV , which employs hybrid compression techniques to reduce GPU memory usage. We introduce an identification metric to classify Transformer layers into two distinct types: (i) quantization-friendly layers, which preserve global information from a macro perspective, and (ii) sparsity-friendly layers, which capture crucial information from a micro perspective. This design enables quantization-friendly layers to employ static quantization, achieving a high compression ratio (1-bit per floating-point number) while maintaining model quality. Meanwhile, for sparsity-friendly layers, the system offloads the KV cache to CPU memory during prefilling and dynamically retrieves the Top-K tokens during decoding. By aligning compression strategies with the characteristics of each layer, this tailored approach significantly reduces overall memory consumption. The accuracy and efficiency of TailorKV are evaluated on various backbone LLMs using long-context benchmarks. The results demonstrate that TailorKV drastically reduces memory usage by quantizing 1 to 2 layers to 1-bit precision and loading only 1% to 3% of the tokens for the remaining layers while maintaining nearly lossless performance. Specifically, TailorKV achieves a decoding latency of 82 ms for Llama-3.1-8B with a 128k-context on a single RTX 3090 (PCIe 1.0) 2 2 2 We combine TailorKV with 4-bit weight-only quantization Lin et al. (2024) for prefill phase memory allocation. , yielding a 53.7% reduction in peak GPU memory usage. The key contributions are summarized as follows. • We identify layer-specific compression preferences and develop an identification metric to determine optimal compression strategies for different layers in the model. • We present TailorKV, a hybrid KV cache compression framework that combines quantization and offloading techniques through an algorithm-system co-design, preserving both model accuracy and execution efficiency. • Extensive experiments on long-context benchmarks demonstrate the nearly lossless performance of TailorKV with minimal GPU memory consumption and acceptable latency. 2 Preliminaries (a) Dense (left) and sparse (right) attention. (b) Sparse error on different LLMs. (c) Sparse error on different datasets. Figure 1: Observations on attention. (a) Attention weights on Llama-2-7B-32K-Instruct. Detailed visualizations are in Appendix I . (b) Sparse error of different models on the 2WikiMQA dataset, with only the top 5% of attention scores retained. (c) Sparse error on different datasets, with only the top 5% of attention scores retained. 2.1 Attention and KV Cache LLM inference consists of two stages: prefill and decode . During prefilling, the entire prompt is used to generate the first token. Consider the prompt embedding \mathbf{X}\in\mathbb{R}^{n\times d} along with the weight matrices \mathbf{W}_{i}^{q},\mathbf{W}_{i}^{k},\mathbf{W}_{i}^{v}\in\mathbb{R}^{d\times d_{h}} for head i\in[1,h] , where n is the sequence length, d is the hidden dimension and d_{h} is the head dimension. The keys and values for head i are computed and cached, as follows: \mathbf{K}_{i}=\mathbf{XW}_{i}^{k},~\mathbf{V}_{i}=\mathbf{XW}_{i}^{v}. (1) During decoding, the new token embedding \mathbf{x}\in\mathbb{R}^{1\times d} is computed iteratively to produce the query, key, and value vectors. The cache is updated and the output \mathbf{o} of each attention head is computed as: \displaystyle\mathbf{K}_{i}\leftarrow\mathrm{Cat}[\mathbf{K}_{i},\mathbf{x}\mathbf{W}_{i}^{k}],\mathbf{V}_{i}\leftarrow\mathrm{Cat}[\mathbf{V}_{i},\mathbf{x}\mathbf{W}_{i}^{v}], (2) \displaystyle\mathbf{a}_{i}=\mathrm{Softmax}\bigl(\mathbf{q}_{i}\mathbf{K}_{i}^{\top}/\sqrt{d_{h}}\bigr),\mathbf{o}_{i}=\mathbf{a}_{i}\mathbf{V}_{i}, (3) where \mathbf{q}_{i}=\mathbf{x}\mathbf{W}_{i}^{q} , and the attention outputs from all heads are concatenated and sent to the FFN. 2.2 Quantization of KV Cache Quantization converts continuous or high-precision values into lower-precision discrete representations. Given a tensor \mathbf{X} in high precision, the typical uniform quantization process can be expressed as: \displaystyle\mathbf{X}_{Q}= \displaystyle\mathrm{Quant}_{b}(\mathbf{X},s,z) (4) \displaystyle= \displaystyle\mathrm{clamp}(\lfloor\frac{\mathbf{X}-z}{s}\rceil,0,2^{b}-1), where \mathbf{X}_{Q} represents the quantized tensor in b -bit precision, with z=\min\mathbf{X} as the zero-point and s=\frac{\max\mathbf{X}-\min\mathbf{X}}{2^{b}-1} as the scaler. The clamp function restricts values to the b -bit integer range and \lfloor\cdot\rceil denotes the rounding function. 2.3 GPU-CPU Co-execution As the sequence length increases, the size of the KV cache grows, significantly raising the demand for GPU resources. For example, with a sequence length of 512k, Llama-2-7B

Read the complete text