KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference

Sai Gokhale, Devleena Das, Rajeev Patwari, Ashish Sirasao, Elliott Delaye · 2025

Read KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference by Sai Gokhale, Devleena Das, Rajeev Patwari, Ashish Sirasao, Elliott Delaye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01953.txt.

Introduction

Long documents strain large language models because the key-value cache, their running memory of past tokens, grows in a straight line with text length. Sai Gokhale and four co-authors study in 2025 how to shrink that memory and the model itself, especially for deployment on small edge devices.

Single tricks are well explored: quantizing the cache, chunking the input, or quantizing the model's weights. What happens when they are combined, and which combination is best, was largely unknown. The authors examine these techniques jointly at the systems level, mapping the trade-offs. Their map helps engineers fit long-context models into tight memory budgets.

Catalog record

CREATORSai Gokhale, Devleena Das, Rajeev Patwari, Ashish Sirasao, Elliott Delaye
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01953

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference Sai Gokhale, Devleena Das, Rajeev Patwari, Ashish Sirasao, Elliott Delaye Abstract Long-context Large Language Models (LLMs) face significant memory bottlenecks during inference due to the linear growth of key-value (KV) cache with sequence length. While individual optimization techniques like KV cache quantization, chunked prefill, and model weight quantization have shown promise, their joint effects and optimal configurations for edge deployment remain underexplored. We introduce KV Pareto, a systems-level framework that systematically maps the trade-off frontier between total memory consumption and task accuracy across these three complementary optimization techniques. Our framework evaluates multiple LLM architectures (Qwen, Llama, Mistral) with varying KV quantization schemes (int2/4/8, mixed-precision), granularities (per-token, per-tensor, per-block), and 4-bit weight quantization via AWQ. Our framework identifies model-specific Pareto-optimal configurations that achieve 68-78% total memory reduction with minimal (1-3%) accuracy degradation on long-context tasks. We additionally verify the selected frontiers on additional benchmarks of Needle-in-a-Haystack, GSM8k and MMLU as well as extended context lengths of up to 128k to demonstrate the practical need of joint optimization for efficient LLM inference. 1 Introduction Large Language Models (LLMs) have become useful in many applications, such as code generation Jiang et al. 2024b , long question-answering Liu et al. 2025 and retrieval-augmented generation (RAG) Arslan et al. 2024 . These tasks increasingly demand longer-context capabilities, pushing models like Qwen Bai et al. 2023 , Mistral Jiang et al. 2024a and Llama Grattafiori et al. 2024b to support long context lengths. The bottleneck for efficient inference arises from the transformer architecture which operates primarily in two phases: prefill and decode Raiaan et al. 2024 . During prefill, the input context is processed and stored in the key-value (KV) cache. During decode, outputs are generated autoregressively by repeatedly accessing the KV cache. Importantly, the KV Cache size grows linearly with sequence length Patwari et al. 2025 and increases time-to-first-token (TTFT) during prefill, and time-per-output-token (TPOT) during decode. The resulting increased latency is a bottleneck for practical deployment of long-context models on edge devices. To reduce the latency and scalability challenges, several optimization techniques have been proposed for long-context inference, including KV Quantization Hooper et al. 2024 ; Li et al. 2025 ; Liu et al. 2024b , token eviction Xiao et al. 2023 ; Corallo and Papotti 2024 , and chunked prefill Agrawal et al. 2023 . These methods are typically evaluated in isolation of one another in the context of accuracy degradation Li et al. 2025 ; Liu et al. 2024b ; Corallo and Papotti 2024 . This creates a practical question for deployment: which memory optimizations, together, offer the best-trade offs between memory savings and task accuracy? To this end, we introduce KV Pareto , a framework for evaluating and understanding the trade-offs between KV memory compression and task performance in long-context LLMs. KV Pareto focuses on studying the impact of two widely accessible optimization techniques for long context, KV quantization, and chunked prefill in conjunction with 4-bit model weight quantization. Prior work evaluates these optimization techniques in isolation Li et al. 2025 ; Liu et al. 2024b ; Corallo and Papotti 2024 ; Lin et al. 2024 ; Agrawal et al. 2023 . Instead, our KV Pareto provides a joint assessment of optimization techniques, considering total memory savings and accuracy degradation. This enables practitioners to identify the Pareto-optimal configurations for edge deployment. Our KV Pareto spans multiple models (Mistral, Qwen, LLaMA), KV cache quantization granularities (per-token, per-tensor, per-block), group sizes (32, 64, 128), precision formats (int2, int4, int8), as well 4-bit weight quantization via AWQ Lin et al. 2024 . We benchmark across long context evaluations including LongBench Bai et al. 2024b , Needle-in-a-Haystack (NIAH) Kamradt 2023 , and traditional tasks such as GSM8k Cobbe et al. 2021 and MMLU Hendrycks et al. 2020 , measuring total memory through peak activation memory, KV memory, and model memory, as well as task accuracy. Our contributions are: 1. KV Pareto Framework: A KV optimization Pareto framework that systematically maps the trade-off search space between total memory savings and task accuracy. 2. Joint Optimization Study: A comprehensive evaluation of chunked prefill, KV cache quantization and AWQ weight-only quantization across multiple KV cache quantization granularities and precisions, identifying Pareto-optimal configurations using LongBench. 3. KV Pareto Validation: Validation our framework’s selected frontiers on NIAH, showing strong task performance even at 20-32k context lengths, as well as MMLU, GSM8k. 2 Related Works 2.1 KV Quantization KV quantization reduces the precision of stored key and value tensors, thereby lowering memory usage Li et al. 2024 . For example, KIVI Liu et al. 2024b and KVQuant Hooper et al. 2024 introduce tuning-free asymmetric quantization schemes that apply per-channel and per-token quantization, achieving up to 2-bit compression. More recently, KVTuner Li et al. 2025 proposes an adaptive framework that searches for the optimal layer-wise KV quantization precision pairs and demonstrates near lossless 3.25 bit mixed precision KV quantization for mathematical reasoning tasks. Inspired by KVTuner Li et al. 2025 , our KV Pareto framework also considers mixed-precision quantization schemes. While KVTuner Li et al. 2025 focuses on layerwise adaptability, our KV Pareto framework focuses on additional important quantization hyperparameters such as, quantization scheme (blockwise, per tensor, per token), as well as system-level interactions with other optimizations such as, PC and model quantization. 2.2 Prefill Chunking Prefill chunking (PC) reduces the peak memory consumption by dividing input prompts into equal-sized segments that are processed sequentially Agrawal et al. 2023 . PC is adopted in inference systems like vLLM Kwon et al. 2023 . Additionally, follow on works such as WiM Russak et al. 2024 leverage the concept of smaller chunks to improve model reasoning. However, the benefits and effects of PC has not yet been studied in conjunction with model quantization as well as KV cache quantization. We address this gap by analyzing PC at a systems-level, understanding its tradeoffs when combined with KV cache and model quantization. 2.3 Model Quantization Model quantization algorithms are popular for efficient inference as they significantly compress model size. Popular methods include, GPTQ Frantar et al. 2023 , and AWQ Lin et al. 2024 . GPTQ Frantar et al. 2023 uses second-order information derived from the Hessian matrix to enable 3-4 bit quantization. AWQ Lin et al. 2024 preserves accuracy in 4-bit quantization by using activation metrics to identify the most important weight channels, which are then scaled prior to quantization to reduce error. In our framework, we study model memory savings via AWQ Lin et al. 2024 to provide practical insights for edge deployment on how model compression, with KV quantization and PC, can provide the most memory savings with the least task performance degradation. 3 Background The KV cache is a crucial component for LLM inference, storing intermediate representations that are used for autoregressive generation. KV cache memory can be represented as the follows: \text{KV Cache Memory}=B\times H\times N\times D\times L\times s , where B is batch size, H is number of attention heads, N is number of tokens stored in the cache, L is number of layers, D is head dimension and s is the size per element. At a systems level, KV cache optimizations for edge deployment remain

Read the complete text