SAW-INT4: System-AWare 4-Bit KV-Cache Quantization for Real-World LLM Serving
Jinda Jia, Jisen Li, Zhongzhu Zhou, Jung Hwan Heo, Jue Wang, Tri Dao · 2026
Read SAW-INT4: System-AWare 4-Bit KV-Cache Quantization for Real-World LLM Serving by Jinda Jia, Jisen Li, Zhongzhu Zhou, Jung Hwan Heo, Jue Wang, Tri Dao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19157.txt.
Introduction
Jia and ten colleagues pin down the smallest viable set of 4-bit KV quantization methods for language model serving under constraints such as paged layouts, regular access, and fused attention. Their finding: token INT4 quantization with block Hadamard rotation gives the best trade-off. It recovers nearly all accuracy lost by naive INT4, while vector and Hessian quantization add only marginal gains once compatibility counts. A fused rotation kernel fits into paged layouts with zero measurable overhead, matching INT4 throughput across concurrency levels.
Catalog record
| CREATOR | Jinda Jia, Jisen Li, Zhongzhu Zhou, Jung Hwan Heo, Jue Wang, Tri Dao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65578 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.19157
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SAW-INT4: System-AWare 4-Bit KV-Cache Quantization for Real-World LLM Serving Thanks: Project lead. Jinda Jia, Jisen Li, Zhongzhu Zhou, Jung Hwan Heo, Jue Wang, Tri Dao Abstract KV-cache memory is a major bottleneck in real-world LLM serving, where systems must simultaneously support latency-sensitive small-batch requests and high-throughput concurrent workloads. Although many KV-cache compression methods improve offline accuracy or compression ratio, they often violate practical serving constraints such as paged memory layouts, regular memory access, and fused attention execution, limiting their effectiveness in deployment. In this work, we identify the minimal set of 4-bit KV-cache quantization methods that remain viable under these constraints. Our central finding is that a simple design—token-wise INT4 quantization with block-diagonal Hadamard rotation—consistently achieves the best accuracy–efficiency trade-off. Across multiple models and benchmarks, this approach recovers nearly all of the accuracy lost by naive INT4, while more complex methods such as vector quantization and Hessian-aware quantization provide only marginal additional gains once serving compatibility is taken into account. To make this practical, we implement a fused rotation–quantization kernel that integrates directly into paged KV-cache layouts and introduces zero measurable end-to-end overhead, matching plain INT4 throughput across concurrency levels. Our results show that effective KV-cache compression is fundamentally a systems co-design problem: under real serving constraints, lightweight block-diagonal Hadamard rotation is a viable method that delivers near-lossless accuracy without sacrificing serving efficiency. Together AI Code: https://github.com/togethercomputer/saw-int4 1 Introduction Large language models (LLMs) have made remarkable progress in reasoning ( Wei et al. 2022 ) , tool use ( Qin et al. 2023 ) , and interaction with complex environments ( Yao et al. 2022 ) , driving adoption across an ever-widening range of applications ( Kaddour et al. 2023 ) . As these capabilities advance, long-context inference and agentic workloads have become increasingly important ( Belcak et al. 2025 ) ; frontier models now support context windows of millions or even tens of millions of tokens ( Comanici et al. 2025 ; Meta AI 2025 ) . This trend poses a fundamental challenge for key-value (KV) cache management, whose memory footprint grows linearly with sequence length ( Pope et al. 2023 ) . In production serving systems, KV cache memory is one of the primary constraints on throughput ( Kwon et al. 2023 ) , as it directly limits the number of requests that can be served concurrently on a given set of hardware. Architectural innovations such as grouped-query attention (GQA) ( Ainslie et al. 2023 ) and multi-head latent attention (MLA) ( Liu et al. 2024a ) reduce the per-token KV footprint relative to standard multi-head attention (MHA) ( Vaswani et al. 2017 ) , yet the absolute memory demand remains substantial at long context lengths. For example, Llama 4 Scout ( Meta AI 2025 ) has a model size of only 218 GB, but serving a single request at its maximum context length of 10M tokens requires roughly 1.8 TiB of KV cache memory alone—an order of magnitude larger than the model weights themselves. As context lengths continue to scale and inference-time compute becomes a more prominent scaling axis ( Snell et al. 2024 ) , this problem will only intensify. Consequently, KV cache compression has attracted extensive research attention ( Zhang et al. 2023 ; Liu et al. 2024c ; Zhang et al. 2024 ) . The System Gap in Existing KV Cache Compression. Despite achieving impressive compression ratios in isolation, existing KV cache compression methods face significant challenges integrating into real LLM serving systems and often fail to deliver end-to-end gains in practice. Production serving engines such as vLLM ( Kwon et al. 2023 ) , SGLang ( Zheng et al. 2024 ) , and TensorRT-LLM ( NVIDIA 2025 ) are built around tightly optimized primitives—PagedAttention ( Kwon et al. 2023 ) , continuous batching ( Yu et al. 2022 ) , and FlashAttention kernels ( Dao et al. 2022 ) —that leave very little room for additional overhead. Because autoregressive LLM decoding is predominantly memory-bandwidth-bound, even modest extra computation or irregular memory access during the decode phase translates directly into measurable latency increases and throughput degradation. Existing compression techniques violate these constraints in several ways. Methods such as KIVI ( Liu et al. 2024c ) and Kitty ( Xia et al. 2025 ) maintain a fixed-length residual buffer of unquantized key–value pairs alongside quantized tokens, creating a mixed-precision KV cache. However, PagedAttention manages cache memory in fixed-size, uniform-type blocks; accommodating two distinct precisions within the same paged pool requires either fragmented memory layouts or separate page tables, both of which complicate memory management and break the assumptions of existing fused attention kernels. Vector-quantization-based approaches ( Li et al. 2025a ; Zhang et al. 2024 ) face a different but equally problematic bottleneck: encoding and decoding through codebook lookups introduce irregular, data-dependent memory access patterns that are poorly suited to GPU execution and add non-trivial latency to every decoding step. More broadly, techniques that rely on channel-wise quantization, token eviction, or learned codebooks introduce cross-token dependencies or irregular access patterns that are difficult to incorporate into fused FlashAttention kernels and PagedAttention-based memory managers without substantial kernel-level modifications. In summary, many existing methods fundamentally violate the structural and execution constraints of modern LLM serving systems, leading to limited or negative end-to-end benefits despite strong offline accuracy results. This observation motivates a key question: Among the many proposed KV-cache compression techniques, which ones actually work under the constraints of real-world serving systems—and how simple can an effective solution be? Rather than proposing yet another compression method, this paper takes a step back and conducts a systematic empirical study of serving-compatible INT4 KV-cache quantization. We evaluate a broad spectrum of techniques—naïve token-wise INT4, vector quantization with varying codebook sizes, Hessian-aware quantization, orthogonal rotation, and hybrid combinations thereof—all under a unified, system-aware evaluation protocol that jointly measures accuracy and real serving throughput. Our finding is that Block-Diagonal Rotation (BDR) before token-wise INT4 quantization recovers nearly all accuracy lost by naïve quantization. We summarize our contributions: • Systematic empirical study of serving-compatible KV-cache quantization. We identify the key system constraints that practical KV-cache compression must satisfy (Section 2 ), and conduct a controlled comparison of five families of token-wise quantization techniques—naïve INT4, k-means vector quantization, Hessian-aware quantization, Hadamard rotation, and their combinations—across multiple models and benchmarks (Section 3 ). • Block-Diagonal Rotation (BDR) is the key enabler; complexity yields diminishing returns. Our experiments reveal that orthogonal rotation is the single most important ingredient for effective INT4 KV-cache quantization, recovering accuracy to within 1–3 points of BF16 on fragile models where naïve INT4 scores zero. More sophisticated methods (k-means with up to 2048 centroids, Hessian-aware calibration) provide only marginal improvements on top of rotation and introduce calibration overhead and deployment complexity that are difficult to justify. • Fused implementation with zero serving overhead. We develop a fused rotation–quantization CUDA kernel based on block-diagonal Hadamard transforms that integrates seamlessly into paged KV-cache layouts and FlashAttention-style decoding. End-to-end serving benchmarks on production workloads confirm that fused INT4 with rotation matches the throughput of plain INT4 within measurement noise, while preserving the latency and throughput advantage of INT4 over BF16. • Extensive evaluation across models and tasks. We validate our findings on Qwen3-4B, Qwen3-8B,