OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization
Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, Shuaiwen Leon Song, Ben Athiwaratkun, Xiaoxia Wu, [6pt] Together AI, [4pt] Equal contribution · 2026
Read OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization by Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, Shuaiwen Leon Song, Ben Athiwaratkun, Xiaoxia Wu, [6pt] Together AI, [4pt] Equal contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17757.txt.
Introduction
Zhongzhu Zhou and six colleagues present OSCAR, a method for squeezing language-model KV caches down to 2 bits. Plain rotations like Hadamard transforms tame outliers but still fail at 2 bits, since they ignore what attention downstream actually uses. OSCAR learns attention-aware covariance offline, derives fixed rotations and clipping limits from it, and adds a custom 2-bit attention kernel for paged serving. On Qwen3 models it nears full-precision accuracy, where naive rotation collapses near zero. It cuts cache memory about eightfold.
Catalog record
| CREATOR | Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, Shuaiwen Leon Song, Ben Athiwaratkun, Xiaoxia Wu, [6pt] Together AI, [4pt] Equal contribution |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105282 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17757
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, Shuaiwen Leon Song, Ben Athiwaratkun, Xiaoxia Wu, [6pt] Together AI, [4pt] Equal contribution Abstract INT2 KV-cache quantization is attractive for long-context LLM serving, but it remains difficult to make both accurate and deployable. Simple rotations such as Hadamard transforms reduce outliers, but still degrade at INT2 because they are not aligned with downstream attention. We propose OSCAR, an Ultra-low-bit KV Cache quantization method that estimates attention-aware covariance structures offline and uses them to derive fixed rotations and clipping thresholds for quantization. In this way, it aligns KV quantization with the covariance structures that attention actually consumes. More importantly, we not only provide theoretical justification but also develop a fully deployable OSCAR system with a custom INT2 attention kernel that remains compatible with paged KV-cache serving and fused kernel pipelines, enabling seamless integration into modern LLM serving frameworks such as SGLang and vLLM. We evaluate our methods on recent reasoning models with reasoning traces of up to 32k tokens across 5 tasks. On Qwen3-4B-Thinking-2507 and Qwen3-8B, OSCAR reduces the BF16 accuracy gap to 3.78 and 1.42 points, respectively, while naive rotation INT2 collapses to nearly zero. We further scale OSCAR to Qwen3-32B and GLM-4.7 (358B params), where it remains effectively on par with BF16. On long context - RULER-NIAH up to 128K, OSCAR remains robust on both Qwen3 models, while naive rotation INT2 collapses. System-wise, OSCAR reduces KV-cache memory by approximately 8 × 8\times , improves throughput by up to 7 × 7\times at large batch sizes under the same memory budget, and accelerates batch-size-1 decoding by up to 3 × 3\times over BF16 due to reduced memory bandwidth overhead. Code | Website | RotationZoo 1 Introduction Long-context inference has made the key–value (KV) cache one of the main costs of serving large language models. During autoregressive decoding, the cache grows with context length, batch size, and model depth, and every new token must read a large fraction of it from GPU memory. Compressing the KV cache is therefore a direct way to increase batch size and reduce memory traffic [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] . Among the available design points, INT2 quantization is especially attractive: it promises a large memory reduction while retaining a hardware-friendly fixed-width representation. Aggressively compressing KV caches to ultra-low precision (e.g., INT2) remains challenging because KV activations contain severe channel-wise outliers: a small subset of channels often exhibit extremely large magnitudes, while most channels remain relatively well-behaved [ 4 ] . Under low-bit quantization, these outliers dominate the quantization scale, compressing most normal values into only a few effective quantization levels and substantially degrading attention quality. Rotation-based quantization addresses this issue by applying a fixed orthogonal transform, such as a Hadamard rotation, that redistributes a few extreme activation values across many channels, producing a more uniform activation distribution that is easier to quantize [ 9 , 10 , 11 , 12 ] . In addition, rotation preserves tensor dimensionality and applies a fixed linear transform without introducing per-channel routing or irregular sparse metadata. This makes it naturally compatible with paged KV-cache layouts [ 13 ] , and FlashAttention-style fused decode kernels [ 14 , 15 , 16 , 17 ] : each KV vector is simply moved into a better-conditioned basis before quantization and moved back when used by page attention [ 12 ] . Figure 1 : OSCAR pipeline overview. Offline, OSCAR estimates attention-aware key/value covariance rotation and shows how the resulting rotation makes KV activations more uniform: Hadamard mixing flattens raw peaks, while the OSCAR rotation separates directions that matter more or less to attention. Online, the serving path keeps sink and recent tokens in BF16 while applying the fixed rotate–clip–INT2 path to history KV tokens inside the paged SGLang cache. The right panels summarize that OSCAR preserves accuracy across models while achieving throughput close to the QuaRot(INT2) serving path and higher throughput than BF16. However, a random rotation is still data-oblivious. It can smooth activation ranges, but it does not know which directions are important to attention. At INT2, this distinction matters: only four quantization levels are available, so the error should be pushed into directions that the model reads less strongly. Attention operates on the correlations and score-weighted interactions induced by keys and values, rather than on their raw Euclidean representations. This suggests the optimal rotation target should be derived from attention statistics themselves. Based on this observation, we propose OSCAR, an INT2 KV quantizer that estimates attention-aware covariance structures through a lightweight calibration pass and uses them to derive separate rotations for keys and values, along with per-layer clipping thresholds. Figure 1 (left) illustrates that while data-oblivious rotations can partially smooth outliers, they remain insufficient for INT2 quantization. In contrast, covariance-aware rotations produce substantially smoother activation distributions, enabling effective quantization. Our contributions are summarized as follows • We identify the missing target in INT2 rotation-based KV. Generic rotations mainly scatter activation outliers, but INT2 accuracy depends on the errors in attention scores and layer outputs; the rotation should be induced by downstream attention not by raw cache reconstruction alone. • We propose OSCAR, an attention-aware calibration framework for ultra-low-bit KV-cache quantization. OSCAR uses a lightweight calibration set to obtain attention-aware rotations for keys and values, enabling the quantized cache to better preserve downstream attention computation. A theoretical analysis is provided to show that the resulting covariance-target rotations are optimal under a natural frozen-error surrogate. Empirically, we evaluated across a wide range of state-of-the-art models from 4B to 400B, and retains near-BF16 accuracy at only 2.28 bits per KV element across multiple LLM families, including on a challenging code benchmark (LiveCodeBench). • We develop a production-ready INT2 KV-cache serving system. OSCAR preserves compatibility with paged and prefix KV-cache serving by keeping the bulk KV cache in a dense rotated INT2 representation. The system integrates into production SGLang decoding pipelines with customized Triton decoding kernels, such that one can fully utilize the prefix cache techniques [ 18 ] . Our system delivers up to 6.2\times higher throughput at 100k length and achieves roughly 2\times gains in both per-user speed and per-GPU throughput under full-cache workloads. This shows it is both user-friendly (lower latency) and system-efficient (higher GPU utilization). Due to space constraints, we provide a full discussion in Appendix C , D on how our ideas are inspired by and connected to prior work in the research community. 2 Preliminaries and Motivation Attention and KV cache. We use row-vector notation throughout. For simplicity, we define a single-head attention [ 19 ] as follows. Given a sequence T of hidden states \{x_{t}\}_{t=1}^{T} with x_{t}\in\mathbb{R}^{1\times d} and the weights W_{Q},W_{K},W_{V},\in\mathbb{R}^{d\times d} , we formulate the query, key, and value as Q=[q_{1};\dots;q_{T}]\in\mathbb{R}^{T\times d},K=[k_{1};\dots;k_{T}]\in\mathbb{R}^{T\times d},V=[v_{1};\dots;v_{T}]\in\mathbb{R}^{T\times d}, where d is the head dimension; q_{t}=x_{t}W_{Q}\in\mathbb{R}^{1\times d} , k_{t}=x_{t}W_{K}\in\mathbb{R}^{1\times d} , and v_{t}=x_{t}W_{V}\in\mathbb{R}^{1\times d} .The attention scores is defined as S=\mathrm{softmax}_{\mathrm{row}}(QK^{\top}/\sqrt{d})\in\mathbb{R}^{T\times T} , and the attention output is O=SV\in\mathbb{R}^{T\times d} , equivalently o_{i}=\sum_{t=1}^{T}S_{i,t}v_{t} . During autoregressive inference, K_{1:t}=[k_{1};\dots;k_{t}] and V_{1:t}=[v_{1};\dots;v_{t}] are stored in cache – this is the so-called KV cache. Covariance and more details are in Appendices A ,