Quantize What Counts: More for Keys, Less for Values
Mohsen Hariri, Alan Luo, Weicong Chen, Shaochen Zhong, Tianyi Zhang, Qifan Wang, Xia Hu, Xiaotian Han, Vipin Chaudhary · 2025
Read Quantize What Counts: More for Keys, Less for Values by Mohsen Hariri, Alan Luo, Weicong Chen, Shaochen Zhong, Tianyi Zhang, Qifan Wang, Xia Hu, Xiaotian Han, Vipin Chaudhary (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15075.txt.
Description
Mohsen Hariri and co-authors prove that when a language model's memory cache is compressed, the keys deserve more bits than the values. Their 2025 paper attacks the inference bottleneck of large models: the stored key-value cache grows with model and context size, and shrinking it by quantization is the standard relief. How to split the bit budget between the two halves was guesswork.
Two theorems end the guessing. Key matrices provably carry larger norms, hence denser information, than value matrices. And for any fixed budget, favoring keys strictly lowers the compression error. Tests across models back the math. The title's advice is literal: quantize what counts.
Catalog record
| CREATOR | Mohsen Hariri, Alan Luo, Weicong Chen, Shaochen Zhong, Tianyi Zhang, Qifan Wang, Xia Hu, Xiaotian Han, Vipin Chaudhary |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-15075