Quantize What Counts: More for Keys, Less for Values

Mohsen Hariri, Alan Luo, Weicong Chen, Shaochen Zhong, Tianyi Zhang, Qifan Wang, Xia Hu, Xiaotian Han, Vipin Chaudhary · 2025

Read Quantize What Counts: More for Keys, Less for Values by Mohsen Hariri, Alan Luo, Weicong Chen, Shaochen Zhong, Tianyi Zhang, Qifan Wang, Xia Hu, Xiaotian Han, Vipin Chaudhary (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15075.txt.

Description

Mohsen Hariri and co-authors prove that when a language model's memory cache is compressed, the keys deserve more bits than the values. Their 2025 paper attacks the inference bottleneck of large models: the stored key-value cache grows with model and context size, and shrinking it by quantization is the standard relief. How to split the bit budget between the two halves was guesswork.

Two theorems end the guessing. Key matrices provably carry larger norms, hence denser information, than value matrices. And for any fixed budget, favoring keys strictly lowers the compression error. Tests across models back the math. The title's advice is literal: quantize what counts.

Catalog record

CREATORMohsen Hariri, Alan Luo, Weicong Chen, Shaochen Zhong, Tianyi Zhang, Qifan Wang, Xia Hu, Xiaotian Han, Vipin Chaudhary
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15075

Browse the full library · About the Tower of Babel