VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

Yixuan Wang 1,2,3 , Qingyu Shi 4 , Jiayu Zhou 1 , Dianbo Liu 5 , Ziwei He 2 , Zhouhan Lin 1,2,3‡ 1 LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University 2 Shanghai Innovation Institute 3 Shanghai Artificial Intelligence Laboratory 4 University of Cambridge 5 National University of Singapore · 2026

Read VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization by Yixuan Wang 1,2,3 , Qingyu Shi 4 , Jiayu Zhou 1 , Dianbo Liu 5 , Ziwei He 2 , Zhouhan Lin 1,2,3‡ 1 LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University 2 Shanghai Innovation Institute 3 Shanghai Artificial Intelligence Laboratory 4 University of Cambridge 5 National University of Singapore (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16435.txt.

Description

Wang and co-authors shrink the key-value cache of large language models. Long contexts swell the cache and strain small machines. Older training-free tricks use low-rank fits or scalar quantization. Those fail to give high compression and high fidelity at once. Their VQKV method applies vector quantization instead. Thousands of values become a few integer indices. On LLaMA3.1-8B it compresses the cache by 82.8% while keeping 98.6% of baseline scores, and allows 4.3 times longer output on the same memory.

Catalog record

CREATORYixuan Wang 1,2,3 , Qingyu Shi 4 , Jiayu Zhou 1 , Dianbo Liu 5 , Ziwei He 2 , Zhouhan Lin 1,2,3‡ 1 LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University 2 Shanghai Innovation Institute 3 Shanghai Artificial Intelligence Laboratory 4 University of Cambridge 5 National University of Singapore
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34860
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16435

Browse the full library · About the Tower of Babel