BaKlaVa - Budgeted Allocation of KV cache for Long-context Inference
Ahmed Burak Gulhan, Krishna Teja Chitty-Venkata, Murali Emani, Mahmut Kandemir, Venkatram Vishwanath · 2025
Read BaKlaVa - Budgeted Allocation of KV cache for Long-context Inference by Ahmed Burak Gulhan, Krishna Teja Chitty-Venkata, Murali Emani, Mahmut Kandemir, Venkatram Vishwanath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13176.txt.
Description
Gulhan and co-authors wrote a paper in 2025 on BaKlaVa, a budgeted allocation approach for the key-value cache used by large language models processing long texts. The cache stores past computations so the model need not redo them, but it grows linearly with the length of the input, consuming scarce graphics memory. Earlier shortcuts treated the cache's many attention heads uniformly, clearing or compressing them by a single fixed rule.
BaKlaVa instead estimates the importance of each cache individually. A one-time profiling step then assigns each cache a share of a total memory budget. Important caches retain more of their data, while unimportant ones are trimmed harder. The team tested the approach on two widely used open models. It compressed the cache by up to seventy percent while keeping baseline performance, and at higher compression rates it improved accuracy by as much as tenfold over uniform methods.
Catalog record
| CREATOR | Ahmed Burak Gulhan, Krishna Teja Chitty-Venkata, Murali Emani, Mahmut Kandemir, Venkatram Vishwanath |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13176