BaKlaVa - Budgeted Allocation of KV cache for Long-context Inference

Ahmed Burak Gulhan, Krishna Teja Chitty-Venkata, Murali Emani, Mahmut Kandemir, Venkatram Vishwanath · 2025

Read BaKlaVa - Budgeted Allocation of KV cache for Long-context Inference by Ahmed Burak Gulhan, Krishna Teja Chitty-Venkata, Murali Emani, Mahmut Kandemir, Venkatram Vishwanath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13176.txt.

Description

Gulhan and co-authors wrote a paper in 2025 on BaKlaVa, a budgeted allocation approach for the key-value cache used by large language models processing long texts. The cache stores past computations so the model need not redo them, but it grows linearly with the length of the input, consuming scarce graphics memory. Earlier shortcuts treated the cache's many attention heads uniformly, clearing or compressing them by a single fixed rule.

BaKlaVa instead estimates the importance of each cache individually. A one-time profiling step then assigns each cache a share of a total memory budget. Important caches retain more of their data, while unimportant ones are trimmed harder. The team tested the approach on two widely used open models. It compressed the cache by up to seventy percent while keeping baseline performance, and at higher compression rates it improved accuracy by as much as tenfold over uniform methods.

Catalog record

CREATORAhmed Burak Gulhan, Krishna Teja Chitty-Venkata, Murali Emani, Mahmut Kandemir, Venkatram Vishwanath
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13176

Browse the full library · About the Tower of Babel