PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation

Ao Wang, Hui Chen, Jianchao Tan, Kefeng Zhang, Xunliang Cai, Zijia Lin, Jungong Han, Guiguang Ding · 2024

Read PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation by Ao Wang, Hui Chen, Jianchao Tan, Kefeng Zhang, Xunliang Cai, Zijia Lin, Jungong Han, Guiguang Ding (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03409.txt.

Description

Ao Wang and co-authors present PrefixKV in a 2024 paper on making large vision-language models faster. These models read images and text and write replies, but they store a large cache of past keys and values as they work. That cache drives up cost and slows replies, more so with long inputs and outputs.

Most past fixes shrink the cache by the same amount in every layer, which can drop key facts in some layers. PrefixKV instead keeps the most valued cache entries in each layer and uses a search to set each layer's share. Tests show a better balance of speed and quality than rival methods. The authors share their code for others to use and build on.

Catalog record

CREATORAo Wang, Hui Chen, Jianchao Tan, Kefeng Zhang, Xunliang Cai, Zijia Lin, Jungong Han, Guiguang Ding
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03409

Browse the full library · About the Tower of Babel