SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs

Shibo Jie, Yehui Tang, Kai Han, Zhi-Hong Deng, Jing Han · 2025

Read SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs by Shibo Jie, Yehui Tang, Kai Han, Zhi-Hong Deng, Jing Han (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16163.txt.

Description

Shibo Jie and co-authors solve the memory crunch of long-context language models by parking the full memory in the CPU and fetching smartly. A model's key-value cache grows with every token until graphics memory bursts. Compression schemes shrink it by forgetting, permanently. SpeCache refuses to forget: the complete cache lives in roomy system memory.

A low-bit copy kept on the GPU scores which entries matter; each decoding step speculatively fetches the important ones just ahead of need, hiding the transfer time. The design aims for lossless long-context generation at consumer hardware memory budgets. If speculation keeps pace, the cache wall that caps chat length and document size gets much thinner.

Catalog record

CREATORShibo Jie, Yehui Tang, Kai Han, Zhi-Hong Deng, Jing Han
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16163

Browse the full library · About the Tower of Babel