Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache

Hanchen Li (Student), Yuhan Liu (Student), Yihua Cheng (Student), Kuntai Du (Student), Junchen Jiang{ lihanc2002, yuhanl, yihua98, kuntai, junchenj } @uchicago.edu The University of Chicago · 2025

Read Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache by Hanchen Li (Student), Yuhan Liu (Student), Yihua Cheng (Student), Kuntai Du (Student), Junchen Jiang{ lihanc2002, yuhanl, yihua98, kuntai, junchenj } @uchicago.edu The University of Chicago (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14647.txt.

Description

Hanchen Li, Yuhan Liu, Yihua Cheng, and two colleagues ask a money question about language model caches, in a paper published in two thousand twenty-five. Many apps now reuse stored key-value caches. The model can then skip re-reading repeated input, cutting the wait for a first reply.

The open question is cost. Keeping those caches on cloud services means paying for storage, network transfer, and compute. The savings depend on how often text repeats, how long replies run, and how big the model is. The authors build a tested analytical model of those cloud costs. Their early results map when reuse pays for itself and when it quietly loses money. The work gives developers a way to price a popular speed trick before adopting it. It treats the cache not as free memory but as rented shelf space.

Catalog record

CREATORHanchen Li (Student), Yuhan Liu (Student), Yihua Cheng (Student), Kuntai Du (Student), Junchen Jiang{ lihanc2002, yuhanl, yihua98, kuntai, junchenj } @uchicago.edu The University of Chicago
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE9 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14647

Browse the full library · About the Tower of Babel