Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache
Hanchen Li (Student), Yuhan Liu (Student), Yihua Cheng (Student), Kuntai Du (Student), Junchen Jiang{ lihanc2002, yuhanl, yihua98, kuntai, junchenj } @uchicago.edu The University of Chicago · 2025
Read Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache by Hanchen Li (Student), Yuhan Liu (Student), Yihua Cheng (Student), Kuntai Du (Student), Junchen Jiang{ lihanc2002, yuhanl, yihua98, kuntai, junchenj } @uchicago.edu The University of Chicago (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14647.txt.
Description
Hanchen Li, Yuhan Liu, Yihua Cheng, and two colleagues ask a money question about language model caches, in a paper published in two thousand twenty-five. Many apps now reuse stored key-value caches. The model can then skip re-reading repeated input, cutting the wait for a first reply.
The open question is cost. Keeping those caches on cloud services means paying for storage, network transfer, and compute. The savings depend on how often text repeats, how long replies run, and how big the model is. The authors build a tested analytical model of those cloud costs. Their early results map when reuse pays for itself and when it quietly loses money. The work gives developers a way to price a popular speed trick before adopting it. It treats the cache not as free memory but as rented shelf space.
Catalog record
| CREATOR | Hanchen Li (Student), Yuhan Liu (Student), Yihua Cheng (Student), Kuntai Du (Student), Junchen Jiang{ lihanc2002, yuhanl, yihua98, kuntai, junchenj } @uchicago.edu The University of Chicago |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 9 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14647