FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference
Dongwei Wang, Zijie Liu, Song Wang, Yuxin Ren, Jianing Deng, Jingtong Hu, Tianlong Chen, Huanrui Yang · 2025
Read FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference by Dongwei Wang, Zijie Liu, Song Wang, Yuxin Ren, Jianing Deng, Jingtong Hu, Tianlong Chen, Huanrui Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08256.txt.
Description
Dongwei Wang, Zijie Liu, Song Wang, and a wider team wrote this machine learning paper, published in 2025. It speeds up long-context language models by fetching only the stored context tokens that truly matter, one by one rather than in blocks.
As chats grow long, reading the whole stored key-value cache slows each new word. Older tricks drop tokens by fixed rules or fetch them in pages, but key tokens are scattered thinly, so pages waste space and miss gems. The authors' method, FIER, scores each token with a squeezed one-bit form of its key, then pulls the top ones. In tests it matched full-cache quality while using about a tenth of the cache, and cut decoding time notably. The team released the code for others to use.
Catalog record
| CREATOR | Dongwei Wang, Zijie Liu, Song Wang, Yuxin Ren, Jianing Deng, Jingtong Hu, Tianlong Chen, Huanrui Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08256