FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference

Dongwei Wang, Zijie Liu, Song Wang, Yuxin Ren, Jianing Deng, Jingtong Hu, Tianlong Chen, Huanrui Yang · 2025

Read FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference by Dongwei Wang, Zijie Liu, Song Wang, Yuxin Ren, Jianing Deng, Jingtong Hu, Tianlong Chen, Huanrui Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08256.txt.

Description

Dongwei Wang, Zijie Liu, Song Wang, and a wider team wrote this machine learning paper, published in 2025. It speeds up long-context language models by fetching only the stored context tokens that truly matter, one by one rather than in blocks.

As chats grow long, reading the whole stored key-value cache slows each new word. Older tricks drop tokens by fixed rules or fetch them in pages, but key tokens are scattered thinly, so pages waste space and miss gems. The authors' method, FIER, scores each token with a squeezed one-bit form of its key, then pulls the top ones. In tests it matched full-cache quality while using about a tenth of the cache, and cut decoding time notably. The team released the code for others to use.

Catalog record

CREATORDongwei Wang, Zijie Liu, Song Wang, Yuxin Ren, Jianing Deng, Jingtong Hu, Tianlong Chen, Huanrui Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08256

Browse the full library · About the Tower of Babel