You Need an Encoder for Native Position-Independent Caching

Shiju Zhao, Junhao Hu, Jiaqi Zheng, Guihai Chen · 2026

Read You Need an Encoder for Native Position-Independent Caching by Shiju Zhao, Junhao Hu, Jiaqi Zheng, Guihai Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01519.txt.

Description

Shiju Zhao and co-authors propose native position-independent caching for language models, needing an encoder. Prefix-based Key-Value caches handle arbitrary-order retrieved contexts poorly. Position-independent caching promises KV reuse without positional constraints, yet methods degrade accuracy badly. The team reintroduces the encoder into decoder-only models, training it for position-independent caching. Their COMB system integrates with existing inference frameworks, cutting time-to-first-token by 51-94% and tripling throughput with similar accuracy. DeepSeek-V2-Lite-Chat gains show COMB applies elsewhere.

Catalog record

CREATORShiju Zhao, Junhao Hu, Jiaqi Zheng, Guihai Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39136
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01519

Browse the full library · About the Tower of Babel