Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies

Benjamin Clavié, Sean Lee, Aamir Shakir, Makoto P. Kato · 2026

Read Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies by Benjamin Clavié, Sean Lee, Aamir Shakir, Makoto P. Kato (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29384.txt.

Description

Benjamin Clavié presents Latent Terms, showing that retrievers learn forms that split into sparse parts. Trained on frozen retrievers, sparse autoencoders with no tweaks for retrieval extract a vocab with Zipfian stats. That vocab is ready for sparse retrieval scoring via BM25. This enables sparse retrieval with no learned expansion goal or sparse supervision, and applies to any retriever. Latent Terms matches or beats its base model scoring and SPLADE variants. It beats its base model by a lot on LIMIT, a task to highlight failures.

Catalog record

CREATORBenjamin Clavié, Sean Lee, Aamir Shakir, Makoto P. Kato
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42292
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29384

Browse the full library · About the Tower of Babel