Visual Words Meet BM25: Sparse Auto-Encoder Visual Word Scoring for Image Retrieval
Donghoon Han, Eunhwan Park, Seunghyeon Seo · 2026
Read Visual Words Meet BM25: Sparse Auto-Encoder Visual Word Scoring for Image Retrieval by Donghoon Han, Eunhwan Park, Seunghyeon Seo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05781.txt.
Introduction
Han and colleagues adapt BM25 ranking to image retrieval by scoring sparse visual words. Dense retrieval is accurate but hard to interpret and costly at scale, so the authors score sparse autoencoder activations over vision transformer patches. Since visual-word frequencies skew, inverse document frequency weighting suppresses common words and lifts rare ones. A sparse inverted index supplies candidates, with only 200 per query passed to dense reranking. Trained once on ImageNet-1K, the encoder transfers untuned to seven fine-grained benchmarks, and each decision traces to visual words.
Catalog record
| CREATOR | Donghoon Han, Eunhwan Park, Seunghyeon Seo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38904 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05781
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Visual Words Meet BM25: Sparse Auto-Encoder Visual Word Scoring for Image Retrieval Donghoon Han, Eunhwan Park, Seunghyeon Seo Abstract Dense image retrieval is accurate but offers limited interpretability and attribution, and it can be compute-intensive at scale. We present BM25-V , which applies Okapi BM25 scoring to sparse visual-word activations from a Sparse Auto-Encoder (SAE) on Vision Transformer patch features. Across a large gallery, visual-word document frequencies are highly imbalanced and follow a Zipfian-like distribution, making BM25’s inverse document frequency (IDF) weighting well suited for suppressing ubiquitous, low-information words and emphasizing rare, discriminative ones. BM25-V retrieves high-recall candidates via sparse inverted-index operations and serves as an efficient first-stage retriever for dense reranking. Across seven benchmarks, BM25-V achieves Recall@200 ≥ \geq 0.993, enabling a two-stage pipeline that reranks only K = 200 K{=}200 candidates per query and recovers near-dense accuracy within 0.2 0.2 % on average. An SAE trained once on ImageNet-1K transfers zero-shot to seven fine-grained benchmarks without fine-tuning, and BM25-V retrieval decisions are attributable to specific visual words with quantified IDF contributions. 1 Introduction Image retrieval at scale is dominated by dense retrieval : a vision encoder [ 24 , 20 , 30 , 28 ] maps images to continuous embeddings, and retrieval is performed via Approximate Nearest Neighbor (ANN) search [ 12 , 17 ] . Since this paradigm is simple yet effective, it remains several drawbacks. First, dense retrieval offers limited interpretability and attribution, thereby raising the concerns in real-world applications where decisions should be auditable, such as medical imaging, forensic search, and e-commerce, respectively. Second, since retrieving full-precision indices require O(N\cdot D) float32 memory where N,D indicate images and dimension, this is not practical for real-world application when billion-scale. Product Quantization (PQ) [ 11 ] mitigates the usage of memory by 8\sim 32\times , however it has inevitably accuracy – memory trade-off. Lastly, dense retrieval tends to lose fine-grained spatial evidence because it typically aggregates patch-level representations into a single global embedding via pooling. While this global summarization is effective for coarse semantics, it can suppress local discriminative evidence that are critical for fine-grained retrieval, such as subtle part shapes, textures, and localized patterns. To recover interpretable and index-friendly evidence without discarding patch-level information, we revisit a classic sparse retrieval by represent each image as a small set of discriminative “terms” and retrieve with an inverted index. Concretely, inspired by findings of transformer representations that reveal monosemantic features in language models, we apply a SAE to late-layer ViT patch features and obtain sparse “visual words” activations [ 2 ] . We observe that SAE visual words exhibit the document-frequency imbalance across the gallery, consistent with a Zipfian-like distribution [ 31 ] . Rare words tend to capture discriminative visual evidence, while frequent words largely reflect generic content such as backgrounds and common textures. This observation implies that a heavy-tailed term distribution is aligned to Okapi BM25 [ 26 ] , which is developed for sparse retrieval under document-frequency imbalance. In particular, BM25 employs inverse document frequency weighting that attenuates ubiquitous terms and emphasizes rare, discriminative ones. Motivated by this alignment, we adopt SAE-derived visual words with BM25 and refer to the resulting sparse retriever as BM25-V . BM25-V enables efficient inverted-index retrieval by representing each image as a compact set of active visual words. Given late-layer ViT patch features, we apply the SAE encoder and aggregate activations across patches to obtain image-level visual-word weights, which we treat as term frequencies. We then retain only the top- k words per image to form a sparse bag-of-visual-words representation. At query time, scoring is restricted to the term-specific inverted lists of the active query words, yielding per-query complexity on the order of O(k\cdot\overline{\mathrm{df}}) rather than the O(N\cdot D) dense similarity computation required by full-scan retrieval. In our experiments, BM25-V achieves Recall@100 comparable to dense Recall@10 across all seven benchmarks. This motivates a two-stage pipeline where BM25-V serves as a fast first-stage retriever and dense reranking is applied only to the top- K retrieved candidates. As a result, dense similarity scoring is performed for only K candidates per query rather than the full gallery, while maintaining near-dense accuracy within 0.2 percentage points on average. In contrast, product quantization (PQ) reduces memory by compressing dense embeddings and can incur non-trivial accuracy degradation. Our two-stage design instead preserves dense representations and reduces compute through candidate pruning. Moreover, the rerank stage stores dense embeddings at 4D bytes per vector in float32, and BM25-V adds a lightweight sparse index of 96 bytes per image when k{=}16 visual words are stored using a 4-byte id and a 2-byte activation. Although the combined memory footprint is higher than PQ (24–48 bytes per vector in our experiments), it avoids the 1–6% accuracy degradation observed with PQ. Our contributions are summarized as follows: 1. BM25-V: first application of Okapi BM25 to SAE-derived sparse visual words. We introduce BM25-V, which applies Okapi BM25 to SAE-derived visual words for image retrieval. We show that the resulting visual-word document frequencies are heavy-tailed (Zipfian-like), making BM25 a principled choice in this sparse feature space. We further validate key design choices, including sparsity k and expansion factor e , via systematic ablations ( Table 3 and Appendix). 2. Two-stage retrieval with substantial compute reduction. BM25-V provides a high-recall candidate set (e.g., R@100 comparable to dense R@10; R@200 \geq 0.993 across seven benchmarks), and dense reranking over K candidates recovers near-dense rank-1 accuracy (within 0.2 % on average). This reduces dense similarity evaluations from N gallery items to K candidates per query. 3. Zero-shot cross-domain generalization. An SAE trained once on ImageNet-1K [ 6 ] transfers to seven fine-grained retrieval benchmarks without fine-tuning, suggesting that the learned visual vocabulary generalizes beyond the training distribution. 4. Interpretability by construction. BM25-V retrieval decisions are attributable to discrete visual words with explicit IDF contributions, enabling transparent, term-level explanations (see supplementary material). 2 Related Work Dense visual retrieval. Standard image retrieval encodes images with a vision backbone [ 24 , 20 , 30 , 28 ] and performs ANN search over MAP-pooled embeddings [ 12 , 17 ] . FAISS-HNSW [ 17 ] provides near-lossless approximate search, while FAISS-IVF+PQ [ 12 ] reduces memory via product quantization at the cost of recall. However, dense embeddings offer limited interpretability because retrieval evidence is distributed across many entangled dimensions. BM25-V complements dense retrieval with a sparse channel that produces a high-recall candidate set for dense reranking, reducing dense similarity evaluations while preserving near-dense accuracy. Inverted-index retrieval also supports efficient dynamic gallery updates, where inserting or deleting an image requires updating only its sparse entries. Sparse text retrieval and BM25. Okapi BM25 [ 26 ] , rooted in the Probabilistic Relevance Framework [ 25 ] , remains a strong zero-shot baseline for text IR. Its effectiveness is closely tied to long-tailed term statistics, where inverse document frequency (IDF) downweights ubiquitous terms and emphasizes rare, discriminative ones. Recent learned sparse retrievers such as DeepCT [ 5 ] , doc2query [ 19 ] , uniCOIL [ 15 ] , and SPLADE [ 8 ] preserve the inverted-index paradigm while learning term weights, and SPLADE in particular produces sparse activations that are naturally scored with BM25-style weighting. We draw a direct parallel in vision.