Multivector Reranking in the Era of Strong First-Stage Retrievers
Silvio Martinico, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini · 2026
Read Multivector Reranking in the Era of Strong First-Stage Retrievers by Silvio Martinico, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05200.txt.
Introduction
Silvio Martinico and 3 co-authors revisit multivector reranking for search in a 2026 paper. Learned multivector representations, which describe each document with many vectors, give modern search systems strong retrieval quality. Exhaustive token-level search over them is too expensive, so real systems gather candidates with a cheap first stage and refine. That gather step needs costly searches through large token-level indexes, and it often misses documents that full scoring would rank highest.
The authors re-run several leading multivector retrieval methods on two public datasets for comparison. Their study lands in an era of strong first-stage retrievers, which changes what reranking must add. Clear comparisons help search engineers decide where extra computation actually buys relevance.
Catalog record
| CREATOR | Silvio Martinico, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05200
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multivector Reranking in the Era of Strong First-Stage Retrievers Silvio Martinico, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini Abstract Learned multivector representations power modern search systems with strong retrieval effectiveness, but their real-world use is limited by the high cost of exhaustive token-level retrieval. Therefore, most systems adopt a gather-and-refine strategy, where a lightweight gather phase selects candidates for full scoring. However, this approach requires expensive searches over large token-level indexes and often misses the documents that would rank highest under full similarity. In this paper, we reproduce several state-of-the-art multivector retrieval methods on two publicly available datasets, providing a clear picture of the current multivector retrieval field and observing the inefficiency of token-level gathering. Building on top of that, we show that replacing the token-level gather phase with a single-vector document retriever—specifically, a learned sparse retriever (LSR)—produces a smaller and more semantically coherent candidate set. This recasts the gather-and-refine pipeline into the well-established two-stage retrieval architecture. As retrieval latency decreases, query encoding with two neural encoders becomes the dominant computational bottleneck. To mitigate this, we integrate recent inference-free LSR methods, demonstrating that they preserve the retrieval effectiveness of the dual-encoder pipeline while substantially reducing query encoding time. Finally, we investigate multiple reranking configurations that balance efficiency, memory, and effectiveness, and we introduce two optimization techniques that prune low-quality candidates early. Empirical results show that these techniques improve retrieval efficiency by up to 1.8 × \times with no loss in quality. Overall, our two-stage approach achieves over 24 × 24\times speedup over the state-of-the-art multivector retrieval systems, while maintaining comparable or superior retrieval quality. 1 Introduction Transformer-based neural architectures [ 48 ] and large pre-trained language models such as BERT [ 12 ] have reshaped the field of Information Retrieval (IR), thanks to their ability to encode rich semantic representations of text. These models have enabled a shift from purely lexical matching to representation-based retrieval, where queries and documents are mapped into vector spaces that capture deep contextual meaning. This paradigm has led to a new generation of neural retrievers that substantially improve retrieval effectiveness over traditional term-based models such as BM25 [ 42 ] . These models can be broadly categorised into sparse and dense encoders. Sparse encoders, such as Splade [ 21 ] , produce high-dimensional representations, where only a small fraction of components are non-zero. In contrast, dense encoders represent each query or document as a dense vector in a low-dimensional space that captures global semantic similarity. They can be further divided into single-vector and multivector architectures. Single-vector encoders [ 25 , 49 , 50 ] map an entire passage to a single dense vector, providing compact document-level semantics but limiting fine-grained query–document interactions. Differently, multivector models such as ColBERT [ 26 ] encode each query and document as a set of token-level vectors. In this framework, similarity between a query and a document is computed via late interaction through the max similarity operator, MaxSim(q,D)=\sum_{i=1}^{n_{\mathtt{q}}}\max_{j=1,\dots,n_{\mathtt{D}}}\langle q_{i},D_{j}\rangle , which lets every query token interact with every document token [ 26 ] . These models achieve state-of-the-art effectiveness because they preserve fine-grained token interactions. However, performing retrieval over a full multivector collection is computationally infeasible. For this reason, all scalable multivector systems follow a gather-and-refine strategy [ 43 , 38 , 2 , 45 ] : 1) a gather phase that selects a relatively small set of candidate documents; 2) a refine phase that applies full MaxSim scoring on these candidates to produce the final ranking. Most existing systems perform the gather step at the token-embedding level. As already noted in [ 29 ] , this design has two fundamental drawbacks: 1) the index cardinality (the number of vectors to search) grows with the number of tokens, which is typically one or two orders of magnitude larger than the number of documents, leading to significantly higher search-space complexity and slower queries, and 2) token-level similarity alone does not predict document-level relevance well: MaxSim depends on the joint interaction of all query tokens with all document tokens, and the final score depends on many different document tokens. The token-based gather step is therefore both inefficient , because it must retrieve and evaluate many more candidates, and often ineffective , because it fails to identify the documents that would score highest under full MaxSim . To mitigate this, XTR [ 29 ] redefines the training objective to make token retrieval itself more predictive of document-level relevance, partially closing the training–inference gap. Through a reproducibility study (Section 4.1 ), we observe that state-of-the-art multivector methods exhibit limited efficiency. Despite their effort to optimize the gather phase, this step remains fundamentally inadequate. As emerges in recent works [ 38 , 45 , 2 ] , token-level gathering not only incurs high computational costs but also lacks the selectivity required for a compact candidate set, often returning hundreds or thousands of documents that must undergo expensive full MaxSim scoring. Driven by this observation, in contrast to the approach taken by XTR [ 29 ] , we take a different direction: we propose to replace the token-level gather step entirely with a document-oriented gather strategy, where candidates are selected based on single-vector document representations. This design naturally aligns with the classical two-stage retrieval architecture. In literature, BM25 and similar lexical bag-of-words models have been widely used for first-stage retrieval. However, the limited semantic coverage of BM25 comes at a price: if the initial candidate set omits a large fraction of relevant documents, no reranking strategy can fully recover them. Indeed, a recent work shows that the effectiveness achieved by reranking BM25 candidates using multivectors is limited [ 33 ] . We claim that recent advances in first-stage retrievers based on learned sparse representations make this shift particularly effective, yielding more semantically coherent candidate sets while drastically reducing retrieval cost. To support this, Figure 1 (left) compares the recall at various cut-offs \kappa between BM25 and a learned sparse representation obtained with the Splade CoCondenser EnsembleDistill model on the Ms Marco-v1 passage dataset. The recall gap is substantial. To compensate for the recall deficit of BM25 , pipelines typically increase the cut-off \kappa to hundreds or even thousands of documents, which in turn inflates reranking cost. From Figure 1 (right), it is clear that reranking time grows rapidly with the number of candidates, making larger cut-offs increasingly inefficient. Figure 1 : 1) Recall@ \kappa on Ms Marco-v1 . 2) MRR@10 after reranking with ColBERTv2 on Ms Marco-v1 . 3) Time for reranking \kappa candidates for different multivectors compression schemes. ColBERTv2 on Ms Marco-v1 . Even if with large \kappa BM25 manages to reach quite high recall, Figure 1 (center) shows that the resulting MRR@10 after reranking is still much lower. This indicates that BM25 misses many semantically relevant candidates and that increasing \kappa introduces noisy documents whose scores hurt reranking quality. In this paper, we analyse a modern two-stage pipeline where both stages use neural representations: a learned sparse retriever for the first stage and a dense multivector late-interaction reranker. Our contributions are: • We reproduce several state-of-the-art multivector retrieval methods on two publicly available datasets, providing a clear picture of the current multivector retrieval field. • We introduce a hybrid architecture that couples LSR with