A Brief Comparison of Training-Free Multi-Vector Sequence Compression Methods
Rohan Jha, Chunsheng Zuo, Reno Kriz, Benjamin Van Durme · 2026
Read A Brief Comparison of Training-Free Multi-Vector Sequence Compression Methods by Rohan Jha, Chunsheng Zuo, Reno Kriz, Benjamin Van Durme (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22434.txt.
Introduction
Jha and co-authors compare ways to shrink multi-vector retrieval indexes without any training. Such models beat single-vector ones on quality, but their document embeddings grow with sequence length. That size drives both memory cost and query delay, so compression matters for real use. The team tests methods that cut the token sequence itself. They find that merging tokens beats pruning them, keeping retrieval quality while shrinking the index.
Catalog record
| CREATOR | Rohan Jha, Chunsheng Zuo, Reno Kriz, Benjamin Van Durme |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 11907 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22434
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Brief Comparison of Training-Free Multi-Vector Sequence Compression Methods Rohan Jha, Chunsheng Zuo, Reno Kriz, Benjamin Van Durme Abstract While multi-vector retrieval models outperform single-vector models of comparable size in retrieval quality, their practicality is limited by substantially larger index sizes, driven by the additional sequence-length dimension in their document embeddings. Because document embedding size dictates both memory overhead and query latency, compression is essential for deployment. In this work, we present an evaluation of training-free methods targeting the token sequence length, a dimension unique to multi-vector retrieval. Our findings suggest that token merging is strictly superior to token pruning for reducing index size while maintaining retrieval effectiveness. [orcid=0009-0001-5008-4949, [email protected], url=https://www.cs.jhu.edu/ rjha5/, ] *1 1 [orcid=0009-0000-4988-2401, [email protected], url=https://scholar.google.com/citations?user=Uy79AH0AAAAJ, ] 1 [ orcid=0000-0002-0239-9989, [email protected], url=https://hltcoe.jhu.edu/researcher/reno-kriz/, ] [ orcid=0000-0003-4328-4288, [email protected], url=https://www.cs.jhu.edu/ vandurme/, ] keywords Information Retrieval ,Multi-Vector Retrieval ,Index Compression ,Token Pruning ,Token Pooling † † copyrightyear: 2026 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: LIR’26: The 1st Late Interaction Workshop @ ECIR 2026, April 02, 2026, Delft, the Netherlands † † address: Johns Hopkins University, 3400 N. Charles St., Baltimore, MD 21218, USA † † address: Human Language Technology Center of Excellence, 810 Wyman Park Drive, Baltimore, Maryland 21211, USA † † corresp: Corresponding author. † † These authors contributed equally. 1 Introduction Multi-vector retrieval models, exemplified by ColBERT 8 , represent documents as sets of small token-level embeddings, enabling fine-grained semantic matching via late interaction. This approach consistently delivers stronger retrieval quality than single-vector models of comparable size, particularly in out-of-domain settings 7 . However, naively storing per-token embeddings for every document in a collection yields index sizes orders of magnitude larger than their single-vector counterparts, increasing both storage costs and query latency — albeit not linearly in practice. Compression is therefore essential to practical multi-vector deployment, and existing work has addressed this along several axes: reducing embedding precision, reducing embedding dimensionality, or reducing the number of embeddings stored per document. In practice, many effective methods operate across more than one of these axes — for instance, product quantization jointly reduces dimensionality and precision (number of codebooks and code length), while PLAID 13 combines centroid-based encoding with low-bit residuals to a similar effect. The third axis, token sequence length, is unique to multi-vector models and remains comparatively underexplored. Methods along this axis fall into three broad families: fixed-length embeddings 10 ; 12 ; 15 , token pruning 10 ; 1 ; 5 , which discards embeddings deemed uninformative, and token pooling 6 ; 4 , which merges groups of similar tokens into representative supervectors. Crucially, several methods in the latter two families are training-free, making them attractive as drop-in post-hoc compression steps for any existing multi-vector model. Yet there is no direct comparison of these approaches under controlled conditions. In this work, we evaluate training-free token pruning and token merging methods across BEIR datasets 14 , and confirm the intuition cited in 4 that token pooling methods consistently outperform token pruning approaches across all compression ratios in our examined text retrieval domains. Pooled sequence compression techniques often matched uncompressed performance with 2-5 \times fewer embeddings. 2 Methods We study training-free , document-side sequence compression for multi-vector retrieval. A document is encoded as token embeddings \mathbf{D}=[\mathbf{d}_{1},\dots,\mathbf{d}_{L}] ; retrieval uses the MaxSim operator: \operatorname{score}(\mathbf{Q},\mathbf{D})=\sum_{i}\max_{j}\;\mathbf{q}_{i}^{\top}\mathbf{d}_{j} . Compression replaces \mathbf{D} with a shorter \widetilde{\mathbf{D}} of length \widetilde{L}<L , reducing both index size and consequently query latency. Following 4 , the first token (the document marker token [D] ) is protected from compression. Token Pruning Pruning assigns each non-protected token a scalar importance score and retains only the top- C tokens, discarding the rest. We evaluate three scoring functions: (i) Random selection as a stochastic baseline; (ii) Attention 10 , which scores each token by the total attention it receives across all heads and source positions in the final transformer layer; and (iii) IDF 10 ; 1 , scoring tokens by corpus-level inverse document frequency to preferentially retain rare terms. Targeting a fixed compression ratio, IDF pruning is applied document-wise using globally collected IDF statistics. Token Pooling Pooling groups non-protected tokens into C clusters and replaces each cluster with the mean of its member embeddings, preserving information rather than discarding it. Clusters are formed by assigning each token to its nearest anchor ; we select the k anchors via the top- k scores according to Random , Attention , or IDF (as above). We additionally evaluate two clustering-based variants that partition tokens directly: Spherical k -Means on \ell_{2} -normalised embeddings, and Hierarchical (Ward) clustering with cosine distance as proposed in 4 . Experimental Setup We compare an uncompressed baseline against our compression methods across 5 keep ratios ( r\in\{0.10,0.20,0.33,0.50,0.75\} ) on six BEIR 14 datasets ( arguana , scidocs , fiqa , nfcorpus , scifact , trec-covid ) and three CoIR 11 text-to-code retrieval datasets ( apps , cosqa , synthetic-text2sql ), reporting NDCG@10 and Recall@100 via ranx 2 . Our compression targets only document embeddings, which are encoded with a maximum length of L=300 tokens for BEIR, and L=2048 for CoIR datasets. We use the default query lengths in 3 ’s BEIR baseline for BEIR datasets and 1024 query tokens for CoIR datasets. Both settings’ models use no [MASK] query expansion tokens 9 during encoding. After compression, embeddings are \ell_{2} -normalized and indexed with PLAID. 3 Results Figure 1 : nDCG@10 (left) and Recall@100 (right) relative to Baseline (no compression) averaged across 6 BEIR datasets. Pooling methods (solid lines) consistently degrade much more gracefully than pruning methods (dotted lines) under heavy compression. Figure 2 : Dataset-specific breakdowns of nDCG@10 vs. average document length on 6 BEIR datasets. Axes are scaled individually to highlight per-dataset compression characteristics. Figure 3 : Dataset-specific breakdowns of Recall@100 vs. average document length on 6 BEIR datasets. Axes are scaled individually to highlight per-dataset compression characteristics. Our evaluation across the BEIR benchmark reveals a consistent and distinct advantage for token pooling (merging) over token pruning methods, particularly at aggressive compression rates. Furthermore, our results challenge several conventional assumptions regarding embedding importance and clustering complexity. The overall trends for nDCG@10 and Recall@100 across different keep ratios are presented in Figure 1 , while dataset-specific breakdowns are provided in Figures 2 and 3 . The empirical results strongly support the hypothesis that aggregating semantic information is strictly superior to discarding it. Token pooling methods preserve the underlying semantic density of the original document embeddings much better than their pruning counterparts. Our results show that pooling can achieve higher compression ratios at equivalent retrieval quality. For instance, across the averaged BEIR datasets, Hierarchical Pooling at a keep ratio of r=0.20 (averaging roughly 38 tokens per document) achieves an average nDCG@10 of 0.497 (95.7%) and Recall@100 of 0.588 (98.1%). In contrast, IDF Pruning requires a much larger keep ratio of r=0.50 (95 tokens per document) just to reach an nDCG@10 of 0.474 (92.4%) and a Recall@100 of 0.593 (98.9%). In practical settings, pooling allows for more than double the index size reduction compared to pruning at the same target performance threshold. At severe compression levels ( r\leq 0.20 ), the divergence between the two methods becomes