Your Embedding Model is SMARTer Than You Think

Jianrui ZhangUW-Madison [email protected] Hyun Jung LeeKorea University [email protected] Sukanta GangulyNetApp, Inc. [email protected] Tae-Eui KamKorea University [email protected] Donghyun KimKorea University [email protected], Yong Jae LeeUW-Madison [email protected] · 2026

Read Your Embedding Model is SMARTer Than You Think by Jianrui ZhangUW-Madison [email protected] Hyun Jung LeeKorea University [email protected] Sukanta GangulyNetApp, Inc. [email protected] Tae-Eui KamKorea University [email protected] Donghyun KimKorea University [email protected], Yong Jae LeeUW-Madison [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24938.txt.

Introduction

Jianrui Zhang leads SMART, a trick coaxing multi-vector retrieval skill out of ordinary single-vector embedding models. Single-vector retrievers squash rich token sequences into one global vector. That is fast but drops fine local evidence dense retrieval needs. Multi-vector rivals need fresh training and often drop the global summary. The team notes pooled contrastive gradients already reach earlier hidden states, already arranged for cosine matching. Direct late interaction over frozen states at inference lifts scores, with gains focused on fine-grained and visual-document tasks.

Catalog record

CREATORJianrui ZhangUW-Madison [email protected] Hyun Jung LeeKorea University [email protected] Sukanta GangulyNetApp, Inc. [email protected] Tae-Eui KamKorea University [email protected] Donghyun KimKorea University [email protected], Yong Jae LeeUW-Madison [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48891
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24938

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Your Embedding Model is SMARTer Than You Think Jianrui ZhangUW-Madison [email protected] Hyun Jung LeeKorea University [email protected] Sukanta GangulyNetApp, Inc. [email protected] Tae-Eui KamKorea University [email protected] Donghyun KimKorea University [email protected], Yong Jae LeeUW-Madison [email protected] Abstract Multimodal retrieval relies heavily on single-vector retrievers, which compress rich, sequential token sequences into one single global representation. While efficient, they discard fine-grained, local evidence critical for dense retrieval tasks. Multi-vector approaches were introduced as a solution, but they strictly require training and many ignore the necessity of a globally summarizing representation. To address this, we introduce SMART, a framework that unlocks the latent multi-vector capabilities of standard single-vector models. We first demonstrate that standard contrastive training on the pooled embedding implicitly shapes the retrieval geometry of preceding hidden states via gradient flow. By applying direct late-interaction over these frozen hidden states during inference, SMART acts as a plug-and-play upgrade that consistently improves performance across diverse modalities, improving even the state-of-the-art models further on MMEB-V2. We also reveal SMART’s superior performance, as simple lightweight post-training not only saves time and compute, but also brings forth further improvement on Visual Document retrieval, allowing a single-vector model to outperform SoTA multi-vector counterparts. Ultimately, SMART offers both a highly efficient inference enhancement and a powerful finetuning technique for multimodal retrieval. We open source our code and weights at https://github.com/HanSolo9682/SMART . Figure 1: Standard single-vector embedding models (Qwen3-VL-Embedding-8B in this figure) compress sequences into single token representations, which often results in losing local information and underutilizing the remaining hidden states. While existing multi-vector approaches require expensive retraining, SMART reveals that the hidden states of single-vector embedders are already geometrically aligned for local matching. SMART thus transforms any single-vector model into a multi-vector variant both during inference and via lightweight post-training , the former significantly improving with no training, and the latter allowing us to save time and rival SoTA multi-vector embedders. 1 Introduction Multimodal Large Language Models (MLLMs) have recently unified dense retrieval across text, images, visual documents, and videos [ 21 , 7 ] . State-of-the-art (SoTA) systems, such as the Qwen3-VL-Embedding series [ 10 ] , map these diverse modalities into a highly expressive, shared representation space, enabling efficient global similarity matching. However, these architectures predominantly rely on a single-vector paradigm, collapsing the entire sequence of multimodal hidden states into a single pooling token, such as the end-of-text ( <eot> ) token. While this compression ensures highly efficient indexing and nearest-neighbor search, recent theoretical analyses demonstrate fundamental limitations in its capacity [ 22 , 14 , 18 ] . Since the number of distinct subset rankings a single-vector paradigm can reliably return is strictly bounded by the embedding dimensionality, fine-grained multimodal queries that depend on fine-grained details such as local text, specific visual attributes, or regional bindings can often fail. This is because localized evidence encoded by the transformer can be lost when the input is compressed into the pooled representation used for scoring. To overcome this expressive bottleneck, researchers have increasingly turned to multi-vector architectures, pioneered in the text domain by ColBERT [ 8 ] and recently adapted for multimodal tasks via models like Colpali [ 3 ] and jina-embeddings-v4 [ 4 ] . Yet, these approaches require either full-scale task-specific finetuning or the introduction of learnable tokens (e.g., MetaEmbed [ 23 ] ), which incurs significant computational and memory costs during training as they scale quadratically with respect to sequence length. Moreover, methods like Colpali and jina emphasize local token- or patch-level matching without explicitly preserving the global pooled readout that single-vector models use effectively. To bridge this gap, we introduce SMART ( S ingle-to- M ulti A daptation for R etrieval T ransformers), a framework that converts a single-vector retriever into a multi-vector retriever possible both at inference time and via lightweight finetuning, while preserving its global compatibility signal. We make the observation that the gradients from the contrastive loss on the pooled token propagate through the transformer’s computation graph, implicitly organizing the preceding hidden states into a geometry highly compatible with cosine retrieval. SMART initially exploits this by applying an additional late-interaction mechanism ( MaxSim ) [ 8 ] over the pre-pooling hidden states at inference time, combined with the pooled score in a hybrid scoring scheme, effectively recovering localized details without the overhead of training a multi-vector retriever from scratch. Building on this foundation, we further show that lightweight finetuning under the SMART objective yields additional gains, transforming single-vector embedders into competitive multi-vector retrievers. Critically, this conversion saves at least 20% of training time and computation compared to training a multi-vector retriever from scratch under the same recipe. Our core contributions are as follows: • We show that single-vector retrievers, despite being trained only with a pooled contrastive objective, already retain localized semantic evidence in their non-pooling hidden states. This makes it possible to convert an existing single-vector model into a multi-vector retriever by reusing these hidden states for token-level matching. • We propose SMART, which can act as a training-free, plug-and-play upgrade using our hybrid scoring technique. It steadily improves retrieval accuracy across various complex retrieval tasks and backbones, pushing even the SoTA Qwen3-VL-Embedding [ 10 ] models to new performance heights. • We demonstrate that SMART can be further improved with efficient post-training , either by attaching a lightweight projection adapter while freezing the pretrained single-vector backbone, or by finetuning the single-vector embedder with our hybrid scoring objective. These variants enable single-vector embedders to achieve strong multi-vector retrieval performance, saving the time to train a dedicated multi-vector model from scratch. 2 Related Work Single-Vector Embedding Models Early work on contrastive models (CLIP [ 17 ] , BLIP [ 9 ] , SigLIP [ 25 ] ) and MLLMs [ 12 , 28 , 11 , 1 ] paved the way for modern MLLM-based dense retrievers like UniIR [ 21 ] and VLM2Vec [ 7 ] . Recent advances focus on efficient training strategies (E5-V [ 6 ] , GME [ 27 ] ) and highly expressive unified spaces, with Qwen3-VL-Embedding [ 10 ] currently achieving SoTA. These approaches, however, waste the compute spent on the local fine-grained non-pooled hidden states. By applying SMART to these off-the-shelf single-vector models, we use that information to demonstrate an easy, training-free approach to convert them into multi-vector architectures for improved retrieval accuracy. Multi-Vector Embedding Models Because single-vector models face theoretical capacity limits [ 22 ] , researchers have increasingly turned to multi-vector architectures. Pioneered in text by ColBERT [ 8 ] , late-interaction mechanisms have been adapted for multimodal tasks via models like Colpali [ 3 ] , jina-embeddings-v4 [ 4 ] , and MetaEmbed [ 23 ] . Unlike these approaches—which require full-scale task-specific training, adapters, or learnable tokens—SMART can be used entirely inference-only. By combining MaxSim over all hidden states and the global summary token directly to single-vector models, training-free SMART already provides multi-vector performance benefits. On the other hand, lightweight post-training with SMART further improves performance and can convert SoTA single-vector models into SoTA multi-vector while saving time and compute. Multimodal Retrieval Benchmarks Standardizing evaluation has evolved from foundational baselines like M-BEIR [ 21 ] to comprehensive collections like MMEB [ 7 ] and MMEB-V2 [ 15 ] , which span diverse modalities

Read the complete text