MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction

Zilin Xiao, Qi Ma, Mengting Gu, Jason Chen, Xintao Chen, Vicente Ordonez, Vijai Mohan · 2025

Read MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction by Zilin Xiao, Qi Ma, Mengting Gu, Jason Chen, Xintao Chen, Vicente Ordonez, Vijai Mohan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18095.txt.

Introduction

Zilin Xiao and 6 co-authors scale up mixed-media search with a small set of learned tokens. Their 2025 frame, MetaEmbed, codes a query or an item as a few vectors, not one. One vector can blur fine detail, while too many vectors make search slow and costly.

A fixed set of extra tokens is added to each input at training. Their last-layer states act as the stored vectors, sorted by level of detail. At run time, users pick how many tokens to use, and trade score for speed. On two large bench sets, the frame set top scores and scaled well.

Catalog record

CREATORZilin Xiao, Qi Ma, Mengting Gu, Jason Chen, Xintao Chen, Vicente Ordonez, Vijai Mohan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-18095

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction Zilin Xiao, Qi Ma, Mengting Gu, Jason Chen, Xintao Chen, Vicente Ordonez, Vijai Mohan Abstract Universal multimodal embedding models have achieved great success in capturing semantic relevance between queries and candidates. However, current methods either condense queries and candidates into a single vector, potentially limiting the expressiveness for fine-grained information, or produce too many vectors that are prohibitively expensive for multi-vector retrieval. In this work, we introduce MetaEmbed , a new framework for multimodal retrieval that rethinks how multimodal embeddings are constructed and interacted with at scale. During training, a fixed number of learnable Meta Tokens are appended to the input sequence. At test-time, their last-layer contextualized representations serve as compact yet expressive multi-vector embeddings. Through the proposed Matryoshka Multi-Vector Retrieval training, MetaEmbed learns to organize information by granularity across multiple vectors. As a result, we enable test-time scaling in multimodal retrieval where users can balance retrieval quality against efficiency demands by selecting the number of tokens used for indexing and retrieval interactions. Extensive evaluations on the Massive Multimodal Embedding Benchmark (MMEB) and the Visual Document Retrieval Benchmark (ViDoRe) confirm that MetaEmbed achieves state-of-the-art retrieval performance while scaling robustly to models with 32B parameters. † † affiliation: Meta Superintelligence Labs † † affiliation: Rice University † † contribution: Work done at Meta † † date: August 10, 2026 † † correspondence: Zilin at [email protected] 1 Introduction Multimodal embedding models play an essential role in image search ( Gordo et al. 2016 ) , visual question answering ( Hu et al. 2018 ; Zheng et al. 2021 ) and visual document retrieval ( Faysse et al. 2025 ) , where models project heterogeneous inputs, such as images and texts, into a unified representation space. While existing methods, including CLIP ( Radford et al. 2021 ) , BLIP ( Li et al. 2022 ) and SigLIP ( Zhai et al. 2023 ) have demonstrated superior performance in cross-modal retrieval, their performance remains limited in scenarios where the inputs involve complex and diverse instructions. Thanks to recent advances in building embeddings through foundation vision-language models (VLMs), one could apply contrastive learning on the extracted embedding from the hidden states of the last layer of a VLM to learn meaningful multimodal representations while retaining pre-trained knowledge. Despite growing progress in multimodal embedding VLMs, the common practice of condensing the entire query and candidate into a single vector is not an optimal choice, as fine-grained details are lost between modalities ( Yao et al. 2022 ; Thrush et al. 2022 ) and this process has theoretical limitations ( Weller et al. 2025 ) . In text retrieval, ColBERT ( Khattab and Zaharia 2020 ) introduced a multi-vector late interaction mechanism that retains multiple token-level embeddings and uses a lightweight scoring between query and document token representations. This approach preserves significantly more contextual information than single-vector methods while still allowing independent encoding of queries and documents, and has motivated a recent trend of devising multi-vector embeddings for multimodal retrieval ( Faysse et al. 2025 ; Xu et al. 2025 ; Günther et al. 2025 ) . Despite their effectiveness, multi-vector methods incur substantial efficiency costs in terms of index size , retrieval latency and feasibility . In these methods, each image is encoded into hundreds of patch embeddings, and each query text into several token embeddings, all of which must be stored and compared during retrieval. This results in large index sizes and slower retrieval processing. Moreover, multimodal-to-multimodal retrieval becomes impractical where both query and candidate sides have images, as similarity computation for each pair of query and candidate involves interactions between thousands of query tokens and thousands of candidate tokens, making both training and inference prohibitive due to computational demands. Figure 1: Different multimodal retrieval frameworks at training-time. Upper Left: Single vector retrieval method computes a score for each pair of query and candidate and uses contrastive objective to maximize the score for corresponding pairs. Upper Right: In multi-vector retrieval, scores are computed via maximum similarity across vector pairs and aggregated before applying the contrastive objective. Lower: MetaEmbed structures query and candidate vectors into hierarchical nested groups and trains coarse-to-fine multi-vector embeddings that enable scalable retrieval with flexible test-time control over embedding granularity. In this work, we propose MetaEmbed as a scalable late-interaction training recipe that advances multimodal retrieval with a flexible multi-vector method, illustrated in Figure 1 . Instead of encoding the query and candidate into one vector, we introduce a small number of learnable Meta Tokens appended to the input sequence of the query and candidate. Their last-layer hidden states serve as a set of contextualized representations for late interaction, namely Meta Embeddings . To enable flexible late interaction, where users can trade off retrieval accuracy against computational budget and retrieval latency, we draw inspiration from Matryoshka Representation Learning ( Kusupati et al. 2022 ) and design the Matryoshka Multi-Vector Retrieval (MMR) module in MetaEmbed . By performing contrastive learning across parallel nested groups of representations at training-time , the model learns coarse-to-fine multi-vector embeddings that can be selectively utilized for late interactions depending on the computation budget at test-time . Increasing the number of Meta Embeddings used at indexing improves the retrieval quality at the cost of index storage budget and retrieval latency, thus enabling test-time scaling in multimodal retrieval. We first validate MetaEmbed on the Massive Multimodal Embedding Benchmark (MMEB) ( Jiang et al. 2024 ) and Visual Document Retrieval Benchmarks (ViDoRe) v2 ( Faysse et al. 2025 ; Macé et al. 2025 ) , which represent a comprehensive suite of retrieval tests covering images, text and visual documents. Our experiments show that MetaEmbed achieves state-of-the-art retrieval performance across diverse scenarios. To further examine its generality and training scalability, we evaluate MetaEmbed with different VLM architectures and model sizes, including decoder-only models such as Qwen2.5-VL ( Bai et al. 2025 ) and PaliGemma ( Beyer et al. 2024 ) , as well as cross-attention-based models such as Llama-3.2-Vision ( Grattafiori et al. 2024 ) . Notably, test-time scalability remains effective even at the 32B scale, showing minimal diminished returns with larger models. We hope MetaEmbed charts a path toward multimodal retrieval systems that are both accurate and deployable at scale, advancing the pursuit of generality, efficiency, and flexibility. 2 Related Work Multimodal Embedding. These methods aim to project heterogeneous inputs, such as text and images, into a shared representation space for cross-modal understanding and retrieval (e.g Frome et al. 2013 ; Kiros et al. 2014 ; Faghri et al. 2018 ). More recent large scale models such as CLIP ( Radford et al. 2021 ) , MetaCLIP ( Xu et al. 2023 ; Chuang et al. 2025 ) , BLIP ( Li et al. 2022 ) and SigLIP ( Zhai et al. 2023 ) encode each modality independently and apply contrastive training to enforce cross-modal alignment. More recent methods are built upon stronger VLMs ( Xiao et al. 2024 ; Kong et al. 2025 ; Qin et al. 2025 ; Ju and Lee 2025 ; Lin et al. 2025 ) . For instance, VLM2Vec ( Jiang et al. 2025 ) adapts Phi-3.5-V ( Abdin et al. 2024 ) , VLM2Vec-V2 (

Read the complete text