Closing the Modality Gap for Mixed Modality Search

Binxu Li, Yuhui ZhangXiaohan WangWeixin Liang Ludwig Schmidt Serena Yeung-Levy Stanford University · 2025

Read Closing the Modality Gap for Mixed Modality Search by Binxu Li, Yuhui ZhangXiaohan WangWeixin Liang Ludwig Schmidt Serena Yeung-Levy Stanford University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19054.txt.

Introduction

Binxu Li and five co-authors from Stanford University tackle mixed modality search in a two thousand twenty-five arXiv paper. Mixed modality search means retrieving answers from a mixed pool of images, texts, and multimodal documents.

The authors show that vision-language models such as CLIP keep image and text embeddings in separate clusters. That gap causes ranking bias within each type and failed fusion across types. Their fix, GR-CLIP, is a light post-hoc calibration that removes the gap. On their new MixBench benchmark it beat plain CLIP by up to twenty-six points while using far less computing power than rival models.

Catalog record

CREATORBinxu Li, Yuhui ZhangXiaohan WangWeixin Liang Ludwig Schmidt Serena Yeung-Levy Stanford University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-19054

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Closing the Modality Gap for Mixed Modality Search Binxu Li, Yuhui ZhangXiaohan WangWeixin Liang Ludwig Schmidt Serena Yeung-Levy Stanford University Abstract Mixed modality search—retrieving information across a heterogeneous corpus composed of images, texts, and multimodal documents—is an important yet underexplored real-world application. In this work, we investigate how contrastive vision-language models, such as CLIP, perform on the mixed modality search task. Our analysis reveals a critical limitation: these models exhibit a pronounced modality gap in the embedding space, where image and text embeddings form distinct clusters, leading to intra-modal ranking bias and inter-modal fusion failure. To address this issue, we propose GR-CLIP, a lightweight post-hoc calibration method that removes the modality gap in CLIP’s embedding space. Evaluated on MixBench—the first benchmark specifically designed for mixed modality search—GR-CLIP improves NDCG@10 by up to 26 percentage points over CLIP, surpasses recent vision-language generative embedding models by 4 percentage points, while using 75 × \times less compute. † † footnotetext: \star Equal contribution. \dagger Correspondence to: [email protected] (designed and supervised the project). Project page: https://yuhui-zh15.github.io/MixedModalitySearch/ 1 Introduction Information in the digital world exists across multiple modalities—text, images, video, audio, and their various combinations. While traditional retrieval systems have primarily focused on searching within a homogeneous corpus (e.g., text-to-text or text-to-image retrieval) ( robertson2009probabilistic ; rwdense ; lee2018stacked ; clip ) , real-world applications increasingly demand the ability to search and retrieve relevant content across heterogeneous modalities (e.g., text-to-{text, image, or both} retrieval) vovage . For instance, a user searching for “Mountain Fuji” might expect to find text documents, standalone images, and multimodal webpages that combine both modalities to describe the mountain (Figure 1 a). Despite its practical importance, the task of mixed modality search remains largely underexplored vovage . The central challenge lies in constructing a unified embedding space where semantically similar content across modalities—such as an image and a textual description of “Mountain Fuji”—can be mapped to nearby locations. This enables accurate measurement of semantic similarity between queries and documents, regardless of their modality. Recent advances in multimodal contrastive learning, particularly CLIP-based models ( clip ; openclip ; siglip ) , offer a promising solution by aligning text and image embeddings through training on large-scale paired image-text datasets. In this work, we investigate how well these contrastive models perform in realistic mixed modality search scenarios. Specifically, CLIP consists of two separate encoders for vision and language ( clip ) . For each corpus item, we encode image-only and text-only documents using their respective encoders. For multimodal documents containing both image and text, we compute a linear combination of the image and text embeddings to represent them (Figure 1 b). Once the embeddings are obtained, we perform similarity search by computing cosine similarity between the query embedding and each corpus item, and evaluate performance using standard retrieval metrics such as NDCG@10 ndcg , which measures the quality of the top-10 ranked results based on relevance. Our analysis reveals a fundamental limitation of CLIP-style contrastive models: they exhibit a pronounced modality gap ( mindgap ; diag ; c3 ) in their embedding space, significantly degrading retrieval performance in mixed modality settings. Although these models are trained to align image-text pairs, image and text embeddings form separate clusters and remain far apart in the embedding space (Figure 1 c). This clustering causes a strong intra-modal ranking bias (§ 3 ), where similarities between items of the same modality (e.g., image-to-image or text-to-text) are much higher than those across modalities (e.g., image-to-text), skewing retrieval rankings (Figure 1 d). For instance, given the text query “Mountain Fuji,” an image that depicts Mountain Fuji is ranked below an unrelated text snippet like “this is a great paper.” Additionally, the modality gap hurts inter-modal fusion (§ 4 ): combining image and text embeddings via linear interpolation often pushes the features to a suboptimal region, weakening their semantics and performing worse than using image or text alone. Figure 1: Overview of mixed modality search. (a) Problem Formulation: Mixed modality search aims to retrieve relevant information from a heterogeneous corpus containing multimodal documents. This is achieved by embedding both the query and documents, followed by similarity-based retrieval. (b) Embedding Method: Unimodal documents are embedded using CLIP’s modality-specific encoder, while multimodal documents are embedded via a weighted fusion of image and text features. (c) Modality Gap: CLIP’s embedding space exhibits a modality gap: embeddings form distinct clusters for each modality and remain largely separated across modalities. (d) Cosine Similarity Across Modalities: Due to this modality gap, documents that share the same modality as the query tend to have higher cosine similarity scores and are ranked higher, introducing systematic ranking bias. (e) Performance on MixBench: On our newly created MixBench benchmark—specifically designed for the task of mixed modality search—GR-CLIP, a lightweight post-hoc calibration method that closes the modality gap, significantly improves performance and outperforms the state-of-the-art VLM2Vec ( vlm2vec ) baseline with substantially lower computational cost. To address the ranking bias and fusion failure caused by the modality gap, we introduce GR-CLIP , a lightweight post-hoc calibration method that removes the modality gap in CLIP’s embedding space (GR stands for gap-removed). Prior work diag ; c3 has shown that the modality gap in CLIP-like models can be approximated by a constant vector that is orthogonal to the image and text embedding subspaces. Based on this theory, we compute the mean embeddings of all image and text data, use their difference to estimate the modality gap, and subtract this vector from all embeddings before performing retrieval. This method requires only a single pass over the dataset to compute mean embeddings and introduces negligible computational overhead. Evaluated on MixBench , our benchmark explicitly designed for mixed modality search with four subsets (Google-WIT ( googlewit ) , MSCOCO ( mscoco ) , OVEN ( oven ) , VisualNews ( visualnews ) ), GR-CLIP consistently outperforms the original CLIP models, achieving up to 26 percentage points improvement in NDCG@10. It also surpasses recent vision-language generative embedding methods such as VLM2Vec vlm2vec by 4 percentage points, while reducing computational cost by 75 \times . Furthermore, we demonstrate that our method generalizes across different CLIP variants (e.g., OpenAI CLIP ( clip ) , OpenCLIP ( openclip ) , SigLIP ( siglip ) ) and modalities (e.g., text-to-image, text-to-audio, text-to-video). In summary, we formulate and study the problem of mixed modality search , which reflects real-world scenarios such as web search engines, where users query a heterogeneous corpus containing diverse modality types. We show that state-of-the-art contrastive models suffer from ranking bias and fusion failure due to the modality gap, and we propose a lightweight post-hoc calibration method to address this issue. Our findings highlight the importance of constructing truly unified embedding spaces for effective mixed modality search. 2 Preliminaries In this section, we define the mixed modality search task, introduce its challenges and three settings related to the challenge, and describe the methods and evaluation metrics used. 2.1 Problem Formulation Mixed modality search aims to retrieve semantically relevant content when both the query and the documents may consist of different combinations of modalities, such as text, image, audio, or video. Let \mathcal{M} be the set of supported modalities (e.g., \mathcal{M}=\{\text{text},\text{image},\text{audio},\text{video}\} ). A query is denoted by q , with

Read the complete text