Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

Fanheng Kong, Jingyuan Zhang, Yahui Liu, Hongzhi Zhang, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Victoria W, Fuzheng Zhang, Guorui Zhou · 2025

Read Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval by Fanheng Kong, Jingyuan Zhang, Yahui Liu, Hongzhi Zhang, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Victoria W, Fuzheng Zhang, Guorui Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19650.txt.

Introduction

Fanheng Kong and co-authors present in 2025 a framework for search systems that must match text with images and other media. Mixed-media search is hard because data sources differ wildly and their feature spaces drift apart. Earlier work noticed the gaps between media types, but no system tackled them in an organized way.

Their framework, UNITE, works on two fronts. First, it curates training data by media type, after the first broad study of how each type's traits shape later performance. Second, a training method masks and contrasts examples in a media-aware way, which eases the tug-of-war between types. Tests across many retrieval benchmarks put the system ahead of prior methods by clear margins. The authors offer the work as a blueprint: careful data choice and tailored training, they argue, are the levers that matter.

Catalog record

CREATORFanheng Kong, Jingyuan Zhang, Yahui Liu, Hongzhi Zhang, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Victoria W, Fuzheng Zhang, Guorui Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19650

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval Fanheng Kong, Jingyuan Zhang, Yahui Liu, Hongzhi Zhang, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Victoria W, Fuzheng Zhang, Guorui Zhou Abstract Multimodal information retrieval (MIR) faces inherent challenges due to the heterogeneity of data sources and the complexity of cross-modal alignment. While previous studies have identified modal gaps in feature spaces, a systematic approach to address these challenges remains unexplored. In this work, we introduce 0.28235 0.72941 0.76863U0.28235 0.65882 0.77647n0.28235 0.59216 0.78039i0.28235 0.52157 0.78824t0.28235 0.4549 0.79608e \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: , a universal framework that tackles these challenges through two critical yet underexplored aspects: data curation and modality-aware training configurations . Our work provides the first comprehensive analysis of how modality-specific data properties influence downstream task performance across diverse scenarios. Moreover, we propose Modal-Aware Masked Contrastive Learning (MAMCL) to mitigate the competitive relationships among the instances of different modalities. Our framework achieves state-of-the-art results on multiple multimodal retrieval benchmarks, outperforming existing methods by notable margins. Through extensive experiments, we demonstrate that strategic modality curation and tailored training protocols are pivotal for robust cross-modal representation learning. This work not only advances MIR performance but also provides a foundational blueprint for future research in multimodal systems. Our project is available at https://friedrichor.github.io/projects/UNITE . 0 0 footnotetext: * Equal contributions. \dagger Corresponding author. \ddagger Work done during an internship at Kuaishou Technology. Figure 1: Performance comparison on instruction-based retrieval benchmarks (left: MMEB [ 32 ] and right: WebVid-CoVR [ 79 ] ). Our 0.28235 0.72941 0.76863U0.28235 0.65882 0.77647n0.28235 0.59216 0.78039i0.28235 0.52157 0.78824t0.28235 0.4549 0.79608e \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: achieves leading performance on various tasks, even surpassing models with larger parameter scales. 1 Introduction Multimodal Information Retrieval (MIR) is a critical research topic [ 69 , 28 ] , aiming to satisfy users’ information requirements for diverse media, such as text, images, and videos. As multimedia applications continue to progress and develop, a series of more complex and demanding tasks come to the fore, which are collectively referred to as fused-modal retrieval such as the retrieval of composite images or videos [ 53 , 108 , 79 ] . These tasks require highly sophisticated approaches for handling interleaved multimodal queries and candidates, highlighting the necessity for the development of a one-piece framework for unified multimodal representations. Recently, large multimodal models (LMMs) have shown powerful capabilities on various vision-language tasks, such as visual question answering (VQA) [ 3 , 34 , 83 , 59 ] and multimodal fact-checking [ 98 , 1 ] . In MIR field, several methods [ 32 , 52 , 24 ] have explored adapting large language models (LLMs) for retrieval tasks via contrastive learning, aiming to produce unified embeddings. For example, E5-V [ 31 ] finetunes LLaVA-NeXT [ 49 ] with text-only NLI ( 22 ) data, demonstrating the portability of LMMs for multimodal retrieval. GME ( 109 ) achieves leading performance in various image-text retrieval by finetuning Qwen2-VL [ 82 ] on diverse image-text datasets. InternVideo2 [ 86 ] stands out prominently in text-video retrieval, due to its training process that involves on hundred millions of video-text pairs. Although achieving notable success in specific domains, these models are hindered by their limited modalities, which inherently restrict their ability to fully capitalize on the potential of LMMs for generating unified multimodal embeddings. Despite ongoing researches [ 63 , 106 , 120 ] exploring training strategies for LMMs in MIR–including model architectures, training methodologies, and dataset considerations–critical questions remain unresolved. Specifically, the optimal data composition and proportions, as well as the nuanced impact of different modal data configurations across various retrieval tasks, have yet to be comprehensively understood. In our empirical investigations, we find that inappropriate combinations of multimodal data or data training sequences can easily disrupt the harmonious integration of diverse data modalities, causing the model to misinterpret the relationships between different types of information. In this paper, through a meticulous analysis of how various data compositions impact retrieval results, we make efforts to achieve a balance among the three modalities of text, image, and video. In particular, we find that introducing a small number of fine-grained video-text pairs during the retrieval adaptation stage can significantly enhance the fine-grained retrieval performance of LMMs. Meanwhile, existing works [ 31 , 109 ] have proven that there is a substantial distribution gap in the data of various modalities within the feature space. Simply mixing data from different modalities for contrastive learning will affect the quality of representation learning ( i.e. , introducing noise). Therefore, we propose Modal-Aware Masked Contrastive Learning (MAMCL) to balance the competitive relationships among the instances of various modalities. Finally, we develop a UNI versal mul T imodal E mbedder, named 0.28235 0.72941 0.76863U0.28235 0.65882 0.77647n0.28235 0.59216 0.78039i0.28235 0.52157 0.78824t0.28235 0.4549 0.79608e \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: , that effectively handles text, images, videos, and their combinations. During the training, we employ an evolving training strategy to gradually unlock the retrieval capability of LMMs. After two training stages ( i.e. , retrieval adaptation and instruction tuning), we validate our approach through comprehensive evaluation on 40+ diverse retrieval tasks, spanning coarse-grained, fine-grained, and instruction-based retrieval across text, images, and video. Experimental results show that 0.28235 0.72941 0.76863U0.28235 0.65882 0.77647n0.28235 0.59216 0.78039i0.28235 0.52157 0.78824t0.28235 0.4549 0.79608e \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: achieves state-of-the-art performance in various tasks, and outperforms existing specialized domain-specific models in numerous scenarios. In summary, our contributions are as follows: • We unveil the appropriate method for curating modality data during the process of learning unified multimodal embeddings, so as to balance the gap between the feature spaces of different modalities. • We propose MAMCL to mitigate the competitive relationships among the instances of various modalities. Specially, the MAMCL strategy can serve as a general method and be applied to any extended modal scenarios. • To the best of our knowledge, 0.28235 0.72941 0.76863U0.28235 0.65882 0.77647n0.28235 0.59216 0.78039i0.28235 0.52157 0.78824t0.28235 0.4549 0.79608e \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: is the first model capable of enabling text, image, video and fused-modal to concurrently focus on fine-grained and instruction-based retrieval tasks. Notably, 0.28235 0.72941 0.76863U0.28235 0.65882 0.77647n0.28235 0.59216 0.78039i0.28235 0.52157 0.78824t0.28235 0.4549 0.79608e \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: \__color_backend_reset: achieves state-of-the-art performance in 40+ different tasks, surpassing specialized domain-specific models in numerous scenarios, as shown in Figure 1 . 2 Related Work Large Multimodal Models. LMMs have mushroomed, showcasing impressive capabilities in multimodal information understanding [ 41 , 107 ] . Pioneering efforts such as LLaVA [ 50 , 48 ] , MiniGPT-4 [ 117 ] , InternVL [ 15 , 118 ] , and Qwen-VL [ 82 , 5 ] , bridge visual encoders and LLM with lightweight intermediate architectures. Recent advances [ 40 , 88 , 120 , 95 , 58 , 43 ] have extended these techniques from static images to sequential videos, demonstrating promising results in video understanding through frame-based processing. These developments have catalyzed the widespread adoption of LMMs across various applications [ 51 , 66 , 110 , 37 ] . Cross-Modal Retrieval. Traditional multimodal retrieval primarily focuses on cross-modal scenarios, particularly text-image retrieval [ 81 , 84 ] and text-video retrieval [ 116 , 29 ] . Foundational methods, such as

Read the complete text