MMMORRF: Multimodal Multilingual MOdularized Reciprocal Rank Fusion
Saron Samuel, Dan DeGenaro, Jimena Guallar-Blasco, Kate Sanders, Oluwaseun Eisape, Tanner Spendlove, Arun Reddy, Alexander Martin, Andrew Yates, Eugene Yang, Cameron Carpenter, David Etter, Efsun Kayi, Matthew Wiesner, Kenton Murray, Reno Kriz · 2025
Read MMMORRF: Multimodal Multilingual MOdularized Reciprocal Rank Fusion by Saron Samuel, Dan DeGenaro, Jimena Guallar-Blasco, Kate Sanders, Oluwaseun Eisape, Tanner Spendlove, Arun Reddy, Alexander Martin, Andrew Yates, Eugene Yang, Cameron Carpenter, David Etter, Efsun Kayi, Matthew Wiesner, Kenton Murray, Reno Kriz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20698.txt.
Introduction
Saron Samuel and fifteen co-authors build a video search system that listens as well as looks in a 2025 paper. Videos carry pictures, sounds, speech, and on-screen text, yet leading search models lean almost entirely on the visual channel. Benchmarks reward that bias by asking visual questions.
Their system pulls text and features from both audio and video. It then merges the separate ranked lists with a fusion rule that weights each channel by how much it should matter for the query. On two benchmarks built around real information needs rather than picture descriptions, the system lifted a standard ranking score sharply. It beat top multimodal encoders by eighty-one percent and single-channel search by thirty-seven percent. The lesson is plain: a question about what was said cannot be answered by frames alone. Modular fusion lets each sense speak.
Catalog record
| CREATOR | Saron Samuel, Dan DeGenaro, Jimena Guallar-Blasco, Kate Sanders, Oluwaseun Eisape, Tanner Spendlove, Arun Reddy, Alexander Martin, Andrew Yates, Eugene Yang, Cameron Carpenter, David Etter, Efsun Kayi, Matthew Wiesner, Kenton Murray, Reno Kriz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20698
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMMORRF: Multimodal Multilingual MOdularized Reciprocal Rank Fusion Saron Samuel, Dan DeGenaro, Jimena Guallar-Blasco, Kate Sanders, Oluwaseun Eisape, Tanner Spendlove, Arun Reddy, Alexander Martin, Andrew Yates, Eugene Yang, Cameron Carpenter, David Etter, Efsun Kayi, Matthew Wiesner, Kenton Murray, Reno Kriz Abstract . Videos inherently contain multiple modalities, including visual events, text overlays, sounds, and speech, all of which are important for retrieval. However, state-of-the-art multimodal language models like VAST and LanguageBind are built on vision-language models (VLMs), and thus overly prioritize visual signals. Retrieval benchmarks further reinforce this bias by focusing on visual queries and neglecting other modalities. We create a search system MMMORRF that extracts text and features from both visual and audio modalities and integrates them with a novel modality-aware weighted reciprocal rank fusion. MMMORRF is both effective and efficient, demonstrating practicality in searching videos based on users’ information needs instead of visual descriptive queries. We evaluate MMMORRF on MultiVENT 2.0 and TVR, two multimodal benchmarks designed for more targeted information needs, and find that it improves nDCG@20 by 81% over leading multimodal encoders and 37% over single-modality retrieval. Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730157 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Video search CCS: Information systems Combination, fusion and federated search CCS: Information systems Multilingual and cross-lingual retrieval 1. Introduction Online content has increasingly shifted toward video, many of which are multilingual and span multiple modalities, including visual events, spoken audio, embedded text, and non-speech sounds, such as music. As a result, users now require retrieval systems that handle both linguistic and modality diversity. Current commercial systems, such as YouTube search, rely heavily on non-visual metadata like titles, descriptions, and user engagement signals. While useful, these features are often incomplete or missing. To address this, we propose a multilingual video retrieval system that relies solely on visual, audio, and embedded textual content. Figure 1 . Diagram of MMMORRF pipeline and fusion system for event-centric video retrieval. Components to the left of the blue dotted line are processed at indexing time; components to the right are processed at search time. Prior research has shown strong results using vision-language models ( Chen et al., 2023b ; Zhu et al., 2024 ; Wang et al., 2024 ) on academic benchmarks like VALOR-32k ( Chen et al., 2023a ) and MSR-VTT ( Xu et al., 2016 ) . However, these datasets are small, English-centric, and rely on descriptive captions that do not reflect real-world search behaviors. Rather than relying on large vision and audio-language models, we introduce a practical and efficient fusion search engine, MMMORRF, that leverages information extracted using mature modality-specific technologies. Figure 1 illustrates our approach, which achieves state-of-the-art results on MultiVENT 2.0 , with more than 80% improvement over the best vision-language system. Our contributions are three-fold: (1) we present, MMMORRF, a system that combines state-of-the-art components for multilingual, event-centric video retrieval; (2) we introduce a novel modality-aware fusion method for integrating multiple modalities; and (3) we provide a comprehensive comparison with vision-language models, demonstrating the effectiveness of our approach on a more realistic video retrieval task. A video demonstrating MMMORRF is available on YouTube 1 1 1 https://youtu.be/7jLisTeSjKM , and the implementation is available on GitHub. 2 2 2 https://github.com/hltcoe/video-retrieval-demo 2. Background Multilingual and cross-language retrieval. Multilingual information retrieval (MLIR) involves retrieving documents in multiple languages using queries in a different language ( Oard and Dorr, 1998 ; Lawrie et al., 2024 ) . Cross-language information retrieval (CLIR) is a specialized case where the corpus is monolingual but queries are in another language ( Lawrie et al., 2023a ) . Recent advancements focus on end-to-end retrieval systems that eliminate translation dependencies ( Nair et al., 2022 ; Li et al., 2022 ) , improving scalability in multilingual settings ( Lawrie et al., 2023b ) . Techniques such as language model pretraining ( Yang et al., 2022 ) , data curation and translation ( Nair et al., 2022 ; Nair et al., 2023 ) , and knowledge distillation ( Li et al., 2022 ; Yang et al., 2024b ; Yang et al., 2024a ) have enhanced the effectiveness of neural retrieval models ( Lawrie et al., 2024 ; Yang et al., 2024a ) . Cross-modal retrieval. Cross-modal retrieval retrieves data in one modality using queries from another, such as retrieving videos using textual queries. Traditional methods treat each modality independently ( Wang et al., 2016 ) , while modern deep learning enables joint multimodal representations, allowing direct comparisons across modalities. Models based on Contrastive Language–Image Pretraining ( CLIP ) ( Radford et al., 2021 ) have advanced this field and are further adapted for video retrieval ( Fang et al., 2021a ; Jin et al., 2022 ; Xue et al., 2023 ) . However, most existing retrieval models remain limited to single-language and single-modality settings, whereas real-world online content spans multiple languages and integrates textual, auditory, and visual information, posing new challenges for retrieval systems. Video retrieval. Video retrieval involves identifying the most relevant videos for a given query ( Cao et al., 2024 ; Tang et al., 2024 ) . Early systems relied on metadata such as titles and descriptions ( Wactlar and Christel, 2002 ) , but deep learning has enabled retrieval using visual, audio, and embedded text features ( Bain et al., 2021 ; Fang et al., 2021b ) . Recent multimodal systems integrate these signals via fusion transformers ( Chen et al., 2023a ; Wang et al., 2024 ) . However, most existing benchmarks focus on monolingual collections and simple queries ( Sanders and Van Durme, 2024 ) , making the task of true multilingual, multimodal retrieval particularly challenging to evaluate. Extracted video text has been explored in monolingual retrieval ( Wu et al., 2025 ) , video question-answering ( Zhao et al., 2022 ) , and captioning ( Wu et al., 2023 ) , but robust multilingual text extraction for video retrieval remains underexplored ( Sanders et al., 2023 ) . 3. Video Retrieval Pipeline In this section, we present the components of our MMMORRF retrieval system for multilingual video retrieval. Visual Content. To capture signals from the visual content, we develop a sub-pipeline that selects, encodes, and searches video frames, illustrated at the top of Figure 1 . To balance efficiency and effectiveness, we uniformly sample 16 frames per video. Through pilot testing on MultiVent 2.0 training set ( Kriz et al., 2024 ) , we found that the benefit of sampling more frames saturates dramatically after 16. We also investigated key-frame detection tools, such as PySceneDetect ( Castellano, [n.d.] ) , but found no substantial difference between uniform sampling. To bridge the modality gap between textual queries and video frames, we use SigLIP ( Zhai et al., 2023 ) , a variant of CLIP that replaces the standard contrastive learning softmax normalization with a sigmoid loss function. For efficient retrieval, frame embeddings are indexed using FAISS , a scalable vector search engine ( Douze et al., 2024 ) . At search time, queries are encoded