Indexing Multimodal Language Models for Large-scale Image Retrieval

Bahey Tharwat, Giorgos Kordopatis-Zilos, Pavel Suma, Ian Reid, Giorgos Tolias · 2026

Read Indexing Multimodal Language Models for Large-scale Image Retrieval by Bahey Tharwat, Giorgos Kordopatis-Zilos, Pavel Suma, Ian Reid, Giorgos Tolias (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13268.txt.

Introduction

Tharwat and four colleagues test whether multimodal language models can rank similar images without any extra training. They prompt a model with image pairs and turn its next-token chances into similarity scores, then re-rank shortlists from large retrieval pipelines. The models beat specialist re-rankers outside their home domains and hold up better against clutter, occlusion, and tiny objects, though they still fail when appearance changes severely.

Catalog record

CREATORBahey Tharwat, Giorgos Kordopatis-Zilos, Pavel Suma, Ian Reid, Giorgos Tolias
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58093
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13268

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Indexing Multimodal Language Models for Large-scale Image Retrieval Bahey Tharwat, Giorgos Kordopatis-Zilos, Pavel Suma, Ian Reid, Giorgos Tolias Abstract Multimodal Large Language Models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, yet their potential for vision-only tasks remains underexplored. We investigate MLLMs as training-free similarity estimators for instance-level image-to-image retrieval. Our approach prompts the model with paired images and converts next-token probabilities into similarity scores, enabling zero-shot re-ranking within large-scale retrieval pipelines. This design avoids specialized architectures and fine-tuning, leveraging the rich visual discrimination learned during multimodal pre-training. We address scalability by combining MLLMs with memory-efficient indexing and top- k k candidate re-ranking. Experiments across diverse benchmarks show that MLLMs outperform task-specific re-rankers outside their native domains and exhibit superior robustness to clutter, occlusion, and small objects. Despite strong results, we identify failure modes under severe appearance changes, highlighting opportunities for future research. Our findings position MLLMs as a promising alternative for open-world large-scale image retrieval. code: http://github.com/gkordo/mllm-ilr 1 Introduction Multimodal Large Language Models (MLLMs) OpenAI 2024 ; Team et al. 2023 ; Ataallah et al. 2024 ; Liu et al. 2023 ; Dubey et al. 2024 have recently emerged as powerful tools for bridging vision and language. Their ability to jointly process images and text enables rich cross-modal reasoning. From a computer vision perspective, MLLMs offer a unique advantage: they use natural language both as supervision during training and as an interaction mechanism at inference, providing remarkable flexibility. This makes them suitable for diverse tasks such as visual question answering (VQA) Liu et al. 2024 ; Dai et al. 2023 ; Alayrac et al. 2022 , image captioning Hua et al. 2025 ; Kim et al. 2024 , object recognition Cheng et al. 2024 ; Yao et al. 2024 ; Ren et al. 2024 , and scene understanding Shi et al. 2025 ; Sodano et al. 2024 ; Kirillov et al. 2023 . In these settings, language serves as an additional source of context, enhancing interpretability and adaptability. The appeal of MLLMs is twofold: (i) natural language supervision is expressive and diverse, enabling learning from large multimodal corpora without task-specific annotations, and (ii) text-based interaction allows zero-shot adaptation to novel tasks through prompting Liu et al. 2024 ; Alayrac et al. 2022 . This combination of scalable training and flexible deployment motivates their use beyond traditional multimodal tasks. 0.1 1 35 40 45 50 55 4 4 4 4 7 7 7 7 13 13 13 13 33 33 33 33 81 81 81 81 162 162 162 162 100 200 400 1000 2500 5000 time per query (s) mAP@1k Qwen-C Qwen3-C Qwen3R-C AMES Figure 1 : Performance vs . re-ranking time. MLLM-based re-rankers have higher per-image inference time than methods trained specifically for instance-level image retrieval with re-ranking (AMES), but under a fixed query-time budget, they achieve better retrieval performance, already with as few as 20 re-ranked images, indicated by the colored numbers. All reported methods have roughly the same memory footprint. In this work, we explore MLLMs in a training-free manner for a vision-only task, i.e . , large-scale image retrieval. Our focus is on instance-level retrieval Sivic and Zisserman 2003 ; Philbin et al. 2007 , where the goal is to find the exact object rather than a category. Specifically, we provide the query and candidate images to the model and prompt it with a simple question, e.g . “Do the images show the same object?”, requiring a binary (yes/no) response. We convert the next-token probabilities for “yes” and “no” into a similarity score, effectively turning the model into a zero-shot classifier for object identity. This constitutes an unconventional application of MLLMs, as image similarity does not inherently involve language understanding. Our goal is to repurpose their reasoning capabilities, avoiding specialized architectures Arandjelovic et al. 2016 ; Cao et al. 2020 , domain-specific training data Weyand et al. 2020 ; Radenović et al. 2019 , and costly fine-tuning typically used for instance-level retrieval. We leverage the extensive pre-training of MLLMs across diverse domains, which implicitly equips them with strong visual discrimination in an open-world setting. Interestingly, their internal design often involves comparing image patches, reminiscent of similarity-learning architectures Tan et al. 2021 ; Suma et al. 2024 , albeit without explicit supervision for this task. We integrate this similarity estimation into a large-scale retrieval pipeline as a re-ranking step on top of an efficient initial ranking using compact image descriptors. Since MLLMs require raw, often high-resolution images, direct application to large collections is impractical. To address this, we adopt indexing-friendly design choices: database images are pre-processed into patch-level features, quantized or pruned, and stored in memory-efficient structures. During retrieval, only the top- k candidates are processed by the MLLM, reducing computational overhead. Our findings show that MLLMs, when prompted appropriately, serve as effective training-free similarity estimators for instance-level retrieval, offering not only a favorable accuracy-memory tradeoff but also a favorable accuracy-speed tradeoff when operating under a sufficiently generous time budget (see Figure 1 ). Compared to task-specific models, MLLMs perform better outside their native training domains, demonstrating superior generalization across diverse object categories. We show that our indexing scheme is effective not only for training-free usage of generic MLLMs but also for MLLM models specifically fine-tuned as re-rankers. Qualitatively, MLLMs outperform global descriptors and task-specific re-rankers in challenging cases involving small objects, heavy clutter, and occlusion, showcasing strong partial matching capabilities. Despite these advantages, we identify unexpected failure modes in robustness analysis, paving the way for future research to enhance the reliability and scalability of MLLMs for similarity estimation and large-scale retrieval. 2 Related work Image retrieval. Image retrieval has largely relied on learned global descriptors. Early deep methods repurpose classification networks by pooling activation maps Babenko and Lempitsky 2015 ; Radenović et al. 2019 , while later approaches train networks explicitly for retrieval using metric learning objectives Schroff et al. 2015 ; Revaud et al. 2019 and advanced sampling strategies Sohn 2016 ; Lu et al. 2019 . Although effective, these methods require domain-specific training data, which limits adaptability. Foundation vision models such as CLIP Radford et al. 2021 introduced generic representations with strong cross-domain generalization, enabling efficient large-scale retrieval. Yet, global descriptors often lack the precision needed for fine-grained and instance-level similarity estimation. To improve precision, recent work has incorporated re-ranking based on local descriptors. State-of-the-art methods such as RRT Tan et al. 2021 , R^{2} Former Zhu et al. 2023 , and CVNet Lee et al. 2022 refine similarity estimation by matching dense features or modeling cross-image correlations with transformer or convolutional architectures. Since such dense approaches often incur high indexing costs, AMES Suma et al. 2024 is designed for an asymmetric setup with binarized descriptors Kordopatis-Zilos et al. 2022 , while dense-to-sparse CVNet Lee et al. 2024 sparsifies database descriptors. LOCORE Xiao et al. 2025 uses only a fraction of local descriptors and departs from pairwise similarity by fine-tuning an LLM for listwise re-ranking. Despite their strong performance, these methods remain computationally and memory-intensive and rely on domain-specific training. In contrast, we propose a training-free re-ranking approach based on MLLMs, leveraging their open-world generalization while avoiding specialized architectures and domain-dependent training. MLLM for image retrieval. MLLMs Dubey et al. 2024 ; Bai et

Read the complete text