Few Shots Text to Image Retrieval: New Benchmarking Dataset and Optimization Methods

Ofer Idan, Vladi Vexler, Gil Lederman, Dima Sivov, Aviad Cohen Zada, Shir Niego Komforti · 2026

Read Few Shots Text to Image Retrieval: New Benchmarking Dataset and Optimization Methods by Ofer Idan, Vladi Vexler, Gil Lederman, Dima Sivov, Aviad Cohen Zada, Shir Niego Komforti (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25891.txt.

Introduction

Idan and co-authors introduce a few-shot task for text-to-image retrieval. Standard vision-language models encode images for nearest neighbor search. They falter on compositional queries and out-of-distribution pairs, the authors note. Their benchmark, FSIR-BD, gives each text query a few reference examples. It holds 38,353 images and 303 queries, split between test and reference sets. Queries stress urban scenes and nature species in specific situations. The team also proposes two optimization methods that use the reference shots. Both beat existing baselines on mean average precision.

Catalog record

CREATOROfer Idan, Vladi Vexler, Gil Lederman, Dima Sivov, Aviad Cohen Zada, Shir Niego Komforti
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46457
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.25891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Few Shots Text to Image Retrieval: New Benchmarking Dataset and Optimization Methods Ofer Idan, Vladi Vexler, Gil Lederman, Dima Sivov, Aviad Cohen Zada, Shir Niego Komforti Abstract Pre-trained vision-language models (VLMs) excel in multimodal tasks, commonly encoding images as embedding vectors for storage in databases and retrieval via approximate nearest neighbor search (ANNS). However, these models struggle with compositional queries and out-of-distribution (OOD) image-text pairs. Inspired by human cognition’s ability to learn from minimal examples, we address this performance gap through few-shot learning approaches specifically designed for image retrieval. We introduce the Few-Shot Text-to-Image Retrieval (FSIR) task and its accompanying benchmark dataset, FSIR-BD—the first to explicitly target image retrieval by text accompanied by reference examples, focusing on the challenging compositional and OOD queries. The compositional part is divided to urban scenes and nature species, both in specific situations or with distinctive features. FSIR-BD contains 38,353 images and 303 queries, with 82% comprising the test corpus (averaging per query 37 positives, ground truth matches, and significant number of hard negatives) and 18% forming the few-shot reference corpus (FSR) of exemplar positive and hard negative images. Additionally, we propose two novel retrieval optimization methods leveraging single shot or few shot reference examples in the FSR to improve performance. Both methods are compatible with any pre-trained image encoder, making them applicable to existing large-scale environments. Our experiments demonstrate that: (1) FSIR-BD provides a challenging benchmark for image retrieval; and (2) our optimization methods outperform existing baselines as measured by mean Average Precision (mAP). Further research into FSIR optimization methods will help narrow the gap between machine and human-level understanding, particularly for compositional reasoning from limited examples. 1 Introduction The rapid increase of smart city infrastructure, CCTV networks, autonomous driving systems, and data centers has created an unprecedented demand for continuous video and image processing at scale. These applications require systems that can handle constant data streams while maintaining low latency, minimizing costs, and maximizing throughput. The challenge is compounded by the diverse nature of visual queries: some involve familiar domains but require understanding of compositionally complex concepts—such as identifying "uncovered manholes" or detecting "illegal occupation of public space by private stores" in urban management contexts. Others venture into specialized territories like medical imaging or satellite analysis, where visual concepts may be entirely unfamiliar. Across all scenarios, the stakes are high: accuracy cannot be compromised, yet collecting comprehensive training datasets for fine-tuning remains prohibitively expensive and time-consuming. At the heart of these challenges lies a fundamental question: how can we efficiently search vast image databases with natural language queries? Foundational Visual Language Models (VLMs) provide an elegant answer by learning joint representations that map both text and images into a shared semantic space. This approach enables intuitive similarity-based retrieval, where systems store precomputed image embeddings in vector databases like FAISS and identify the most semantically similar matches to text query embeddings through simple cosine distance calculations. However, VLMs are far from perfect. While they excel at recognizing frequent concepts well-represented in their training data, they falter when confronted with rare examples, compositional reasoning tasks, logical operations like negation, or nuanced relationships between concepts. When users encounter these limitations in practice, they instinctively adapt—marking a few relevant results from initial searches and using these examples to refine their queries. This natural human response reveals an important opportunity: the Few-Shot Text-Image Retrieval (FSIR) problem, where a handful of visual examples can dramatically improve search performance. Our investigation of smart city deployments revealed a compelling architectural insight. These systems follow a consistent two-phase pattern: during the understanding phase, cameras and field personnel continuously stream visual data to centralized databases, where computational efficiency is paramount. Here, lightweight dual-encoder architectures handle the heavy lifting of image vectorization and storage. But during the search phase, the user base shrinks dramatically—primarily consisting of operators and analysts—creating a computational bottleneck that can accommodate more sophisticated processing. This asymmetry presents a strategic opportunity: we can deploy powerful Multimodal Large Language Models (MLLMs) exclusively during search operations, interfacing them cleverly with the simpler encoders used for data ingestion. Building on these insights, we make several key contributions to advance the field. We begin by formalizing Few-Shot Text-Image Retrieval as a distinct research problem and systematically analyzing the limitations of existing approaches. We then introduce two innovative solutions: first, a prompt learning method that harnesses few-shot visual examples to automatically generate query-specific prompts, and second, a general-purpose MLLM approach that learns to align with external image encoder embedding spaces, enabling dynamic generation of search embeddings from text queries and reference images without requiring prior exposure to specific concepts. To facilitate rigorous evaluation, we contribute FSIR-BD, a carefully curated benchmark dataset for Composed queries of Text plus References (CTR). Our experimental results demonstrate that both proposed methods achieve state-of-the-art performance, substantially outperforming existing baselines across diverse scenarios. The practical implications of FSIR extend well beyond theoretical advances. The paradigm enables rapid, query-specific performance improvements without the traditional overhead of foundational model retraining or fine-tuning. The process is elegantly simple: users collect just one or a few examples per query—often by marking correct results from initial top-k retrievals or identifying near-miss hard negatives that highlight subtle distinctions. These reference images then guide our FSIR-PL or FSIR-CTR methods (and hopefully future innovations from the broader research community) to generate more precise search representations. This approach has already proven its worth in production environments. FSIR-PL currently serves as a core component of a cloud service for video and image semantic management, successfully deployed across multiple large-scale enterprise scenarios. Our field experience confirmed a crucial insight: while fine-tuned encoders provided modest baseline improvements, FSIR methods enabled rapid, targeted enhancements that could be applied immediately to address specific query challenges, without the need to re-encode existing datasets. This real-world validation motivated our commitment to developing the FSIR benchmark and sharing it with the research community, fostering continued innovation in this promising domain. To summarize, our contributions are as follows: • We motivate and formally define the Few-Shot Text-Image Retrieval task. • We publish FSIR-BD, a CTR benchmark with positive and hard negative samples and multiple correct images per query, spanning urban life, natural world, and OOD queries. • We introduce open-query plus few-shots prompt-learning method, the FSIR-PL, and demonstrate its high efficiency compared to the baseline. • We present FSIR-CTR, enabling scaling up a training dataset (or small-dataset training) and learning new concepts from text and reference images as searchable embeddings aligned to any image encoder’s latent space. We demonstrate its efficiency against the baseline. 2 Related Work 2.1 Vision Language Foundation Models Vision language models ( Radford et al. 2021 ; Li et al. 2021 ) pre-train image-text encoders on large-scale data containing hundreds of millions of image-caption pair. Following CLIP, many vision-language “foundation” models incorporate more data into training, introduce new architectural designs, or utilize new objective ( Li et al. 2022 ) . VLM pre-training learns correlate between image and text using large scale image-text pairs, which enables zero shot retrieval on visual recognition tasks. Given image-text pairs, Image encoder and text encoder extract features and then learn visual-language correlation according to pre-training objectives. 2.2 Weaknesses of VLMs As described above, VLMs are trained on image-caption tuples, and

Read the complete text