Inquire : A Natural World Text-to-Image Retrieval Benchmark

Edward Vendrow, Omiros Pantazis, Alexander Shepard, Gabriel Brostow, Kate E. Jones, Oisin Mac Aodha, Sara Beery, Grant Van Horn · 2024

Read Inquire : A Natural World Text-to-Image Retrieval Benchmark by Edward Vendrow, Omiros Pantazis, Alexander Shepard, Gabriel Brostow, Kate E. Jones, Oisin Mac Aodha, Sara Beery, Grant Van Horn (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02537.txt.

Description

Abstract We introduce Inquire , a text-to-image retrieval benchmark designed to challenge multimodal vision-language models on expert-level queries. Inquire includes iNaturalist 2024 (iNat24), a new dataset of five million natural world images, along with 250 expert-level retrieval queries. These queries are paired with all relevant images comprehensively labeled within iNat24, comprising 33,000 total matches.

Catalog record

CREATOREdward Vendrow, Omiros Pantazis, Alexander Shepard, Gabriel Brostow, Kate E. Jones, Oisin Mac Aodha, Sara Beery, Grant Van Horn
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE107 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-02537

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Inquire : A Natural World Text-to-Image Retrieval Benchmark Edward Vendrow, Omiros Pantazis, Alexander Shepard, Gabriel Brostow, Kate E. Jones, Oisin Mac Aodha, Sara Beery, Grant Van Horn Abstract We introduce Inquire , a text-to-image retrieval benchmark designed to challenge multimodal vision-language models on expert-level queries. Inquire includes iNaturalist 2024 (iNat24), a new dataset of five million natural world images, along with 250 expert-level retrieval queries. These queries are paired with all relevant images comprehensively labeled within iNat24, comprising 33,000 total matches. Queries span categories such as species identification, context, behavior, and appearance, emphasizing tasks that require nuanced image understanding and domain expertise. Our benchmark evaluates two core retrieval tasks: (1) Inquire-Fullrank , a full dataset ranking task, and (2) Inquire-Rerank , a reranking task for refining top-100 retrievals. Detailed evaluation of a range of recent multimodal models demonstrates that Inquire poses a significant challenge, with the best models failing to achieve an mAP@50 above 50%. In addition, we show that reranking with more powerful multimodal models can enhance retrieval performance, yet there remains a significant margin for improvement. By focusing on scientifically-motivated ecological challenges, Inquire aims to bridge the gap between AI capabilities and the needs of real-world scientific inquiry, encouraging the development of retrieval systems that can assist with accelerating ecological and biodiversity research. Figure 1: Inquire is a text-to-image retrieval benchmark of 250 expert-level queries comprehensively labeled over a new five million image dataset. The queries span a range of ecological and biodiversity concepts, requiring reasoning, image understanding, and domain expertise. 1 Introduction Recent advances in multimodal learning have resulted in advanced models Radford et al. [2021] , Liu et al. [2023] , Achiam et al. [2023] that demonstrate remarkable generalization capabilities in zero-shot classification Radford et al. [2021] , Zhai et al. [2023] , visual question-answering (VQA) Li et al. [2022] , Yu et al. [2022] , Alayrac et al. [2022] , Li et al. [2023] , and image retrieval Yu et al. [2022] , Li et al. [2023] . These models offer the potential to assist in the exploration, organization, and extraction of knowledge from large image collections. However, despite this success, there remains a significant gap in the evaluation of these models on domain-specific, expert-level queries, where nuanced understanding and precise retrieval are critical. Addressing this gap is essential for future deployment in specialized fields such as biodiversity monitoring and biomedical imaging, among other scientific disciplines. Previous studies of the multimodal capabilities of this new generation of models have primarily focused on the task of VQA. In VQA, it has been demonstrated that there remains a large performance gap between state-of-the-art models and human experts in the context of challenging perception and reasoning queries such as those found on college-level exams Yue et al. [2024] , Zhong et al. [2023] . However, no such expert-level benchmark exists for image retrieval . The most commonly used text-to-image retrieval benchmarks are derived from image captioning datasets, and contain simple queries related to common everyday categories Young et al. [2014] , Lin et al. [2014] . Current multimodal models achieve near perfect performance on some of these benchmarks, indicating that they no longer pose a challenge (e.g., BLIP-2 Li et al. [2023] scores 98.9 on Flickr30K Young et al. [2014] top-10). Existing retrieval datasets are generally small Philbin et al. [2007] , Philbin et al. [2008] , Young et al. [2014] , Lin et al. [2014] , limited to a single visual reasoning task (e.g., landmark-location matching Philbin et al. [2007] , Philbin et al. [2008] , Weyand et al. [2020] ), and lack concepts that would require expert knowledge Philbin et al. [2007] , Philbin et al. [2008] , Weyand et al. [2020] , Young et al. [2014] , Lin et al. [2014] . These limitations impede our ability to track and improve image retrieval capabilities. Figure 2: Category breakdown for the fine-grained queries that make up Inquire . Each query category falls under one of the following supercategories: Species, Context, Behavior, or Appearance. A domain that is well-suited for studying this problem is the natural world, where images collected by enthusiast volunteers provide vast and largely uncurated sources of publicly available scientific data. In particular, the iNaturalist iNa [b] platform contains over 180 million species images and contributes immensely to research in biodiversity monitoring Chandler et al. [2017] , Lohan [2024] . These images also contain a wealth of “secondary data” not reflected in their species labels Pernat et al. [2024] , including crucial insights into interactions, behavior, morphology, and habitat that could be uncovered through searches. However, the time-consuming and expert-dependent analysis needed to extract such information prevents scientists from taking advantage of this valuable data at scale. This cost is amplified as scientists typically want to retrieve multiple relevant images for each text query, so that they can track changes of a property over space and time Young et al. [2019] . This domain serves as an ideal testbed for expert image retrieval, as these images contain expert-level diverse and composite visual reasoning problems, and progress in this field will enhance impactful scientific discovery. In this work, we introduce Inquire , a new dataset and benchmark for expert-level text-to-image retrieval and reranking on natural world images. Inquire includes the iNat24 dataset and 250 ecologically motivated retrieval queries. The queries span 33,000 true-positive matches, pairing each text query with all relevant images that we comprehensively labeled among iNat24’s five million natural world images. iNat24 is sampled from iNaturalist iNa [b] , and contains images from 10,000 different species collected and annotated by citizen scientists, providing significantly more data for researchers interested in fine-grained species classification. The queries contained within Inquire come from discussions and interviews with a range of experts including ecologists, biologists, ornithologists, entomologists, oceanographers, and forestry experts. Our evaluation of multimodal retrieval methods demonstrates that Inquire poses a significant challenge, necessitating the development of models able to perform expert-level retrieval within large image collections. A key finding from our experiments is that reranking, a technique typically used in text retrieval Nogueira and Cho [2019] , Khattab and Zaharia [2020] , Karpukhin et al. [2020] , offers a promising avenue for improvement in image retrieval. We hope that Inquire will inspire the community to build next-generation image retrieval methods towards the ultimate goal of accelerating scientific discovery. We make Inquire , the iNat24 dataset, pre-computed outputs from state-of-the-art models, and code for evaluation available at https://inquire-benchmark.github.io/ . Figure 3: Proportion of queries in Inquire associated with each iconic group of species. Dataset Images Queries MpQ Expert Flickr30k Young et al. [2014] 1,000 5k 1 ✗ COCO Lin et al. [2014] 5,000 25k 1 ✗ Inquire 5,000,000 250 1–1.5k ✓ Table 1: Comparison to common datasets used to evaluate text-to-image retrieval Gadre et al. [2023] . Unlike other datasets, Inquire has significantly more images and many matches per query rather than exactly one. MpQ: Matches per query 2 Related Work Vision-Language Models (VLMs). Large web-sourced datasets containing paired text and images have enabled recent advances in powerful VLMs Bordes et al. [2024] , Zong et al. [2024] . Contrastive methods such as CLIP Radford et al. [2021] and ALIGN Jia et al. [2021] , among

Read the complete text