Maybe you are looking for CroQS Cross-modal Query Suggestion for Text-to-Image Retrieval

Giacomo Pacini, Fabio Carrara, Nicola Messina, Nicola Tonellotto, Giuseppe Amato, Fabrizio Falchi · 2024

Read Maybe you are looking for CroQS Cross-modal Query Suggestion for Text-to-Image Retrieval by Giacomo Pacini, Fabio Carrara, Nicola Messina, Nicola Tonellotto, Giuseppe Amato, Fabrizio Falchi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13834.txt.

Introduction

Giacomo Pacini and co-authors teach search tools to hint at what a seeker may want in a 2024 paper. Hinting works well for text hunts, but the hunt in view runs from words to images, where hints have been little tried.

Their hint takes the form of a small edit to the words, one that points at a tight visual clan in the set. The team built a test set with first queries, grouped finds, and hints penned by people for each group. They also set scores for how true, how sharp, and how close to the first query a hint is. Tools based on big word models and on image captions both did well, though a gap to human hints stayed. Scores for sharpness more than doubled off the base query.

Catalog record

CREATORGiacomo Pacini, Fabio Carrara, Nicola Messina, Nicola Tonellotto, Giuseppe Amato, Fabrizio Falchi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13834

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Maybe you are looking for CroQS Cross-modal Query Suggestion for Text-to-Image Retrieval Giacomo Pacini, Fabio Carrara, Nicola Messina, Nicola Tonellotto, Giuseppe Amato, Fabrizio Falchi Abstract Query suggestion, a technique widely adopted in information retrieval, enhances system interactivity and the browsing experience of document collections. In cross-modal retrieval, many works have focused on retrieving relevant items from natural language queries, while few have explored query suggestion solutions. In this work, we address query suggestion in cross-modal retrieval, introducing a novel task that focuses on suggesting minimal textual modifications needed to explore visually consistent subsets of the collection, following the premise of “Maybe you are looking for”. To facilitate the evaluation and development of methods, we present a tailored benchmark named CroQS. This dataset comprises initial queries, grouped result sets, and human-defined suggested queries for each group. We establish dedicated metrics to rigorously evaluate the performance of various methods on this task, measuring representativeness, cluster specificity, and similarity of the suggested queries to the original ones. Baseline methods from related fields, such as image captioning and content summarization, are adapted for this task to provide reference performance scores. Although relatively far from human performance, our experiments reveal that both LLM-based and captioning-based methods achieve competitive results on CroQS, improving the recall on cluster specificity by more than 115% and representativeness mAP by more than 52% with respect to the initial query. The dataset, the implementation of the baseline methods and the notebooks containing our experiments are available here: paciosoft.com/CroQS-benchmark/ 1 Introduction Figure 1 : Cross-modal Query Suggestion . Given an initial query q_{0} and an image collection \mathcal{I} , a cross-modal query suggestion system \mathfrak{F} returns a set of query suggestions \mathcal{Q} based on the visual content of the result set R(q_{0},\mathcal{I}) . Ideally, each suggestion \hat{q}_{i}\in\mathcal{Q} should represent a semantically coherent group C_{i}\subset R(q_{0},\mathcal{I}) . Query formulation is a central challenge in Information Retrieval (IR), as it consists of aligning user queries with their actual information needs. Query expansion and query suggestion are two families of query reformulation approaches that aim to better align the search results with the user’s information need [ 19 ] . The first one is generally automatic and modifies the IR system’s internal representation of the original query. Query suggestion, instead, is an interactive approach that proposes different queries to the user to refine the search. Those techniques either base their reformulations on the initial search results — as happens in Rocchio’s algorithm [ 23 ] with its Pseudo Relevance Feedback (PRF) — or on external knowledge sources, such as thesauri, ontologies, or large-scale language models. In the case of PRF, the system uses feedback from the initial set of retrieved documents to automatically adjust the query, while knowledge-based approaches leverage pre-defined relationships between terms to suggest more effective formulations. Historically, IR and query reformulation approaches have been designed for document retrieval, where a textual query is employed to retrieve textual documents. Therefore, methods, architectures, and datasets that define and solve this task mainly consider the text modality alone. Recently, cross-modal retrieval and, in particular, text-to-image retrieval is becoming the key to searching large image and video datasets, thanks to the recent development of powerful models that embed images and texts in the same semantic feature space [ 16 , 15 , 21 ] . In particular, CLIP [ 21 ] has simplified research in this area and enabled further possibilities. It embeds text and visual data into the same embedding space and can easily retrieve the most relevant images or videos by computing a maximum inner-product search with the textual query embedding. This model is nowadays the de facto standard in multimedia retrieval, and it has been largely employed in interactive video browsing competitions [ 13 , 1 , 7 ] . Given the growing interest in cross-modal search applications, there is a pressing need to develop a tool that interactively suggests queries in cross-modal scenarios — an area that is under-explored in the literature. For this reason, in this work, we aim to explore the query suggestion task in the scenario where queries are provided in natural language, and the search database comprises a potentially large set of unannotated images. We refer to this task as cross-modal query suggestion , which consists of proposing suitable modifications to the initial user query based on visual clues present in the retrieved images. Ideally, each suggestion should represent different aspects of the original result set and allow the user to interactively understand the semantic groups of images that populate the search collection. Figure 1 shows a schematic overview of the task with an example. The proposed task is challenging for many reasons. In a cross-modal retrieval scenario, document collections are usually made of plain images that do not carry captions, textual descriptions, or any additional metadata of the documents. Furthermore, the information to extract from an image document can change depending on the context. For this reason, textual features statically provided with the images — such as the captions of image-caption datasets — are not effective for the query suggestion task since they lose some of the possible meanings of an image. To solve this problem, it is necessary to define a mechanism that, given an initial query and a semantic representation of the result images, constructs a suggested query. Considering the novelty of this problem in cross-modal retrieval, it is also necessary to define a method and a benchmark for method comparison. To this aim, we introduce the CroQS benchmark. We developed CroQS through a semi-automatic, human-supervised process. We generated every semantic cluster of images from an initial query and a clustering step, further tuning the obtained sets through careful human judgment. A human annotator also defined a reference query suggestion per group to have a solid ground truth. In order to objectively evaluate different methods with the benchmark, we defined a set of metrics, and we adapted several baseline methods from related fields — such as image captioning and content summarization — to solve the task. To summarize, the contributions of this paper are threefold: • We define and tackle the novel task of cross-modal query suggestion, which extends classical text-based query suggestion by directly incorporating latent high-level information extracted from the retrieved images. • We provide the CroQS benchmark, accompanied by proper evaluation metrics, to quantitatively evaluate models on this new challenging task. • We propose some reasonable baselines for this benchmark by adapting some captioning-based and LLM-based methods. Our results show that the methods derived from captioning are more effective in building queries that are specific to their group of images, while LLM-based approaches are balanced throughout the properties and tend to build suggestions very similar to the initial queries. 2 Related Works Cross-modal query suggestion is a task spanning different research domains. Besides classical IR query suggestion, this task borrows from methods extracting image content as textual descriptions, thus relating to image group captioning and image captioning in general. While image group captioning is about building a natural language sentence that describes a set of images, cross-modal query suggestion is about generating a more detailed query that better explains a group of query results. 2.1 Query Suggestion In dense retrieval [ 9

Read the complete text