Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval

Siting Li, Xiang Gao, Simon Shaolei Du · 2025

Read Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval by Siting Li, Xiang Gao, Simon Shaolei Du (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15877.txt.

Introduction

Siting Li, Xiang Gao, and Simon Shaolei Du wrote this paper in two thousand twenty-five. It tackles image search where the query cares about one trait of the scene, such as a color or a pose, not the whole picture.

The team built COCO-Facet, a benchmark of thousands of trait-focused queries on COCO images. Popular CLIP-like retrievers did poorly and unevenly, since their codes stress the main subject and skip side detail. Even stronger multimodal retrievers shared the flaw. The fix is promptable codes: the same retriever, steered by a prompt, codes the image with the sought trait lit up. Two speed tricks make the codes cheap to pre-build or to fake with a linear map.

Catalog record

CREATORSiting Li, Xiang Gao, Simon Shaolei Du
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15877

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval Siting Li, Xiang Gao, Simon Shaolei Du Abstract While an image is worth more than a thousand words, only a few provide crucial information for a given task and thus should be focused on. In light of this, ideal text-to-image (T2I) retrievers should prioritize specific visual attributes relevant to queries. To evaluate current retrievers on handling attribute-focused queries, we build COCO-Facet , a COCO-based benchmark with 9,112 queries about diverse attributes of interest. We find that CLIP-like retrievers, which are widely adopted due to their efficiency and zero-shot ability, have poor and imbalanced performance, possibly because their image embeddings focus on global semantics and subjects while leaving out other details. Notably, we reveal that even recent Multimodal Large Language Model (MLLM)-based, stronger retrievers with a larger output dimension struggle with this limitation. Hence, we hypothesize that retrieving with general image embeddings is suboptimal for performing such queries. As a solution, we propose to use promptable image embeddings enabled by these multimodal retrievers, which boost performance by highlighting required attributes. Our pipeline for deriving such embeddings generalizes across query types, image pools, and base retriever architectures. To enhance real-world applicability, we offer two acceleration strategies: Pre-processing promptable embeddings and using linear approximations. We show that the former yields a 15% improvement in Recall@5 when prompts are predefined, while the latter achieves an 8% improvement when prompts are only available during inference. 1 Introduction Images offer valuable information that facilitates problem-solving and reasoning [ 64 , 36 , 37 ] . Although there may be abundant information in one image, especially when it depicts a complex scene with plenty of elements [ 12 , 54 , 41 ] , usually only a small part is critical to a task or query at a time. For both performance and efficiency considerations, there has been recent effort to focus the models on key aspects of images for better visual reasoning [ 58 , 16 , 42 ] . Similarly, for text-to-image (T2I) retrieval that helps knowledge and fact checking [ 62 , 49 ] , an ideal retriever should also be able to select images with given attributes of interest, such as a specified time, location, or object, which are not necessarily the main elements of the image. Are current T2I retrievers capable of accomplishing attribute-focused queries? We explore this question on various CLIP-like models, from CLIP [ 47 ] to recent SigLIP2 [ 53 ] , and Multimodal Large Language Model (MLLM)-based embedders like VLM2Vec [ 20 ] . Since commonly used T2I benchmarks like MSCOCO [ 32 ] and Flickr30K [ 63 ] only contain general queries on global alignment, we build a new benchmark, COCO-Facet with 9,112 attribute-focused queries of 8 types 1 1 1 The COCO-Facet benchmark can be downloaded at https://www.dropbox.com/scl/fo/hbkknl14pj5wwgpphbt6l/AC15YovOLv65Ek3hE4kib1o?rlkey=fhphyfml0uc6ctnb70v95id1n&st=qrhpzs3o&dl=0 . , based on existing annotations of COCO images [ 32 , 71 , 3 , 8 ] . We find that current retrievers behave worse on attributes other than Animals compared with general T2I retrieval on MSCOCO, and struggle more with detailed or underexplored attributes such as Time . Additionally, we discover that they fail to prioritize images with the correct-but-non-dominant attribute over images with a dominant-but-wrong attribute (See an example in Figure 1 ). Figure 1: Overview. (Above) We study the task of attribute-focused text-to-image retrieval and build COCO-Facet for benchmarking various retrievers. (Below) We show that using promptable image embeddings enhances performance on such queries, and propose two acceleration strategies to improve its applicability. As embedders with various architectures and scales all fall short on such queries, we hypothesize that retrieving with general image embeddings is inefficient and suboptimal for this task. Therefore, we propose to retrieve with promptable image embeddings: We employ MLLM-based universal embedders that can process combination of images and text, and show that using the GPT-written prompt for each category as text helps to highlight key attributes in image embeddings, demonstrating improvement on harder attributes while maintaining good performance on easier ones (See the bar plot in Figure 1 ). Additionally, we design two strategies to accelerate this pipeline: Predefining potentially useful prompts and pre-processing the promptable image embeddings, or deriving linear approximation of the embedder at test time, which can be efficiently applied to the query vectors. Our main contributions are listed as follows: • Introduced in Section 3 , our benchmark COCO-Facet on attribute-focused T2I queries is a good supplement to the current general-purposed T2I retrieval evaluation. We also provide construction pipelines, which can be utilized for building benchmarks focusing on other attributes for future research. • In Section 3.2 , we reveal the limitation of current CLIP-like retrievers and MLLM-based embedders on attribute-focused T2I queries, which affects models with various scales, image resolutions, and output dimensions. • In Section 4 , we propose to use promptable image embedding enabled by MLLM-based universal embedders as the solution. We show that it enhances the retrieval performance and generalizes over query types, image pools, and base retriever architectures. • In Section 5 , we develop two acceleration strategies for real-world usage. Our results demonstrate that the pre-processing technique increases Recall@5 by 15% when prompts are predefined, while the linear approximation achieves an 8% improvement on Recall@5 when prompts are only available at inference time. 2 Related Work Text-to-image Retrieval has been a long-standing research direction not only due to the real-world need for image search, but also as an important step in general problem-solving [ 62 , 15 ] . CLIP-like dual-encoder approaches [ 47 , 65 , 7 , 28 ] are widely-used due to their efficiency and remarkable zero-shot performance on standard T2I benchmarks like MSCOCO [ 32 ] and Flickr30K [ 63 ] . Recent MLLM-based retrievers [ 31 , 25 , 69 , 20 ] extend the input modality to joint image-text pairs. Since these models output image and text embeddings for retrieval, there have been benchmarks for more comprehensive evaluation of embeddings [ 56 , 20 , 59 ] , which include domain-focused retrieval subsets like FashionIQ [ 57 ] , EDIS [ 35 ] , OVEN [ 14 ] , Wiki-SS-NQ [ 38 ] . We note that these subsets either focus on the global semantics of the images or pre-select images with a single main subject, though in-the-wild images can be visually crowded with many attributes. Visual Genome [ 24 ] considers such complexity by annotating the images with detailed descriptions for advanced visual understanding. COCO-Attributes [ 44 ] is built with attribute annotations for multi-label classification but only targets main subjects like people, animals, and objects. On the other hand, previous studies on visual grounding and reasoning have pointed out that CLIP-like models [ 43 , 5 , 52 ] might neglect visual details. Sogi et al. 2024 shows that the retrieval performance degrades when the target objects are small. To overcome this limitation, researchers propose to use two-stage approaches [ 39 , 13 ] or guidance from an LLM [ 26 ] . Notably, pre-CLIP methods represented by scene graphs [ 21 , 48 , 46 ] might capture more visual details, but they lag behind in terms of data availability and inference

Read the complete text