Zero Shot Composed Image Retrieval

Santhosh Kakarla, Gautama Shastry Bulusu Venkata · 2025

Read Zero Shot Composed Image Retrieval by Santhosh Kakarla, Gautama Shastry Bulusu Venkata (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06602.txt.

Introduction

Santhosh Kakarla and Gautama Shastry Bulusu Venkata studied composed image search in a 2025 paper. A user starts with a reference photo and a text edit, such as turning a dress blue. Searching with no task training gave weak results on a fashion benchmark.

Fine-tuning a large vision and language model with a small fusion part raised recall sharply across shirts, dresses, and tops. A second route, which tuned only the text side with preference learning, failed almost fully. The authors trace that failure to missing fusion, a poor match of goal to ranking, and weak negative examples. Real fusion, they argue, is the key step.

Catalog record

CREATORSanthosh Kakarla, Gautama Shastry Bulusu Venkata
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06602

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zero Shot Composed Image Retrieval Santhosh Kakarla, Gautama Shastry Bulusu Venkata Abstract Composed image retrieval (CIR) enables users to perform fine - grained visual edits—such as “turn the dress blue” or “remove stripes”—by combining a reference image with a textual instruction. Zero - shot CIR methods using pretrained multimodal encoders typically encode text and image separately, yielding ∼ \sim 20–25% Recall@10 on benchmarks like FashionIQ. To enhance performance, we apply a BLIP2 - based fine - tuning strategy with a Q - Former to fuse image and text into a unified representation, boosting Recall@10 to 45.6% (shirt), 40.1% (dress), and 50.4% (toptee) and average Recall@50 to 67.6%. Building on this, we implement Retrieval - DPO with CLIP’s separate text and vision encoders: each text prompt is encoded via the CLIP transformer, positives and FAISS - mined hard negatives are encoded via CLIP’s image encoder, and a Direct Preference Optimization loss enforces a margin between their similarity scores. Despite extensive tuning of the DPO scaling factor, index parameters, and sampling strategies, this approach achieves only 0.02% Recall@10—far below zero - shot and prompt - tuned baselines—due to (1) lack of joint image–text fusion, (2) misalignment between margin - based loss and top - K K metrics, (3) poor negative quality, and (4) frozen visual and transformer layers. These findings underscore the necessity of true multimodal fusion, ranking - aligned objectives, and refined negative sampling for effective preference - based CIR. 1 Introduction Composed image retrieval (CIR) enables users to issue semantically rich visual search queries by specifying a reference image alongside a textual modification—examples include “turn the dress blue,” “remove the floral pattern,” or “add stripes to the shirt.” Zero - shot CIR (ZSCIR) methods process these queries by feeding images and text independently into pretrained multimodal encoders (e.g., CLIP) and computing cosine similarity between the two modalities. This pipeline requires no task - specific training data, making it highly flexible; however, it struggles to interpret and combine visual and linguistic nuances. On the FashionIQ benchmark, which contains real - world product images and human - annotated edit instructions, ZSCIR delivers only 20–25% Recall@10. Such limited performance hinders CIR’s utility in critical applications—like e - commerce personalization, where customers expect precise product variations; digital design prototyping, which demands fine - grained control over visual edits; and content curation or moderation, where semantics-driven filtering is essential. To overcome these limitations, we adopt a fine - tuning strategy within the BLIP2 framework, introducing a lightweight Q - Former module that learns to fuse visual and textual inputs into a unified representation. During fine - tuning, the Q - Former consumes both the embedding of the reference image and the token embeddings of the edit instruction, producing a single, dynamic prompt vector. This fused prompt is then compared against candidate images via a contrastive retrieval loss, aligning it more closely with the target image and distancing it from negatives. Without requiring paired examples of edited images, this approach adapts the generic multimodal encoder to the CIR task, improving its sensitivity to subtle attribute changes and object manipulations. Empirically, fine - tuned BLIP2 achieves Recall@10 of 45.6% for shirt edits, 40.1% for dress edits, and 50.4% for toptee edits, with an average Recall@50 of 67.6% on FashionIQ. Building on these gains, we investigate Retrieval - DPO for CIR, a retrieval - augmented preference optimization framework that seeks to inject user - style ranking signals directly into the retrieval model. Unlike prompt - tuning, this method operates on separate CLIP text and vision encoders: each caption is encoded via the CLIP transformer into a prompt embedding, while positive and FAISS - mined hard negative images are encoded by the CLIP vision encoder. We retrieve the hardest negatives for each query using a FAISS index over the gallery and apply a Direct Preference Optimization loss—maximizing the log - odds margin between positive and negative similarity scores. Despite exhaustive hyperparameter tuning (including the DPO scaling factor \beta , index configurations, and negative - sampling strategies) and mixed - precision training, Retrieval - DPO yields only 0.02% Recall@10. We attribute this stark underperformance to four factors: (1) the absence of true multimodal fusion limits the model’s capacity to learn cross - modal relations; (2) DPO’s margin - based objective does not directly optimize top - K ranking metrics; (3) FAISS - mined negatives are either too trivial or too ambiguous, weakening the preference signal; and (4) freezing both visual and transformer layers restricts representational adaptation. These findings illuminate critical gaps in naive preference - based approaches and underscore the need for integrated fusion architectures, ranking - aligned loss functions, and refined negative - sampling strategies in future multimodal preference learning research. 2 Related Work Contrastive Language–Image Pretraining (CLIP) by Radford et al. (2021) introduced a dual-encoder framework that learns aligned image and text representations via a contrastive loss, enabling effective zero-shot transfer across a variety of vision-language tasks, including composed image retrieval (CIR). Building on CLIP’s paradigm, BLIP2 by Li et al. (2023) employs a lightweight Q-Former to learn query embeddings that fuse frozen vision encoders with large language models, significantly improving multimodal fusion for retrieval and other downstream tasks. Direct Preference Optimization (DPO) by Rafailov et al. (2023) reformulates reinforcement learning from human feedback into a supervised, margin-based loss that aligns model outputs with human preferences. Retrieval-DPO extends this idea to retrieval settings by mining hard negatives using FAISS and applying a joint-preference loss to the similarity scores between query and image embeddings. Collectively, these methods motivate our investigation into prompt-tuned CIR and preference-based retrieval optimization. 3 Methodology Composed Image Retrieval (CIR) combines a reference image with a natural-language edit instruction to retrieve a “target” image reflecting the specified modification. Traditional CIR approaches required fully supervised training on triplets of (reference, edit, target), which is labor-intensive to collect. Zero-Shot CIR (ZSCIR) circumvents this by leveraging models like CLIP, which independently embed images and text into a shared vector space via a contrastive pretraining objective (InfoNCE, Rusak et al. 2024). Retrieval is then performed by computing cosine similarity between the text-encoded instruction and image embeddings. While ZSCIR requires no task-specific data, it struggles with fine-grained edits—achieving only 20–25% Recall@10 on FashionIQ—because it cannot explicitly model the interaction between image content and textual modifications. 3.1 Dataset Preparation The FashionIQ benchmark (Wu et al. 2019) serves as our evaluation corpus. It comprises three apparel categories— shirt ( 17 K pairs), dress ( 16 K pairs), and toptee ( 18 K pairs)—each drawn from real-world e-commerce product listings. Each reference–target pair is annotated with one to five human-written captions describing the desired visual modification (e.g., “make the dress red”, “add stripes to the shirt”, “remove floral patterns”), resulting in over 50 000 unique edit instructions across the dataset. FashionIQ provides official train, validation, and test splits via JSON files ( split.<category>.<split>.json ), with approximately 80% of pairs for training, 10% for validation, and 10% for testing. These splits ensure no overlap of reference or target images. We use the training split for model fine-tuning, the validation split for hyperparameter selection and early stopping (monitoring Recall@10), and the test split only for final performance reporting. All images are preprocessed uniformly: loaded in RGB,

Read the complete text