FACap: A Large-scale Fashion Dataset for Fine-grained Composed Image Retrieval

François Gardères, Shizhe Chen, Camille-Sovanneary Gauthier, Jean Ponce · 2025

Read FACap: A Large-scale Fashion Dataset for Fine-grained Composed Image Retrieval by François Gardères, Shizhe Chen, Camille-Sovanneary Gauthier, Jean Ponce (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07135.txt.

Introduction

François Gardères and co-authors build FACap in this 2025 paper, a large fashion data set for search by photo plus a text edit. The task, composed image retrieval, asks for pictures matching a reference image changed as the text describes. General models handle color and texture but stumble on fashion's fine vocabulary. Hand labeling by fashion experts is costly, so a big labeled set was missing.

FACap gathers web fashion photos and writes edit texts with a two-stage pipeline using vision and language models. The team also tunes a retrieval model, FashionBLIP-2, with light adapters and multi-way matching. Training on FACap lifted scores on fashion benchmarks, above all for fine-grained edits. The authors release code and pitch the set for online shopping search.

Catalog record

CREATORFrançois Gardères, Shizhe Chen, Camille-Sovanneary Gauthier, Jean Ponce
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-07135

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FACap: A Large-scale Fashion Dataset for Fine-grained Composed Image Retrieval François Gardères, Shizhe Chen, Camille-Sovanneary Gauthier, Jean Ponce Abstract The composed image retrieval (CIR) task is to retrieve target images given a reference image and a modification text. Recent methods for CIR leverage large pretrained vision-language models (VLMs) and achieve good performance on general-domain concepts like color and texture. However, they still struggle with application domains like fashion, because the rich and diverse vocabulary used in fashion requires specific fine-grained vision and language understanding. An additional difficulty is the lack of large-scale fashion datasets with detailed and relevant annotations, due to the expensive cost of manual annotation by specialists. To address these challenges, we introduce FACap , a large-scale, automatically constructed fashion-domain CIR dataset. It leverages web-sourced fashion images and a two-stage annotation pipeline powered by a VLM and a large language model (LLM) to generate accurate and detailed modification texts. Then, we propose a new CIR model FashionBLIP-2 , which fine-tunes the general-domain BLIP-2 model on FACap with lightweight adapters and multi-head query-candidate matching to better account for fine-grained fashion-specific information. FashionBLIP-2 is evaluated with and without additional fine-tuning on the Fashion IQ benchmark and the enhanced evaluation dataset enhFashionIQ, leveraging our pipeline to obtain higher-quality annotations. Experimental results show that the combination of FashionBLIP-2 and pretraining with FACap significantly improves the model’s performance in fashion CIR especially for retrieval with fine-grained modification texts, demonstrating the value of our dataset and approach in a highly demanding environment such as e-commerce websites. Code is available at https://fgxaos.github.io/facap-paper-website/ . [[email protected], ] [] [] [] keywords Composed Image Retrieval ,Multimodal Fusion ,Fashion Domain † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: GENNEXT@SIGIR’25: The 1st Workshop on Next Generation of IR and Recommender Systems with Language Agents, Generative Models, and Conversational AI, Jul 17, 2025, Padova, Italy † † address: Louis Vuitton † † address: Inria, École normale supérieure, CNRS, PSL Research University † † address: Courant Institute of Mathematical Sciences and Center for Data Science, New York University 1 Introduction (a) Examples from the FashionIQ dataset Wu et al. 2021 . Left: incorrect annotation (the target dress is not pink). Right: vague annotation lacking sufficient details to accurately retrieve the target image, like color or shape. (b) Example from our FACap dataset. Figure 1 : Our automatically constructed FACap dataset offers more detailed and accurate annotations than existing datasets for the fashion CIR task. Efficiently retrieving fashion images based on user preferences is crucial for enhancing e-commerce experience, from online shopping to inspiration and brand discovery. The preferences relate both to search interaction preferences —querying with images for instance —and taste and vocabulary preference to really adapt to user needs. Traditional image-to-image Shao et al. 2023 or text-to-image Rao et al. 2022 retrieval methods primarily support single-modality queries and fall short in handling more complex, real-world scenarios. For instance, a user may want to find a product similar to the one they have seen before, but with specific changes, like a different color, style, or feature. To address this, recent works have increasingly focused on composed image retrieval (CIR) Wu et al. 2021 ; Vo et al. 2019 , which aims to retrieve relevant fashion images by leveraging a reference image along with a modification text that describes specific alterations. Most existing methods for Fashion CIR Baldrati et al. 2022 ; Liu et al. 2024 ; Feng et al. 2024 ; Liu et al. 2024 ; Zhao et al. 2022 ; Zhao et al. 2024 fine-tune pretrained vision-language models (VLMs) like CLIP Radford et al. 2021 or BLIP-2 Li et al. 2023 to map images and texts into a shared multimodal space. The embeddings of the reference image and modification text are then fused and compared with the embeddings of candidate images to identify the most relevant match. However, these approaches are constrained by the limitations of current Fashion CIR datasets. For example, Fashion IQ Wu et al. 2021 , a widely used dataset for this task, is limited in scale, containing only 18k ¡reference image, modification text, target image¿ triplets across just three fashion categories: dresses, shirts, and tops. A larger scale dataset would be able to better represent the concept diversity of fashion-domain knowledge. Furthermore, the crowdsourced captions in Fashion IQ are short, noisy and lack details, as shown in Figure 1(a) and confirmed by our quality evaluation in Table 2 . A better CIR experience is expected from three features of the modification text: faithfulness, levels of detail and discriminative power. Reaching a high level of quality is time-consuming and expensive, as it requires to manually annotate a large number of CIR triplets. The noisy and limited data available today hinder the existing models’ ability to understand fine-grained fashion-related features crucial for fashion search tasks, such as specific collar types or textures. To tackle the data scarcity challenge, some approaches have attempted to increase the data size, for example by generating reverse descriptions Liu et al. 2024 for reference and target images; but generating difference descriptions for two images Park et al. 2019 is itself a challenging task. Other approaches attempt to eliminate the need for training data by performing zero-shot CIR Baldrati et al. 2023 ; Saito et al. 2023 ; Tang et al. 2024 ; Yang et al. 2024 ; Lin et al. 2024 ; Karthik et al. 2023 with the help of pretrained VLMs, but their performance suffers from the absence of domain-specific representation learning. More recent efforts pretrain VLMs using web-crawled fashion images to learn more accurate multimodal representations for fashion Zhao et al. 2022 ; Zhao et al. 2024 , but raw web data are often noisy and lack the necessary comparisons between pairs of images for effective CIR training. In this work, we introduce Fashion Automatic Caption (FACap), a large-scale fashion-domain CIR dataset with fine-grained annotations. FACap automatically pairs web-sourced fashion images and employs a two-stage annotation pipeline to generate modification texts, hence creating CIR triplets. The first stage refines original noisy web captions using a VLM to produce long, faithful, and detailed descriptions for each image. The second stage utilizes a large language model (LLM) to analyze the differences between reference and target image captions, generating concise and accurate modification texts. With over 227k CIR triplets, FACap offers a high-quality dataset addressing the challenges of scale, accuracy, and detail in fashion CIR, as evidenced in our quality evaluation. We also introduce the FashionBLIP-2 model for Fashion CIR task using BLIP-2 Li et al. 2023 as backbone, and lightweight adapter modules to specialize it for fashion retrieval needs. Additionally, instead of relying on global features to match the multimodal query and candidate image, we design a multi-head query-candidate matching method that uses multiple feature representations to capture more fine-grained details. We evaluate the performance of our model in two settings: with and without fine-tuning on downstream fashion datasets. Experimental results demonstrate that pretraining on FACap significantly improves model performance for Fashion CIR and showcase the effectiveness of our FashionBLIP-2 model. To summarize, our contribution is three-fold: • We propose an automatic data

Read the complete text