CoLLM: A Large Language Model for Composed Image Retrieval

Chuong Huynh, Jinyu Yang, Ashish Tawari, Mubarak Shah, Son Tran, Raffay Hamid, Trishul Chilimbi, Abhinav Shrivastava · 2025

Read CoLLM: A Large Language Model for Composed Image Retrieval by Chuong Huynh, Jinyu Yang, Ashish Tawari, Mubarak Shah, Son Tran, Raffay Hamid, Trishul Chilimbi, Abhinav Shrivastava (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19910.txt.

Introduction

Chuong Huynh and co-authors present CoLLM, a framework for composed image retrieval, where a user gives a reference image plus text describing a desired change. Training data for this task are costly triplets of image, change text, and target image. That scarcity has pushed others toward synthetic data or loose image-caption pairs, both with limits.

CoLLM builds triplets on the fly from image-caption pairs, so training needs no hand labeling. It uses a large language model to fuse the image and change text into a joint code. The team also releases a very large new data set and cleans two standard benchmarks. Their method set new best results across benchmarks.

Catalog record

CREATORChuong Huynh, Jinyu Yang, Ashish Tawari, Mubarak Shah, Son Tran, Raffay Hamid, Trishul Chilimbi, Abhinav Shrivastava
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-19910

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoLLM: A Large Language Model for Composed Image Retrieval Chuong Huynh, Jinyu Yang, Ashish Tawari, Mubarak Shah, Son Tran, Raffay Hamid, Trishul Chilimbi, Abhinav Shrivastava Abstract Composed Image Retrieval (CIR) is a complex task that aims to retrieve images based on a multimodal query. Typical training data consists of triplets containing a reference image, a textual description of desired modifications, and the target image, which are expensive and time-consuming to acquire. The scarcity of CIR datasets has led to zero-shot approaches utilizing synthetic triplets or leveraging vision-language models (VLMs) with ubiquitous web-crawled image-caption pairs. However, these methods have significant limitations: synthetic triplets suffer from limited scale, lack of diversity, and unnatural modification text, while image-caption pairs hinder joint embedding learning of the multimodal query due to the absence of triplet data. Moreover, existing approaches struggle with complex and nuanced modification texts that demand sophisticated fusion and understanding of vision and language modalities. We present CoLLM, a one-stop framework that effectively addresses these limitations. Our approach generates triplets on-the-fly from image-caption pairs, enabling supervised training without manual annotation. We leverage Large Language Models (LLMs) to generate joint embeddings of reference images and modification texts, facilitating deeper multimodal fusion. Additionally, we introduce Multi-Text CIR (MTCIR), a large-scale dataset comprising 3.4M samples, and refine existing CIR benchmarks (CIRR and Fashion-IQ) to enhance evaluation reliability. Experimental results demonstrate that CoLLM achieves state-of-the-art performance across multiple CIR benchmarks and settings. MTCIR yields competitive results, with up to 15% performance improvement. Our refined benchmarks provide more reliable evaluation metrics for CIR models, contributing to the advancement of this important field. Project page is at collm-cvpr25.github.io . 0 0 footnotetext: This work was done during internship at Amazon. \dagger Project lead. 1 Introduction (a) Composed Image Retrieval Example (b) Recall Sum of state-of-the-art methods on CIR benchmarks Figure 1 : (a) An example of CIR. (b) Recall Sum at {1,10,50} for CIRR and {10, 50} for Fashion-IQ between CoLLM and state-of-the-art (SoTA) models under zero-shot settings. We evaluate two training scenarios: (i) without triplet data and (ii) with synthetic triplet data. Composed Image Retrieval (CIR) enhances traditional image retrieval [ 42 , 62 ] by combining text and image queries, offering greater flexibility in search systems, with applications in e-commerce, fashion, and design [ 9 , 12 , 20 , 63 , 67 ] . As illustrated in Fig. 1(a) , CIR retrieves similar items, such as shirts, where the reference image provides a visual basis, and the modification text specifies desired modifications. By expanding the capabilities of conventional image or text-based searches, CIR allows for more nuanced and specific queries. However, this advanced approach also presents significant challenges compared to traditional image retrieval. Supervised CIR approaches face challenges in data acquisition. They require high-quality CIR triplets (reference image, target image, and modification text), collected through a labor-intensive and expensive process. Consequently, existing CIR triplet datasets are limited in scale and domain coverage, restricting model generalizability. To overcome these limitations, recent CIR methods [ 28 , 14 , 64 , 58 , 23 ] have adopted zero-shot approaches [ 36 ] . These can be broadly categorized into two strategies: (i) leveraging vision-language models [ 45 , 29 , 53 ] (VLMs) for composed query embedding generation, and (ii) generating synthetic triplets for supervised training. VLM-based approaches utilize pre-trained models that already align vision and language features in latent space. These methods rely on large-scale image-caption pairs and follow two main directions, i.e., textual inversion [ 55 , 15 , 8 ] and direct interpolation [ 21 ] . Synthetic triplet generation approaches [ 64 , 28 , 23 ] employ Large Language Models (LLMs) [ 57 , 65 , 7 ] to generate modification text, with CompoDiff [ 14 ] further utilizing diffusion models [ 46 , 5 ] to create synthetic reference-target image pairs. Despite the recent advances in CIR, several critical challenges persist: • [Data] Limitation-1: VLM-based methods may hinder efficient and effective composed query embedding learning by relying solely on image-caption pairs. • [Data] Limitation-2: Existing synthetic triplet datasets often suffer from a lack of diversity and unnatural modification text. Moreover, these datasets are either very small or closed-source, impeding research progress in the field. • [Model] Limitation-3: Current methods for composed query embeddings mainly use shallow transformer models or linear interpolation. While these methods are computationally efficient, they lack the ability to capture the full complexity of the composed understanding tasks. • [Evaluation] Limitation-4: Existing CIR benchmarks are often compromised by noise, particularly in the form of ambiguous samples. Ambiguity arises when multiple target images can correctly match a single query, but only one is labeled as the ground truth, ignoring valid matches. Although CIRCO [ 4 ] attempts to address this issue, its efforts are limited in scale. This ambiguity in benchmarks hinders meaningful model evaluation and comparison. We propose CoLLM, an LLM-based CIR approach to address the aforementioned limitations. CoLLM tackles Limitation-1 by dynamically synthesizing triplets from image-caption pairs, introducing two key components: a reference image embedding synthesis and a modification text synthesis module. The former employs Spherical Linear Interpolation [ 52 ] (Slerp) to generate an intermediate embedding between a given image and its nearest neighbor within the training batch, serving as a synthesized reference image embedding. The latter utilizes a pre-defined text interpolation template to generate modification text based on the current caption and that of the nearest image neighbor. This strategy effectively leverages the vast amount of readily available image-caption pairs on the Internet, enabling a model’s training in a supervised CIR manner. By doing so, CoLLM overcomes the scarcity of labeled triplet data and paves the way for more robust and scalable CIR models. To address Limitation-2 , we introduce Multi-Text CIR (MTCIR), a synthetic dataset of 3.4M image pairs with 17.7M modification texts. MTCIR focuses on two often-overlooked aspects: image diversity and naturalistic modification texts. We curated images from diverse sources to ensure variety. For modification text generation, we employ a two-stage approach using Multi-modal LLM (MLLM) [ 34 , 35 ] for detailed captioning and LLM for describing inter-caption differences. Uniquely, MTCIR provides multiple short modification texts for each image pair, covering various attributes. This better reflects human query formulation, offering a more realistic, comprehensive training foundation for CIR models. To overcome Limitation-3 , CoLLM harnesses the power of LLMs for composed query understanding. It is motivated by the extensive world knowledge embedded in pre-trained LLMs. With their deep semantic understanding, we posit that LLMs are superior to shallow transformers and simple embedding interpolation techniques in comprehending the intricate relationships between reference images and modification texts. By leveraging LLMs, CoLLM aims to capture nuanced semantic connections, enhancing the quality and relevance of composed image retrieval results. To address Limitation-4 , we refine two popular CIR benchmarks: CIRR [ 37 ] and Fashion-IQ [ 61 ] . We use MLLMs to evaluate sample ambiguity in each benchmark and regenerate clear modification text for ambiguous ones. Our pipeline incorporates multiple validation steps to guarantee the enhanced quality of the refined samples. Our main contributions can be summarized as: (i) We propose a method to synthesize CIR triplets on-the-fly from image-caption pairs, which can even outperform models

Read the complete text