TMCIR: Token Merge Benefits Composed Image Retrieval

Chaoyang Wang 1 Zeyu Zhang 2∗ Long Teng 3 Zijun Li 4 Shichao Kan 1† 1 Central South University 2 The Australian National University 3 Wuhan University 4 Tsinghua University ∗ Project lead. † Corresponding author: [email protected]. · 2025

Read TMCIR: Token Merge Benefits Composed Image Retrieval by Chaoyang Wang 1 Zeyu Zhang 2∗ Long Teng 3 Zijun Li 4 Shichao Kan 1† 1 Central South University 2 The Australian National University 3 Wuhan University 4 Tsinghua University ∗ Project lead. † Corresponding author: [email protected]. (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10995.txt.

Introduction

Chaoyang Wang and co-authors present TMCIR, a method for searching images with a picture plus a written change, in a 2025 paper. A user might show a dress and ask for the same dress in red. The hard part is blending what the picture shows with what the words ask to change, without letting one side drown out the other.

Earlier systems lean too far toward the image or too far toward the text. TMCIR first tunes its image and text readers using fake target images made by a diffusion model, so the text side learns fine shades of intent. It then merges word and image tokens in a way that adjusts the balance for each query. On two standard test sets, the method beat prior systems, above all on searches with subtle requested changes.

Catalog record

CREATORChaoyang Wang 1 Zeyu Zhang 2∗ Long Teng 3 Zijun Li 4 Shichao Kan 1† 1 Central South University 2 The Australian National University 3 Wuhan University 4 Tsinghua University ∗ Project lead. † Corresponding author: [email protected].
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-10995

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TMCIR: Token Merge Benefits Composed Image Retrieval Chaoyang Wang 1 Zeyu Zhang 2∗ Long Teng 3 Zijun Li 4 Shichao Kan 1† 1 Central South University 2 The Australian National University 3 Wuhan University 4 Tsinghua University ∗ Project lead. † Corresponding author: [email protected]. Abstract . Composed Image Retrieval (CIR) retrieves target images using a multi-modal query that combines a reference image with text describing desired modifications. The primary challenge is effectively fusing this visual and textual information. Current cross-modal feature fusion approaches for CIR exhibit an inherent bias in intention interpretation. These methods tend to disproportionately emphasize either the reference image features (visual-dominant fusion) or the textual modification intent (text-dominant fusion through image-to-text conversion). Such an imbalanced representation often fails to accurately capture and reflect the actual search intent of the user in the retrieval results. To address this challenge, we propose TMCIR, a novel framework that advances composed image retrieval through two key innovations: 1) Intent-Aware Cross-Modal Alignment. We first fine-tune CLIP encoders contrastively using intent-reflecting pseudo-target images, synthesized from reference images and textual descriptions via a diffusion model. This step enhances the encoder ability of text to capture nuanced intents in textual descriptions. 2) Adaptive Token Fusion. We further fine-tune all encoders contrastively by comparing adaptive token-fusion features with the target image. This mechanism dynamically balances visual and textual representations within the contrastive learning pipeline, optimizing the composed feature for retrieval. Extensive experiments on Fashion-IQ and CIRR datasets demonstrate that TMCIR significantly outperforms state-of-the-art methods, particularly in capturing nuanced user intent. DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 CCS: Composed Image Retrieval 1. Introduction Figure 1. Workflows of existing CIR methods and our proposed TMCIR Retrieving images based on a combination of a reference image and textual modification instructions defines the task of Composed \sim Image Retrieval (CIR) ( Lee, 2005 ; Vo et al., 2019 ; Baldrati et al., 2022b ) . Specifically, the goal of CIR is to retrieve a target image from a candidate set that maintains overall visual similarity to the reference image while fulfilling the localized modification requirements specified in the textual description. CIR enables precise, interactive retrieval, making it valuable for applications like e-commerce and personalized web search. However, composed queries from two distinct modalities introduces unique challenges. Unlike text-to-image or image-to-image retrieval which rely on a single query type, CIR must interpret relative changes described textually and apply them to the specific visual content of the reference image. The core difficulty lies in effectively integrating these cross-modal signals into a unified representation for similarity comparison with candidate images. To achieve this integration, most current approaches predominantly employ visual-dominant feature fusion mechanisms ( Anwaar et al., 2021 ; Chen and Bazzani, 2020 ; Liu et al., 2021 ; Dodds et al., 2020 ) . which extract image and text features separately and then combining them. However, these methods exhibit two critical limitations: 1) They often fail to preserve essential visual details from the reference image; 2) They tend to inadvertently incorporate irrelevant background information ( i.e., regions unrelated to the textual modifications) into the final query representation. These shortcomings become particularly pronounced in scenarios requiring fine-grained image modifications, such as precise color variations or localized texture alterations, where maintaining both visual fidelity and modification accuracy is paramount. Other recent approaches have adopted text-dominant fusion mechanisms that leverage CLIP-based image-to-text conversion ( Gal et al., 2022 ; Baldrati et al., 2022b ) , where reference images are mapped to pseudo-word embeddings for integration with textual descriptions. While this paradigm benefits from established cross-modal alignment, it faces fundamental limitations: 1) The generated pseudo-word tokens primarily capture global image semantics while losing fine-grained visual details; 2) The constrained length of the word tokens restricts comprehensive visual representation. These factors lead to granular-level discrepancies in the cross-modal representations, ultimately compromising retrieval accuracy. As shown in Fig. 1 (a), both the visual-dominant fusion and the text-dominant fusion methods are fail to accurately capture and reflect the actual search intent of the user in the retrieval results. To address these challenges, we propose a TMCIR framework. Our framework is carefully designed to preserve the critical visual information present in the reference image while accurately conveying the modification intent of the user as specified by textual description. The TMCIR comprises two key steps: 1) Intent-Aware Cross-Modal Alignment: This step is designed to precisely capture textual intent from descriptions, addressing a critical limitation in CIR. Conventional target images often contain extraneous variations ( e.g., lighting conditions, irrelevant background objects, or stylistic inconsistencies) that deviate from the specified intent of text, making them suboptimal for fine-tuning. To overcome this, we introduce a pseudo-target generation module that leverages a diffusion model conditioned on both the reference image and the relative textual description. The generated pseudo-target image eliminates noise, serving as a cleaner supervisory signal that faithfully reflects the intended modifications. Using an image-text paired dataset constructed from these pseudo-targets and their corresponding descriptions, we perform contrastive fine-tuning of pre-trained visual and textual encoders. This approach ensures precise cross-modal token alignment in a shared embedding space, with a focused emphasis on text-intent preservation from the description. 2) Adaptive Token Fusion : Following alignment fine-tuning, we introduce an adaptive fusion strategy that computes token-wise cosine similarity between visual and textual encoder outputs, enhanced with positional encoding for weighted feature fusion (Fig. 1 (b). This design serves two key purposes: 1) The positional cues establish explicit correspondences between textual concepts and their spatial counterparts in the image. 2) The similarity-weighted fusion preserves critical visual details while precisely encoding the nuanced modification intents specified in relative descriptions. The fused representations then drive a final contrastive fine-tuning stage, where we optimize all encoders by comparing the adaptive token-fusion features against target images. This dynamic balancing mechanism simultaneously refines both modalities within a unified contrastive framework, ultimately producing composite features that are optimally discriminative for retrieval tasks. Figure 2. Retrieval examples using the proposed TMCIR, CLIP4CIR ( Baldrati et al., 2022a ) (visual-dominant feature fusion), and Pic2word ( Saito et al., 2023 ) (text-dominant fusion) methods, respectively. Experiments conducted on the Fashion-IQ and CIRR datasets demonstrate that our method achieves state-of-the-art performance on both benchmarks. As shown in Figure 2 , a user provides a reference image of a black T-shirt accompanied by the relative description “change the black T-shirt to purple and add intricate patterns”. The method of CLIP4CIR ( Baldrati et al., 2022a ) employs visual-dominant feature fusion, which risks incorporating irrelevant visual elements (e.g., background clutter) that dilute the precise modification intent, leading to erroneous results. Conversely, the method of Pic2word ( Saito et al., 2023 ) uses text-dominant fusion that may suppress relevant visual details while similarly weakening the intended modifications, resulting in comparable failures. Both approaches highlight the need for a balanced fusion strategy that preserves critical visual features while faithfully capturing textual intentions. Our adaptive token fusion approach analyzes the image in finer detail. It partitions the reference image, allowing visual tokens from different regions to be compared with keywords in the description. The mechanism can distinguish the T-shirt region from the background and, via

Read the complete text