FBCIR : Balancing Cross-Modal Focuses in Composed Image Retrieval
Chenchen Zhao, Jianhuan Zhuo, Muxi Chen, Zhaohua Zhang, Wenyu Jiang, Tianwen Jiang, Qiuyong Xiao, Jihong Zhang, Qiang Xu · 2026
Read FBCIR : Balancing Cross-Modal Focuses in Composed Image Retrieval by Chenchen Zhao, Jianhuan Zhuo, Muxi Chen, Zhaohua Zhang, Wenyu Jiang, Tianwen Jiang, Qiuyong Xiao, Jihong Zhang, Qiang Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11520.txt.
Introduction
Zhao and co-authors study why composed image retrieval falters on hard, look-alike negatives. Such search takes an image plus a text edit and must reason over both at once. The team blames focus imbalance: models lean on one input stream and neglect the other. Their tool, FBCIR, finds which visual and textual parts drive each retrieval choice. It shows imbalance is common, most of all under hard negatives. A data workflow adds curated hard negatives to push balanced reasoning, and gains follow across models.
Catalog record
| CREATOR | Chenchen Zhao, Jianhuan Zhuo, Muxi Chen, Zhaohua Zhang, Wenyu Jiang, Tianwen Jiang, Qiuyong Xiao, Jihong Zhang, Qiang Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73628 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11520
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FBCIR : Balancing Cross-Modal Focuses in Composed Image Retrieval Chenchen Zhao, Jianhuan Zhuo, Muxi Chen, Zhaohua Zhang, Wenyu Jiang, Tianwen Jiang, Qiuyong Xiao, Jihong Zhang, Qiang Xu Abstract Composed image retrieval (CIR) requires multi-modal models to jointly reason over visual content and semantic modifications presented in text-image input pairs. While current CIR models achieve strong performance on common benchmark cases, their accuracies often degrades in more challenging scenarios where negative candidates are semantically aligned with the query image or text. In this paper, we attribute this degradation to focus imbalances , where models disproportionately attend to one modality while neglecting the other. To validate this claim, we propose FBCIR , a multi-modal focus interpretation method that identifies the most crucial visual and textual input components to a model’s retrieval decisions. Using FBCIR , we report that focus imbalances are prevalent in existing CIR models, especially under hard negative settings. Building on the analyses, we further propose a CIR data augmentation workflow that facilitates existing CIR datasets with curated hard negatives designed to encourage balanced cross-modal reasoning. Extensive experiments across multiple CIR models demonstrate that the proposed augmentation consistently improves performance in challenging cases, while maintaining their capabilities on standard benchmarks. Together, our interpretation method and data augmentation workflow provide a new perspective on CIR model diagnosis and robustness improvements. 1 Introduction Image retrieval (IR) has long been a fundamental problem in computer vision ( 11 ) and vision-language research ( 17 ) . Traditional IR methods with uni-modal paradigms (i.e., queries are specified either by images or by text) are limited in addressing increasingly complex user requirements that involve multi-modal specifications. Composed image retrieval (CIR) ( 25 ) addresses this limitation by enabling multi-modal queries (images & texts in most cases). Given an image and a related modifying instruction, CIR aims to retrieve images that are semantically consistent with both the visual and the textual content. Early CIR methods ( 3 ; 21 ; 15 ) extend CLIP-like models ( 19 ) with multi-modal input capabilities, while more recent methods ( 1 ) achieve improved performance by adopting vision-language models (VLMs). Figure 1 : Models trained on common-case data learn “shortcuts” to get correct common-case retrieval results, but tend to fail on hard cases that require balanced cross-modal focuses. In this paper, we try to solve this issue by constructing targeted hard negatives. Original positives that are not highly consistent with the query are also considered negative in the proposed framework and replaced by more consistent synthetic images. A central challenge in CIR is the requirement of jointly reasoning over semantics from both modalities. In many common cases, there are large semantic gaps between targets and negative candidates. This setting allows models to rely on “shortcut” strategies, producing correct retrieval results without equally processing information from both modalities, but generally fail in more challenging scenarios where negatives closely resemble targets and are semantically consistent with one of the input modalities. An intuitive example is shown in Figure 1 . In common-case #1, all negative candidates lack the key visual concept (castle) required by the query image, enabling the model to correctly retrieve the target without referring to the text semantics. Similarly, all negatives in common-case #2 lack the key textual concept (winter) specified by the text, enabling correct retrieval without referring to the image semantics. However, in harder cases where both modalities contribute essential but complementary information, such shortcut behavior often leads to retrieval failures. This difficulty is a key factor that makes CIR more challenging than conventional IR, and contributes to the generally lower performance of CIR models ( 13 ) . Additional analyses of such behavior and its impacts on model performance are detailed in Appendix A . In this paper, we term such shortcuts focus imbalances , where a model over-attends to one modality while neglecting the other. To validate the prevalence and severity of this issue, we first introduce a multi-modal focus interpretation method, termed FBCIR ( F ocus- B alancing). Using FBCIR , we reveal that focus imbalances are common across existing CIR models. Building on the analyses, we further propose a CIR data augmentation workflow designed to mitigate focus imbalances. The FBCIR-Data workflow systematically enriches existing CIR triplets with curated hard negatives that require balanced reasoning over both image and text semantics. Using the workflow, we construct a novel benchmark and a finetuning dataset that enable quantitative evaluations and targeted improvements of CIR models from the perspective of focus balancing. Extensive experiments demonstrate that the proposed workflow effectively improves model performance on challenging cases while alleviating focus imbalances. An overview of the FBCIR framework is shown in Figure 2 . Figure 2 : The overall framework of FBCIR , including a multi-modal model focus interpretation method and a dataset augmentation workflow. Given CIR triplets, the focus interpretation method highlights specific image segments and instruction keywords as the model’s focuses, and reveals possible focus imbalances. The dataset augmentation framework facilitates existing CIR triplets with crafted hard negatives that encourage more balanced focuses. The focus interpretation module serves as the problem indicator and post-hoc validator of the data augmentation module. The contributions of this work are three-fold: • We identify and formalize the problem of focus imbalance in CIR, where models exploit shortcut cues from a single modality instead of jointly reasoning over image and text semantics, and empirically demonstrate its prevalence across representative CIR models. • We propose FBCIR , a multi-modal focus interpretation method that provides fine-grained analyses of multi-modal focus behavior, enabling systematic diagnosis and quantitative assessment of focus imbalances in CIR models. • We develop a data augmentation workflow based on FBCIR , yielding a targeted hard-case benchmark and a finetuning dataset with curated hard negatives, which effectively improve models’ hard-case performance and encourage more balanced cross-modal focuses. 2 Related Works 2.1 Composed Image Retrieval (CIR) Models Composed Image Retrieval (CIR) retrieves images by conditioning jointly on a reference image and a modifying textual description. Early methods ( 25 ) focus on explicit feature composition over uni-modal or weakly aligned multimodal representations. With the development of vision-language pretraining, recent CLIP-based ( 19 ; 3 ; 21 ) and VLM-based ( 1 ; 12 ; 16 ) approaches employ more expressive fusion modules or dedicated query encoders, achieving strong performance on standard CIR benchmarks. However, most CIR models are trained and evaluated under relatively easy task settings, where negative candidates are semantically distant from the target. This allows models to over-rely on a single modality rather than integrating multi-modal information. Such “shortcuts” have received limited explicit analysis, motivating in-depth investigation of model focuses under more challenging retrieval scenarios. 2.2 CIR Datasets and Benchmarks Several benchmark datasets drive CIR research. FashionIQ ( 28 ) focuses on fashion images with modification instructions, while CIRR ( 17 ) extends to diverse real-world objects with relative captions. Earlier datasets ( 11 ) also support attribute-based or relative retrieval, but within more constrained domains. Despite their impacts, these benchmarks generally rely on simple negative sampling strategies, making retrieval achievable even when models attend primarily to one modality. Although specific works ( 3 ) introduce harder negatives or refined protocols, systematic evaluation of cross-modal focuses remains limited. In particular,