Do Composed Image Retrieval Benchmarks Require Multimodal Composition?

Matteo Attimonelli Alessandro De Bellis Aryo Pradipta Gema Rohit Saxena, Sapienza University of Rome, University of Edinburgh, Miniml.AI · 2026

Read Do Composed Image Retrieval Benchmarks Require Multimodal Composition? by Matteo Attimonelli Alessandro De Bellis Aryo Pradipta Gema Rohit Saxena, Sapienza University of Rome, University of Edinburgh, Miniml.AI (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14787.txt.

Introduction

Attimonelli and team show that composed image retrieval benchmarks may not require multimodal composition after all. A query pairs a reference image with a textual modification, and strong scores are assumed to need both inputs combined. Across four benchmarks plus eleven generalist multimodal embedding models, a big share of queries, from about thirty-two to eighty-four percent, can be solved from one modality alone. Human validation finds only about one thousand six hundred and eighty-nine of four thousand seven hundred and forty-one shortcut-free queries well-formed.

Catalog record

CREATORMatteo Attimonelli Alessandro De Bellis Aryo Pradipta Gema Rohit Saxena, Sapienza University of Rome, University of Edinburgh, Miniml.AI
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE104083
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14787

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Composed Image Retrieval Benchmarks Require Multimodal Composition? Matteo Attimonelli Alessandro De Bellis Aryo Pradipta Gema Rohit Saxena, Sapienza University of Rome, University of Edinburgh, Miniml.AI Abstract Composed Image Retrieval (CIR) is a multimodal retrieval task where a query consists of a reference image and a textual modification, and the goal is to retrieve a target image satisfying both. In principle, strong performance on CIR benchmarks is assumed to require multimodal composition , i.e., combining complementary information from reference image and textual modification. In this work, we show that this assumption does not always hold. Across four widely used CIR benchmarks and eleven Generalist Multimodal Embedding models, a large fraction of queries can be solved using a single modality (from 32.2% to 83.6%), revealing pervasive unimodal shortcuts. Thus, high CIR performance can arise from unimodal signals rather than true multimodal composition. To better understand this issue, we perform a two-stage audit. First, we identify shortcut-solvable queries through cross-model analysis. Second, we conduct human validation on 4,741 shortcut-free queries, of which only 1,689 are well-formed, with common issues including ambiguous edits and mismatched targets. Re-evaluating models on this validated subset reveals qualitatively different behaviour: queries can no longer be solved with a single modality, and successful retrieval requires combining both inputs. While accuracy decreases, reliance on multimodal information increases. Overall, current CIR benchmarks conflate shortcut-solvable, noisy, and genuinely compositional queries, leading to an overestimation of model capability in multimodal composition. Website: https://matteoattimonelli.github.io/CIRCUS/ . 1 Introduction Composed Image Retrieval (CIR) is a multimodal retrieval task in which a query consists of a reference image and a textual modification, and the goal is to retrieve a target image satisfying both. This formulation is intended to test multimodal composition , i.e., the ability to combine complementary information from the reference image and the textual modification into a representation that cannot be recovered from either modality alone. Benchmarks such as CIRR [ 1 ] , FashionIQ [ 2 ] , LaSCo [ 3 ] , and CIRCO [ 4 ] are commonly used to evaluate this ability. A key challenge in multimodal learning is ensuring that models genuinely integrate information across modalities, rather than relying on a single dominant signal. Prior work has shown that models often exploit shortcut signals: in visual question answering, many questions can be answered from text alone [ 5 ] , and similar effects have been observed in multimodal and language benchmarks [ 6 ] —showcasing weak visual understanding [ 7 ] . This raises the question of whether similar behaviour arises in CIR. As a compositional multimodal retrieval task, CIR is expected to require both modalities. However, if a query can be solved using only text or image information, retrieval does not reflect true multimodal composition, but instead indicates reliance on unimodal shortcuts [ 3 ] . In such cases, benchmark performance may overestimate a model’s ability to combine modalities. In this work, we systematically evaluate whether CIR benchmarks enforce multimodal composition. For each query, we compare retrieval under three conditions: (i) the full multimodal input, (ii) text-only, where the reference image is suppressed, and (iii) image-only, where the textual edit is removed. We perform this analysis using a heterogeneous pool of eleven Generalist Multimodal Embedding models [ 8 , 9 , 10 , 11 , 12 , GMEs;] , which encode multimodal queries into a shared representation space. As these models are trained on large-scale multimodal corpora and support multiple retrieval settings, they provide a suitable probe for identifying dataset-level shortcuts. A query is said to admit a unimodal shortcut if any model can retrieve the target using a single modality. Figure 1 illustrates this phenomenon—in text-only shortcuts, the textual modification alone is sufficient to identify the target; in image-only shortcuts, the reference image alone suffices, often due to uninformative edits. This highlights that strong benchmark performance does not necessarily imply effective multimodal composition. Figure 1: Examples of text-only shortcuts (top), image-only shortcuts (middle), and valid queries (bottom) from three CIR benchmarks. These cases show that high retrieval accuracy does not necessarily imply multimodal composition. Our analysis reveals that unimodal shortcuts are pervasive across standard CIR benchmarks: 83.6% of CIRR, 32.2% of FashionIQ, 38.7% of LaSCo, and 74.5% of CIRCO queries are solvable using a single modality. This shows that a large portion of benchmark performance does not require multimodal composition and can therefore be explained by unimodal shortcuts, undermining the intended role of CIR as a test of compositional reasoning. Removing shortcut-solvable queries is necessary but not sufficient—among the remaining shortcut-free queries, many are ambiguous, underspecified, or incorrectly annotated. To quantify this effect, we conduct a human validation study on 4,741 such queries ( Section 2.3 ) and find that only 1,689 (35.6%) form well-defined CIR instances (e.g., Figure 1 , bottom). The dominant issue is overly broad queries, where multiple images satisfy the modification, making the target non-unique. Therefore, we present CIRCUS (Composed Image Retrieval Cleaned of Unimodal Shortcuts), a cleaned evaluation suite of CIR benchmarks, which provides a more reliable test of multimodal composition. Re-evaluating models on CIRCUS reveals a different behaviour: queries can no longer be solved using a single modality, and successful retrieval requires combining both inputs. Performance drops sharply—for instance, Qwen3-VL-8B’s Recall on CIRR falls from 70.1 to 24.1, indicating that much of the apparent benchmark performance came from unimodal shortcuts rather than genuine multimodal composition ( Section 4.1 ). CIRCUS therefore provides a more faithful assessment of how well models compose image and text. To summarise, our contributions are the following: 1. We quantify the prevalence of unimodal shortcuts in CIR benchmarks ( Section 4.1 ), showing that a large fraction of queries can be solved using only text or only image information. 2. We show, through manual validation ( Section 4.2 ), that the four audited CIR benchmarks suffer from additional issues beyond shortcuts, including overly broad modifications and invalid or non-unique ground-truth targets, and construct CIRCUS , retaining the well-formed CIR queries. 3. We demonstrate that CIRCUS provides a more diagnostic test of multimodal composition ( Section 4.3 ): queries can only be solved by combining both modalities, with balanced reliance on each, whereas original benchmarks overestimate performance due to unimodal shortcuts. Overall, our results show that current CIR benchmarks conflate shortcut-solvable, noisy, and genuinely compositional queries, leading to an overestimation of model capabilities. Consequently, reliance on these benchmarks may overestimate models’ multimodal capabilities, as performance can be driven by unimodal shortcuts rather than multimodal composition. 2 Auditing CIR Benchmarks 2.1 Preliminaries In CIR, a query Q_{i} consists of a reference image R_{i} and a textual modification T_{i} describing a desired change. The goal is to retrieve, from a gallery \mathcal{G} , a target image that best matches the intent of T_{i} applied to R_{i} . A model p produces a ranking over the gallery for each query using a similarity function. We denote by r_{p}(i) the rank of the ground-truth target for query Q_{i} , and say that the query is solved at cutoff K if r_{p}(i)\leq K . 2.2 Unimodal Shortcut Audit We ask whether CIR queries genuinely require combining both modalities, or whether a single modality is sufficient to retrieve the target, signifying the presence of

Read the complete text