A Good CREPE needs more than just Sugar : Investigating Biases in Compositional Vision-Language Benchmarks
Vishaal Udandarao, Mehdi Cherti, Shyamgopal Karthik, Jenia Jitsev, Samuel Albanie, Matthias Bethge · 2025
Read A Good CREPE needs more than just Sugar : Investigating Biases in Compositional Vision-Language Benchmarks by Vishaal Udandarao, Mehdi Cherti, Shyamgopal Karthik, Jenia Jitsev, Samuel Albanie, Matthias Bethge (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08227.txt.
Introduction
Tests that claim to grade how well a vision-and-language model grasps the parts of a scene may grade something else: the test's own slant. Vishaal Udandarao and five co-authors audit seventeen such tests in 2025. Their hunt covers where the data came from and how the wrong options were built.
Blind tricks, such as the length of a caption or its odds under a language model, score on par with the vision models the tests were built to grade. That parity gives the game away: the right and wrong options differ in spread, not just in sense, thanks to how the tests were built. A model can win by reading the slant, not the scene. The team closes with rules for tests that block such cheap wins.
Catalog record
| CREATOR | Vishaal Udandarao, Mehdi Cherti, Shyamgopal Karthik, Jenia Jitsev, Samuel Albanie, Matthias Bethge |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08227
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Good CREPE needs more than just Sugar : Investigating Biases in Compositional Vision-Language Benchmarks Vishaal Udandarao, Mehdi Cherti, Shyamgopal Karthik, Jenia Jitsev, Samuel Albanie, Matthias Bethge Abstract We investigate 17 17 benchmarks ( e.g. , SugarCREPE, VALSE) commonly used for measuring compositional understanding capabilities of vision-language models (VLMs). We scrutinize design choices in their construction, including data source ( e.g. , MS-COCO) and curation procedures ( e.g. , constructing negative images/captions), uncovering several inherent biases across most benchmarks. We find that blind heuristics ( e.g. , token-length, log-likelihood under a language model) perform on par with CLIP models, indicating that these benchmarks do not effectively measure compositional understanding. We demonstrate that the underlying factor is a distribution asymmetry between positive and negative images/captions, induced by the benchmark construction procedures. To mitigate these issues, we provide a few key recommendations for constructing more robust vision-language compositional understanding benchmarks, that would be less prone to such simple attacks. 1 Introduction Compositionality is the notion that the “ the meaning of the whole is a function of the meanings of its parts ” Cresswell 2016 . Several prior works Stone et al. 2017 ; Lake et al. 2017 ; Hudson and Manning 2018 ; Grunde-McLaughlin et al. 2021 posit that visio-linguistic compositional reasoning is essential. Consequently, several benchmarks have been proposed to evaluate the compositional understanding abilities of VLMs Hsieh et al. 2023 ; Ma et al. 2023 ; Yuksekgonul et al. 2022 ; Thrush et al. 2022 ; Zhao et al. 2022 ; Parcalabescu et al. 2021 ; Bexte et al. 2024 ; Kamath et al. 2024 ; Tong et al. 2024 ; Zeng et al. 2024 ; Kamath et al. 2023a ; Awal et al. 2024 ; Beňová et al. 2024 . These benchmarks involve either (1) matching an image to its correct caption from a set of similar captions (image-to-text retrieval), or (2) matching an input text caption to its correct image from a set of similar images (text-to-image retrieval), or (3) performing both these tasks (matching a set of images and texts correctly to each other). While VLMs initially performed almost at chance level Thrush et al. 2022 ; Yuksekgonul et al. 2022 , recent works have been able to significantly improve their performance Mitra et al. 2024 ; Zhang et al. 2023 ; Castro et al. 2024 ; Patel et al. 2024 ; Wang et al. 2024 ; Oh et al. 2024 ; Li and Li 2025 ; Koishigarina et al. 2025 ; Zheng et al. 2024 . Figure 1 : Biases in Compositionality Benchmarks. We identify 4 biases, arising from data sources ( e.g. , COCO bias) and the construction of negative samples ( e.g. , length bias), creating a distributional asymmetry between positives and negatives. This allows simple unimodal heuristics like token-length or LLM likelihood to match CLIP performance, questioning these benchmarks’ validity. Table 1 : Overview of Current Vision-Language Compositionality Benchmarks. We present statistics of 17 datasets, including their primary data sources, the tasks they assess, and the compositional primitives they operate on. The Primitives column denotes the axes of compositionality: attributes ( Att ), relations ( Rel ), objects ( Obj ), and ordering ( Ord ). For Text Types , R indicates real captions from human annotators or web alt-texts, T denotes templated captions from handcrafted templates, and S represents those generated or edited by a large language model (LLM). For Image Types , R refers to naturally collected images, while S indicates those generated by a text-to-image diffusion model. {\dagger} We consider the productivity subset as it is applicable to all VLMs, regardless of pretraining datasets. Benchmark Size Data Source Task Primitives Image Types Text Types License ARO Yuksekgonul et al. 2022 58,685 VG, GQA, COCO, Flickr I \rightarrow T Att,Rel,Ord R R/T MIT CREPE † Ma et al. 2023 42,406 VG I \rightarrow T Att,Rel,Obj R S/T Unknown SugarCREPE Hsieh et al. 2023 7,512 COCO I \rightarrow T Att,Rel,Obj R R/S MIT VL-Checklist Zhao et al. 2022 46,833 VG, SWiG, HAKE, VAW I \rightarrow T Att,Rel,Obj R R/T Unknown VALSE Parcalabescu et al. 2021 8,782 Visual7W, COCO, SWiG, VisDial I \rightarrow T Att,Rel,Obj R R/T/S MIT ColorFoil Samin et al. 2024 3,017 COCO, Flickr I \rightarrow T Att R R/T MIT BLA Chen et al. 2023 1,939 VG I \rightarrow T Att R T MIT What’s-Up Kamath et al. 2023b 4,958 COCO, GQA I \rightarrow T Rel R R/T MIT CC-Neg Singh et al. 2024 228,246 CC-3M I \rightarrow T Att,Rel,Obj R R/S MIT SugarCREPE++ Dumpala et al. 2024 4,759 COCO I \rightarrow T Att,Rel,Obj R R/S CC-BY-4.0 SVO-Probes Hendricks and Nematzadeh 2021 36,841 Google Images T \rightarrow I Rel,Obj R R Apache-2.0 COLA Ray et al. 2023 32,371 VG, GQA, PACO, CLEVR T \rightarrow I Att,Obj R R/T MIT Winoground Thrush et al. 2022 400 Getty Images Both Att,Rel,Obj,Ord R R Custom ColorSwap Burapacheep et al. 2024 300 Text-to-Image Models Both Att S R/T/S MIT COCO-Counterfactuals Le et al. 2023 34,820 Text-to-Image Models Both Obj S R/S CC-BY-4.0 EqBen-Video Wang et al. 2023 243,099 YC2, AG, GEBC Both Att,Rel,Obj R R/T MIT CounterCurate Zhang et al. 2024 11,343 Flickr Both Rel R/S R/S CC Given this increased attention, it is crucial to audit current benchmarks to ensure they accurately measure the desired properties, rather than being solvable through unintended shortcuts Geirhos et al. 2020 . We ask— do current vision-language compositionality benchmarks contain biases that offer “shortcuts” to VLMs? If so, what are these biases, why do they arise, and how can we mitigate them? Although prior work has briefly explored this Hsieh et al. 2023 ; Li et al. 2024 ; Bexte et al. 2024 ; Tschannen et al. 2024a , we comprehensively uncover several concrete issues, identify their root causes grounded in current curation procedures, and provide some simple recommendations to mitigate these issues. To concretely answer these questions, we survey a set of 17 popular benchmarks that are used for evaluating VLMs (see Table 1 ). These benchmarks vary widely in their data-sources, task-types, and negative text/image construction strategies. However, these different design choices can introduce several biases, potentially compromising the benchmark quality. Most of these benchmarks generate “hard-negatives” using handcrafted rules Yuksekgonul et al. 2022 ; Ma et al. 2023 , large language models (LLMs) Hsieh et al. 2023 ; Parcalabescu et al. 2021 , or text-to-image models Zhang et al. 2024 ; Burapacheep et al. 2024 , introducing simple shortcuts for solving the task. For example, in the ARO benchmark Yuksekgonul et al. 2022 , an image, whose positive caption is “The boy is playing in the park”, has a constructed negative caption as “The park is playing in the boy.” (see Fig. 1 bottom left), which is obviously implausible. We systematically uncover 4 such biases across all these benchmarks (see Fig. 1 ). We show that blind models ( e.g. , text-only LLMs) match performance of CLIP on several image-to-text benchmarks, finding that the hard negatives can be easily distinguished from positives due to systemic differences in token-length or caption-likelihoods. Additionally, in text-to-image benchmarks, where hard-negative images are generated using text-to-image models, one can easily distinguish the negative images from real positive images