Clash : A Benchmark for Cross-Modal Contradiction Detection

Teodora Popordanoska, Jiameng Li, Matthew B. Blaschko · 2025

Read Clash : A Benchmark for Cross-Modal Contradiction Detection by Teodora Popordanoska, Jiameng Li, Matthew B. Blaschko (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19199.txt.

Introduction

Teodora Popordanoska, Jiameng Li, and Matthew B. Blaschko introduce CLASH, a benchmark that tests whether vision-language models notice when an image and its caption disagree. Real inputs often conflict across modes, yet most benchmarks assume consistency. CLASH pairs COCO photos with captions altered at the object or attribute level to create controlled contradictions.

Questions come in multiple-choice and open-ended forms, backed by a large fine-tuning set and a smaller human-checked diagnostic set. Leading models show clear limits, with biases toward one mode and weak spots in certain categories. Fine-tuning on CLASH sharply improves conflict detection, showing the skill can be taught.

Catalog record

CREATORTeodora Popordanoska, Jiameng Li, Matthew B. Blaschko
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19199

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Clash : A Benchmark for Cross-Modal Contradiction Detection Teodora Popordanoska, Jiameng Li, Matthew B. Blaschko Abstract Contradictory multimodal inputs are common in real-world settings, yet existing benchmarks typically assume input consistency and fail to evaluate cross-modal contradiction detection – a fundamental capability for preventing hallucinations and ensuring reliability. We introduce Clash , a novel benchmark for multimodal contradiction detection , featuring COCO images paired with contradictory captions containing controlled object-level or attribute-level contradictions. The samples include targeted questions evaluated in both multiple-choice and open-ended formats. The benchmark provides an extensive fine-tuning set filtered through automated quality checks, alongside a smaller human-verified diagnostic set. Our analysis of state-of-the-art models reveals substantial limitations in recognizing cross-modal conflicts, exposing systematic modality biases and category-specific weaknesses. Furthermore, we empirically demonstrate that targeted fine-tuning on Clash substantially enhances conflict detection capabilities. 1 Introduction Multimodal Large Language Models (MM-LLMs) ( 46 ; 26 ; 47 ; 28 ; 44 ) have achieved remarkable progress in cross-modal understanding, demonstrating impressive capabilities in image captioning, visual question answering and complex multimodal reasoning tasks. However, real-world applications often present contradictory information across modalities: medical systems reporting “no abnormalities” while X-rays show fractures, autonomous vehicles detecting “clear roads” despite camera feeds showing barriers, or financial documents where text and scanned amounts differ. In such critical scenarios, models have to recognize contradictions, reason about information reliability across modalities, and flag inconsistencies for human review. Despite these practical demands, existing evaluation paradigms fundamentally assume input consistency, treat a single modality as authoritative, test abstention when information is missing, or analyze robustness to corrupted data – leaving a critical capability systematically unmeasured: can models spontaneously detect when two valid, authoritative sources provide contradictory information? Figure 1 : Left: Three-stage pipeline generates conflicting image–text pairs from MS COCO with targeted questions. Right: Examples in Clash , showing object and attribute contradictions. Models are evaluated on the ability to detect conflicts in multiple-choice or open-ended format. Existing benchmarks, while valuable for assessing general multimodal capabilities ( 5 ; 31 ; 20 ; 18 ; 23 ; 29 ) , focus on different aspects. Recent extensions address unanswerable questions, where information is missing ( 3 ; 32 ; 41 ) , or hallucination detection, where models generate false content ( 39 ; 14 ; 8 ) . Other work examines modality selection under explicit instructions about which source to trust ( 17 ) , conflict localization when models are explicitly told inconsistencies exist ( 41 ) , or knowledge conflicts ( 19 ; 48 ) . These approaches either assume one modality is correct, explicitly indicate conflicts, or instruct models which source to prioritize. Thus, spontaneous conflict detection in dual ground truth scenarios remains systematically unevaluated. Evaluating conflict detection in dual ground truth scenarios , where both modalities are valid, presents distinct challenges from traditional single-source paradigms. First, models must possess sufficient cross-modal reasoning capabilities to compare and contrast information across modalities rather than processing them independently. Second, robust evaluation requires assessing modality bias patterns – understanding whether models systematically favor visual or textual information. Third, effective evaluation should span diverse semantic categories and response formats – testing models on various conflict types (objects, attributes), while accommodating format flexibility from multiple-choice to free-form outputs. Finally, it remains unclear whether models can learn to improve in conflict detection through targeted training, or whether the observed weaknesses reflect deeper architectural limitations. In response, we introduce Clash , a comprehensive vision-language conflict detection benchmark designed to systematically evaluate MM-LLMs’ ability to identify multimodal contradictions. Clash features carefully constructed image-text pairs, sourcing images from MS COCO ( 25 ) and pairing them with contradictory text descriptions across object-level and attribute-level categories. Each sample contains precisely one controlled contradiction between the visual and textual content, accompanied by questions in both multiple-choice and open-ended formats. Our benchmark provides \sim 15k high-quality training samples and a human-verified diagnostic set, enabling detailed analysis of model performance and assessment of improvement through targeted fine-tuning. Fig. 1 shows an overview of the generation pipeline and the resulting benchmark. Using Clash , we evaluate state-of-the-art models and reveal striking performance disparities: leading closed–source models (GPT-5 ( 33 ) , Gemini 2.5 Pro ( 35 ) ) achieve strong performance ( > 85% accuracy) while most open-ended models struggle with near-zero detection rates. Our analysis not only exposes systematic modality biases and categorical weaknesses, but also demonstrates that fine-tuning can dramatically improve the conflict detection capabilities, e.g. , LLaVa-1.5-7b ( 28 ) improves from 0% to 77%. These findings highlight critical gaps in current multimodal systems while establishing targeted fine-tuning as an effective solution for robust conflict detection. In summary, our key contributions are: 1 We introduce Clash , a diagnostic benchmark for multimodal conflict detection, with \sim 15k training samples and 1289 human-verified test cases across fine-grained object and attribute categories. 2 We conduct extensive evaluation of state-of-the-art MM-LLMs, revealing significant performance gaps between open- and closed-source models, systematic modality biases, and category-specific weaknesses in conflict detection. 3 We demonstrate that targeted fine-tuning can dramatically improve conflict detection of MM-LLMs, with some models achieving over 75% accuracy improvement. 2 Related work Multimodal large language models. MM-LLMs use powerful LLMs to enable joint reasoning across visual and textual information, demonstrating emergent capabilities like detailed image captioning, visual question answering, and open-ended multimodal dialogue. Closed-source models including GPT ( 2 ; 33 ) and Gemini ( 35 ) have established strong baselines for multimodal reasoning across diverse benchmarks. In parallel, the open-source community has developed a diverse ecosystem of multimodal models, adopting different designs for processing visual and textual information. Examples include instruction-tuned systems like InstructBLIP ( 12 ) , LLaVA ( 27 ) , and MiniGPT4 ( 47 ) , as well as scaling-focused efforts such as InternVL ( 11 ) , Qwen-VL ( 6 ; 38 ; 7 ; 36 ) , Phi3-Vision ( 1 ) , and mPLUG-Owl ( 44 ; 45 ) . These models vary in architectural design (encoder-decoder vs. decoder-only), training objectives (instruction tuning vs. alignment with human feedback), and supervision signals (synthetic vs. human-curated). Despite remarkable progress, MM-LLMs face reliability challenges when processing multimodal inputs: hallucinations , where models describe non-existing objects or attributes ( 21 ; 10 ) ; modality bias , where models systematically favor one modality over another ( 49 ; 4 ; 37 ; 43 ) ; and cross-modal fusion failures where models struggle to properly combine information from multiple modalities ( 34 ) . These issues motivate diagnostic tools to assess model behavior when multimodal inputs are inconsistent. Unanswerable questions and abstention. Several benchmarks evaluate whether models recognize when questions cannot be answered from the available information. VISREAS ( 3 ) examines complex visual reasoning with inherently unanswerable questions due to logical impossibility or missing context, while MM-UPD ( 32 ) systematically categorizes unsolvable problems including absent answers, incompatible answer sets, or missing visual evidence. UNK-VQA ( 15 ) specifically measures abstention ability when visual evidence is inadequate. These benchmarks address important meta-cognitive skills but fundamentally assess information insufficiency : the ability to recognize when evidence is lacking. In contrast, the questions in Clash are not unanswerable – the information is present in both modalities, but the

Read the complete text