ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation
Shizhe Zhou, Bohan Jia, Kai Wu, Yan Shen, Tongyun Li, Yuyang Wu, Shaohui Lin · 2026
Read ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation by Shizhe Zhou, Bohan Jia, Kai Wu, Yan Shen, Tongyun Li, Yuyang Wu, Shaohui Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29579.txt.
Introduction
Shizhe Zhou and six colleagues present ReactBench, a benchmark that tests why multimodal models hallucinate, not just whether they do. Most tests score wrong answers, while ReactBench builds adversarial images and trick queries in an exam-style format. Four tasks target distinct failure roots: erased relations, counterfactual traits, traced changes, and dense counts. Chain-of-thought reasoning then traces sub-causes within each task, exposing co-occurrence bias, language habits, weak cross-image comparison, and fine perception limits. Tests show current models stay open to cause-specific triggers.
Catalog record
| CREATOR | Shizhe Zhou, Bohan Jia, Kai Wu, Yan Shen, Tongyun Li, Yuyang Wu, Shaohui Lin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60803 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29579
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation Shizhe Zhou, Bohan Jia, Kai Wu, Yan Shen, Tongyun Li, Yuyang Wu, Shaohui Lin Abstract While multimodal large language models (MLLMs) have achieved rapid progress in visual-language understanding, they remain prone to multimodal hallucinations, producing responses that are inconsistent with the visual input. Existing benchmarks predominantly focus on detecting hallucination outcomes rather than evaluating the underlying causes of these failures. Moreover, many benchmarks rely on simplistic scenarios and limited evaluation formats that no longer challenge state-of-the-art models. To address these limitations, we introduce ReactBench, a cause-driven hallucination benchmark featuring multiple tasks and an exam-style evaluation format. By generating adversarial images and hallucination-inducing queries, ReactBench introduces four targeted tasks (Relational Erasure, Counterfactual Attribute, Alteration Tracing and Dense Counting) to systematically expose co-occurrence bias, language priors, cross-image comparative perception deficiencies, and fine-grained perceptual bottlenecks. Beyond standard accuracy-based evaluation, we leverage Chain-of-Thought reasoning to identify fine-grained sub-causes of hallucination within each task. Extensive evaluations reveal that current MLLMs remain notably vulnerable to cause-specific hallucination triggers, demonstrating the value of ReactBench as a systematic and interpretable testbed for diagnosing and improving multimodal model robustness. Our project page is available at https://reactbench.github.io/. 1 Introduction While Multimodal Large Language Models (MLLMs) have demonstrated unprecedented capabilities in visual-language understanding and reasoning Hendrycks et al. 2020 ; Dai et al. 2023 , multimodal hallucination Liu et al. 2024 —where models generate semantically coherent responses that are inconsistent with the input visual information—has emerged as a critical bottleneck for their deployment in real-world applications. Early hallucination benchmarks Li et al. 2023 ; Hu et al. 2023 ; Kaul et al. 2024 have played a foundational role in diagnosing and mitigating these issues. However, as MLLMs continue to advance rapidly, the limitations of current benchmarks are increasingly apparent. Specifically, existing hallucination benchmarks exhibit three major limitations: First, many early benchmarks Rohrbach et al. 2018 ; Wang et al. 2023b lack both adversarial or challenging test images and difficulty gradients for evaluation questions. Since they predominantly rely on simplistic scenes and limited query formats, these benchmarks are increasingly saturated by state-of-the-art models, yielding inflated performance metrics that fail to probe the true capability boundaries of MLLMs. Second, while some recent evaluations Ye-Bin et al. 2024 ; He et al. 2025 ; Huang et al. 2024b have begun to explore underlying hallucination causes—such as language priors or co-occurrence bias—they tend to investigate these factors in isolation, without offering a unified, cause-driven framework that covers diverse hallucination triggers. Moreover, many benchmarks Liu et al. 2025 ; Bitton-Guetta et al. 2023 directly generated counterfactual or adversarial samples by text-to-image models, which may compromise the real-world representativeness of the evaluation data. Finally, although Chain-of-Thought (CoT) reasoning exposes intermediate reasoning traces that may reveal why a model produces a hallucinated answer, existing benchmarks Yin et al. 2026 ; Cui et al. 2023 rarely use such traces for fine-grained causal diagnosis. They typically focus on aggregate performance changes brought by CoT or provide coarse-grained summaries of hallucination behaviors, lacking further analysis that traces hallucinated outputs back to the specific sub-causes manifested in the reasoning process. Figure 1: The examples of ReactBench . The figure showcases four tasks in our ReactBench: Relational Erasure, Counterfactual Attribute, Alteration Tracing and Dense Counting. Only selected questions are shown. To address these gaps, we introduce ReactBench , a systematic, cause-driven multimodal hallucination benchmark designed to shift the evaluation paradigm from outcome-oriented detection toward cause-oriented diagnosis. Examples are shown in Figure 1 . Our benchmark offers the following three major advantages: Systematic Cause-Driven Framework. Derived from four representative causes of multimodal hallucinations, we systematically design four corresponding evaluation tasks. First, targeting co-occurrence bias, we introduce the Relational Erasure task, which removes one entity from a high-frequency object combination and queries the model regarding the existence of the deleted object. Second, to reveal language priors, we design the Counterfactual Attribute task by altering common-sense object features into counterfactual ones and posing questions based on these manipulated attributes. Third, focusing on deficiencies in cross-image comparative perception, we propose the Alteration Tracing task. By applying four types of edits to objects, this task evaluates whether the model can accurately identify and reason about differences between before-and-after image pairs. Finally, to expose fine-grained perceptual bottlenecks, we construct the Dense Counting task, which selects images containing multiple objects of the same class and asks for their exact quantity. By mapping these tasks to specific failure mechanisms, the framework not only stress-tests MLLMs but also serves as an interpretable diagnostic tool for understanding their visual-language vulnerabilities. Semi-Automated Pipeline with Exam-Style Question Design. We develop an automated construction pipeline built on real images, statistical sample selection, MLLM-based instruction generation and task-specific image editing, which eventually produces high-fidelity adversarial visual samples across 3 tasks. Furthermore, through designing task-relevant question templates, MLLMs are prompted to generate large-scale exam-style evaluation questions. Ultimately, this process yields 50K question-answer (QA) pairs, covering multiple formats (multiple-choice, yes/no, and short-answer) and are purposefully designed with hallucination-inducing distractors and internal difficulty gradients. Notably, the generated images and corresponding QA are reviewed by human annotators to ensure the reliability and quality of the benchmark. Fine-Grained CoT Attribution Analysis. Beyond standard quantitative benchmarking, we conduct a fine-grained analysis of CoT reasoning traces. By establishing a sub-cause taxonomy within each primary hallucination category, we trace hallucinated outputs back to specific sub-causes manifested in the intermediate reasoning process. This analysis provides a more interpretable view of the reasoning patterns behind multimodal hallucinations and offers actionable insights for improving model robustness. Table 1: Comparison of ReactBench with existing hallucination benchmarks across data scale, four hallucination causes, and CoT analysis. Benchmark Size Co-occurrence Bias Language Priors Cross-Image Deficiencies Fine-Grained Perception CoT Analysis POPE Li et al. 2023 3K ✓ (Adversarial negative sampling) ✗ ✗ ✗ ✗ BEAF Ye-Bin et al. 2024 2K ✓ (Removed obj.) ✗ ✓ ✗ ✗ AMBER Wang et al. 2023a 1K ✗ ✗ ✗ ✓ (Obj.&Attr.& Rel.) ✗ Hal-Eval Jiang et al. 2024 10K ✗ ✗ ✗ ✗ ✓ VLind-Bench Lee et al. 2025 2.5K ✗ ✓ (Counter-commonsense) ✗ ✗ ✗ PhD Liu et al. 2025 1K ✗ ✓ (Counter-commonsense) ✗ ✓ (Counting) ✗ ROHE He et al. 2025 5.5K ✓ (Removed obj.) ✗ ✗ ✗ ✗ Reefknot Zheng et al. 2025 10K ✗ ✗ ✗ ✓ (Relations) ✓ HalluScope Khayatan et al. 2026 1K ✓ ✓ ✗ ✗ ✗ Causal-HalBench Xu et al. 2026 1K ✓ (Counterfactual inpainting) ✗ ✗ ✗ ✗ MIRAGE Dong et al. 2026 1.3K ✗ ✗ ✗ ✗ ✓ FINER Xiao et al. 2026 5K ✗ ✗ ✗ ✓ (multi queries) ✗ FREAK Yin et al. 2026 1.8K ✗ ✓ (Counter-commonsense) ✗ ✗ ✓ ReactBench (Ours) 4.7K ✓ (Relational Erasure) ✓ (Counterfactual Attribute) ✓ (Alteration Tracing) ✓ (Dense Counting) ✓ (micro-level) 2 Related Works 2.1 Multimodal Hallucinations in MLLMs Multimodal hallucination refers to the cross-modal inconsistency between generated text and the provided visual input Liu et al. 2024 , typically manifesting as object, attribute, or relational hallucinations Bai et al. 2024 . Recent studies trace the origins of these hallucinations to four interconnected levels. At the data level, models suffer statistical co-occurrence biases and noise from pre-training image-text pairs Yu et al. 2023 ; Zhou