MagiC : Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

Chengfei WuPurdue University [email protected], Ronald SeohPurdue University [email protected] Bingxuan LiUniversity of Illinois Urbana-Champaign &University of California Los Angeles [email protected] Liqiang ZhangUniversity of Connecticut [email protected] Fengrong HanUniversity of California · 2025

Read MagiC : Evaluating Multimodal Cognition Toward Grounded Visual Reasoning by Chengfei WuPurdue University [email protected], Ronald SeohPurdue University [email protected] Bingxuan LiUniversity of Illinois Urbana-Champaign &University of California Los Angeles [email protected] Liqiang ZhangUniversity of Connecticut [email protected] Fengrong HanUniversity of California (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07297.txt.

Introduction

Chengfei Wu and five co-authors built a bench to test if vision models truly ground their thought in images. Their paper, from twenty twenty-five, deals with large vision-language models. Such models ace visual questions, but they may ride on surface cues and set bias, not on real sight. Their bench, MagiC, grades more than the end reply: it grades the step path and how that path lines up with image proof.

The set holds thousands of machine made questions and a core of hand built ones with marks, reasons, and boxes. Fifteen models were graded on it. Readers who build vision AI get a way to split true grounded thought from lucky pattern play.

Catalog record

CREATORChengfei WuPurdue University [email protected], Ronald SeohPurdue University [email protected] Bingxuan LiUniversity of Illinois Urbana-Champaign &University of California Los Angeles [email protected] Liqiang ZhangUniversity of Connecticut [email protected] Fengrong HanUniversity of California
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-07297

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MagiC : Evaluating Multimodal Cognition Toward Grounded Visual Reasoning Chengfei WuPurdue University [email protected], Ronald SeohPurdue University [email protected] Bingxuan LiUniversity of Illinois Urbana-Champaign &University of California Los Angeles [email protected] Liqiang ZhangUniversity of Connecticut [email protected] Fengrong HanUniversity of California Berkeley [email protected] Dan GoldwasserPurdue University [email protected] Project Repository Abstract Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, it remains unclear whether these models genuinely perform grounded visual reasoning or rely on superficial patterns and dataset biases. In this work, we introduce MagiC , a comprehensive benchmark designed to evaluate grounded multimodal cognition—assessing not only answer accuracy but also the quality of step-by-step reasoning and its alignment with relevant visual evidence. Our benchmark includes approximately 5,500 weakly supervised QA examples generated from strong model outputs and 900 human-curated examples with fine-grained annotations, including answers, rationales, and bounding box groundings. We evaluate 15 vision-language models ranging from 7B to 70B+ parameters across four dimensions: final answer correctness, reasoning validity, grounding fidelity, and self-correction ability. MagiC further includes diagnostic settings to probe model robustness under adversarial visual cues and assess their capacity for introspective error correction. We introduce new metrics such as MagiScore and StepSense , and provide comprehensive analyses that reveal key limitations and opportunities in current approaches to grounded visual reasoning. 1 Introduction Recent advances in large vision-language models (LVLMs) have substantially improved visual question answering and reasoning. Proprietary models such as GPT-4V OpenAI 2023 , Gemini Google 2023 , and Claude-3 Anthropic 2024 , along with open-source counterparts like LLaVA Liu et al. 2023 , Qwen-VL Bai et al. 2023 , and DeepSeek-VL Lu et al. 2024 , exhibit strong capabilities in perception, object recognition, and complex visual understanding. However, these successes raise a deeper question: Do current multimodal models genuinely reason over visual content, or do they rely on superficial patterns and dataset biases? Figure 1 : Example scenarios from MagiC . Existing work mainly focuses on evaluating the final answer for a given task ignoring the steps model takes to answer. Addressing this question requires a closer examination of grounded multimodal cognition—the ability to selectively attend to relevant visual inputs and integrate them into coherent, multi-step logical reasoning. This ability is central to human intelligence and is critical for building interpretable, trustworthy AI systems. For instance, as illustrated in our example in Figure 1 , robust reasoning requires the model to justify its answer by explicitly referencing the correct visual regions and following a logical inference path. Yet, most existing benchmarks primarily focus on end-task performance, offering limited insight into whether models truly "understand" the image or simply exploit dataset biases. They rarely examine how models arrive at answers, whether intermediate reasoning steps align with relevant visual regions ( Li et al. 2023a ; Liu et al. 2024 ; Yue et al. 2024 ; Yu et al. 2024 ) Recently, increasing attention has been given to the reasoning processes of VLMs, with several works examining their chain-of-thought reasoning Zhang et al. 2024a ; Zhao et al. 2025 ; Chen et al. 2023 , intermediate rationale generation Chen et al. 2023 , and visual grounding consistency Yang et al. 2023 ; He et al. 2024 . However, these efforts remain fragmented, often lacking unified evaluation protocols and comprehensive grounding diagnostics. In this work, we introduce a comprehensive benchmark, MagiC (Bench MA rkin G Mult I modal C ognition), designed to evaluate grounded multimodal cognition. As illustrated in the right section of Figure 1 , our framework goes beyond conventional question answering by measuring a model’s ability to (1) generate correct final answers, (2) articulate coherent, interpretable step-by-step reasoning, (3) ground each reasoning step in the appropriate visual evidence, and (4) intervene upon and revise faulty reasoning when possible. To support this evaluation, we construct a dual-sourced dataset consisting of approximately 5,500 weakly-supervised visual reasoning instances and near 900 human-curated instances with over 15,000 annotated reasoning steps. The weakly supervised data is automatically derived from the outputs of high-performing vision-language models across 78 diverse question types, enabling broad coverage with minimal annotation cost. In contrast, the human-curated set includes fine-grained annotations—answers, reasoning rationales, and bounding box groundings—with a held-out test set of 689 instances for rigorous evaluation. We evaluate 15 vision-language models ranging from 7B to 70B+ parameters, including both open-source and proprietary systems. Our evaluation framework captures four dimensions of model performance: short-form and long-form answer correctness, reasoning validity, grounding quality, and self-correction ability. Grounding quality is quantified using MagiScore , which measures the overlap between predicted and reference bounding boxes. To assess models’ introspective capabilities, we further analyze their ability to detect and revise intermediate reasoning errors. In addition to standard performance metrics, MagiC introduces several diagnostic settings that probe robustness and interpretability. The adversarial grounding setting evaluates models under misleading or irrelevant visual cues to test their reliance on correct evidence. The self-correction analysis examines whether a model can recognize and revise its own reasoning errors. Finally, we conduct a human evaluation of reasoning quality on a subset of four models, deriving a StepSense score that captures coherence and factual consistency. In summary, our contributions are as follows: • First, we introduce a novel task that jointly evaluates answer correctness, step-by-step reasoning quality, and visual grounding fidelity. • Second, we construct a benchmark dataset comprising approximately 5,500 weakly supervised interleaved bounding-box QA examples and near 900 high-quality human-curated examples with detailed annotations for answers, rationales, and bounding boxes. • Third, we introduce novel evaluation metrics MagiScore tailored to multimodal cognition, with StepSense and Self-Heal which measures model reasoning quality and self-correction ability. • Lastly, we conduct a comprehensive analysis of 15 state-of-the-art models, uncovering key insights into both the limitations and potential of current approaches to grounded visual reasoning. Our findings indicate that models exhibiting precise region focus are generally more likely to answer questions correctly. We hope this benchmark will catalyze future research toward building more interpretable, robust, and cognitively aligned multimodal systems. 2 Related Work 2.1 Benchmarks for Large Vision-Language Models Ever since the introduction of large vision-language models (LVLMs) such as BLIP ( Li et al. 2023b ) and LLaVA ( Liu et al. 2023 ) , the field has seen rapid advancements in both model scaling and evaluation methodologies. Researchers have increasingly focused on scaling models at test time to push the boundaries of what these systems can achieve. Alongside these developments, there has been a notable surge in the creation of new benchmarks specifically designed to assess the visual reasoning capabilities of LVLMs, moving beyond traditional vision tasks like Visual Question Answering (VQA). Recent benchmarks such as MMBench ( Liu et al. 2024 ) , MMMU ( Yue et al. 2024 ) , SEED-Bench ( Li et al. 2023a ) , and MM-Vet ( Yu et al. 2024 ) now provide more comprehensive and fine-grained evaluations, covering a wider variety of visual reasoning challenges. However, most of these benchmarks still primarily measure end task performances and do not delve deeply enough into the quality of LVLMs’ reasoning process. Despite efforts on visual chain of thought (CoT) ( Chen et al. 2023 ; Rose et al. 2024 ; Zhang et al. 2024a ; Zhang et

Read the complete text