Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward

Jing Bi 1 Guangyu Sun 2 Ali Vosoughi 1 Chen Chen 2 Chenliang Xu 1 1 University of Rochester 2 University of Central Florida {jing.bi, ali.vosoughi, chenliang.xu}@rochester.edu [email protected], [email protected] · 2025

Read Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward by Jing Bi 1 Guangyu Sun 2 Ali Vosoughi 1 Chen Chen 2 Chenliang Xu 1 1 University of Rochester 2 University of Central Florida {jing.bi, ali.vosoughi, chenliang.xu}@rochester.edu [email protected], [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20696.txt.

Introduction

Jing Bi and four co-authors study how well machines reason from images. Their models mix visual input with text based reasoning. They often use chain of thought, or step by step prompts. Yet the models still invent visual details that are not there. They also lean too much on text instead of images.

The team tests top models in three stages to find failure modes. Then they propose a new agent based design. It pairs a language model with small visual tools. Those tools check fine details and refine each reasoning step. Their system gained clear ground on the MMMU and MathVista tests. It matched or beat much larger models on those tests. The team plans to release its framework and test suite. That should help others build models that look more carefully.

Catalog record

CREATORJing Bi 1 Guangyu Sun 2 Ali Vosoughi 1 Chen Chen 2 Chenliang Xu 1 1 University of Rochester 2 University of Central Florida {jing.bi, ali.vosoughi, chenliang.xu}@rochester.edu [email protected], [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20696

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward Jing Bi 1 Guangyu Sun 2 Ali Vosoughi 1 Chen Chen 2 Chenliang Xu 1 1 University of Rochester 2 University of Central Florida {jing.bi, ali.vosoughi, chenliang.xu}@rochester.edu [email protected], [email protected] Abstract Multimodal large language models (MLLMs) that integrate visual and textual reasoning leverage chain-of-thought (CoT) prompting to tackle complex visual tasks, yet continue to exhibit visual hallucinations and an over-reliance on textual priors. We present a systematic diagnosis of state-of-the-art vision-language models using a three-stage evaluation framework, uncovering key failure modes. To address these, we propose an agent-based architecture that combines LLM reasoning with lightweight visual modules, enabling fine-grained analysis and iterative refinement of reasoning chains. Our results highlight future visual reasoning models should focus on integrating a broader set of specialized tools for analyzing visual content. Our system achieves significant gains (+10.3 on MMMU, +6.0 on MathVista over a 7B baseline), matching or surpassing much larger models. We will release our framework and evaluation suite to facilitate future research. 1 Introduction The ability to perform coherent, structured reasoning is essential for solving complex visual understanding tasks. Unlike recognition, visual reasoning requires models to integrate perceptual cues with contextual knowledge, infer relationships between entities, track logical dependencies, and arrive at conclusions that are not immediately evident from raw pixel data. This cognitive process mirrors human problem-solving, where one sequentially interprets visual inputs and iteratively verifies conclusions Liu et al. 2025b ; Zhang et al. 2025 ; Yang et al. 2025 ; Fu et al. 2025 . Figure 1: This image showcases our agent system that leverages a pure LLM to solve a visual reasoning problem using external tools. It illustrates how complex tasks, such as fitting traffic speed-density data to the Greenshields model, can offload substantial token usage to a code interpreter, highlighting an efficient division of labor between perception, reasoning and computation. Table 1: Comparison of our diagnostic agent with prior modular systems: MM-ReAct Yang et al. 2023 , MC-tree Yao et al. 2024 . System Math OCR Iterative Diagnosis Lightweight Backbone Python Interpreter Backtracing Thought MM-ReAct ✓ ✗ ✓ ✗ ✗ MC-tree ✗ ✓ ✓ ✗ ✓ Ours ✓ ✓ ✓ ✓ ✓ Recent advancements in LLMs have accelerated progress in this direction with strong linguistic reasoning abilities. When extended into the multimodal domain, these capabilities enable models to interpret images, diagrams, and documents extending beyond recognition to include inference and abstraction. The emergence of Reasoning Multimodal LLMs (MLLMs), such as LLaVA-CoT Xu et al. 2024 , LlamaV-o1 Thawakar et al. 2025 , and Heima hei 2025 , reflects this trend and demonstrates how the fusion of vision and language models can unlock new frontiers in visual intelligence. Figure 2: Comparison of token utilization and accuracy between Mulberry-7B and LLaVA-CoT-11B. Both models exhibit less adaptive reasoning, with token usage. We observe that LLaVA-CoT-11B frequently generates more tokens than Mulberry-7B, while Mulberry-7B tends to centralize its token usage around 300–350 tokens. Central to these efforts is to encourage models to produce explicit intermediate steps. This structured reasoning is particularly impactful for visual tasks, where raw perceptual data must be transformed into high-level concepts through a series of inferential stages. For example, LlamaV-o1 combines Chain-of-thought (CoT) reasoning with curriculum learning and beam search to effectively solve multi-step visual tasks, while Heima accelerates inference by encoding CoT into compact representations Shen et al. 2025 Despite these advances, models still hallucinate, producing responses not grounded in the image, and often rely too heavily on textual priors. Most models reason in a single, unidirectional pass, lacking correction or self-reflection. To address these issues, we introduce a three-stage diagnostic framework and an agent-based architecture that tightly integrates stepwise textual reasoning with lightweight visual modules, enabling fine-grained analysis of reasoning failures Author s ; Song et al. 2025 ; Kumar et al. 2025 ; Lu et al. 2025 ; Liu et al. 2025a . Unlike prior work (e.g., MC-tree Yao et al. 2024 , MM-ReAct Yang et al. 2023 ), our agent-based framework routes tool calls at each step for explicit intervention and diagnosis, similar in spirit to recent approaches like MMCTAgent Kumar et al. 2024 and AgentRE Shi et al. 2024 . Our main contributions are: Diagnosis: We present a diagnostic framework for math-centric visual reasoning, enabling granular identification and analysis of failures. Agent-based Architecture: We propose an agent-based approach that decouples perception and reasoning, integrating LLMs with visual modules for iterative reasoning, yielding substantial empirical gains over strong 7B baselines. Evaluation: We provide a comprehensive analysis of reasoning chains and release our diagnostic framework and evaluation suite to support future research in visual reasoning, enabling deeper understanding of model behaviors. 2 Diagnostic Methodology Our analysis focuses on three representative visual reasoning models: QVQ (72B) Team 2024 , Mulberry-7B Yao et al. 2024 (Mulberry), and LLaVA-CoT-11B Xu et al. 2024 (LLaVA-CoT). These models span a range of parameter sizes and are selected for their popularity and relevance. We exclude OpenAI’s O-series reasoning models due to the unavailability of their reasoning paths, which prevents in-depth diagnostic analysis. This selection enables a comprehensive comparison across different model scales and reasoning strategies. Figure 3: Token usage and accuracy trends for QVQ on MMMU. Left: Accuracy as a function of token count, showing diminishing returns and a decline beyond 2,000 tokens. Right: Distribution of token counts for correct and incorrect answers, including unfinished answers Dataset Mulberry LLaVA-CoT QVQ MMMU 52.8% 55.2% 60.9% MathVista 63.1% 57.8% 65.4% Base Model QwenVL2-7B Llama-3.2-7B QwenVL2-72B Table 2: Comparison of model accuracies (%) on MMMU and MathVista, and their base models. 2.1 Comparative Model Analysis We begin our evaluation with the MMMU dataset Yue et al. 2023 and MathVista Lu et al. 2024 , both selected for their comprehensive problem difficulty annotations and widespread use as benchmarks for MLLMs. For the MMMU dataset, we analyze token usage across varying difficulty levels to uncover patterns in reasoning efficiency and inefficiency, as shown in Figures 2 , 3 . As shown in Table 2 , Mulberry is highly succinct, with token counts clustered between 200–400, but this brevity limits its accuracy (52.8%/63.1%). Incorrect Mulberry responses often occur at the upper end of its token range, suggesting that rigid, template-driven reasoning can be counterproductive when the model stretches beyond its typical patterns. LLaVA-CoT, with token counts typically between 800–1,200 for correct Easy/Medium answers, achieves intermediate accuracy. On harder tasks, longer responses often correspond to incorrect answers, suggesting that concise yet sufficiently detailed reasoning chains tend to be optimal, whereas excessive verbosity may signal confusion. Our analysis shows that while more verbose reasoning can indicate higher capability, excessive token usage (beyond 2,000 for QVQ) yields diminishing returns. The best models balance detail and brevity, providing enough reasoning steps without unnecessary verbosity. QVQ’s larger size and flexible reasoning achieve the highest accuracy, but future work should aim to reduce verbosity while preserving reasoning quality. 2.2 In-depth Examination of QVQ As shown in Table 2 , QVQ achieves the highest accuracy—60.9% on MMMU and 65.4% on MathVista—outperforming both LLaVA-CoT and Mulberry. To better understand QVQ’s strengths, we analyze its reasoning behavior in detail. Figure 4 illustrates the relationship between token count and accuracy, revealing that accuracy declines as token usage increases.

Read the complete text