Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks

Jing Jin, Hao Liu, Yan Bai, Yihang Lou, Zhenke Wang, Tianrun Yuan, Juntong Chen, Yongkang Zhu, Fanhu Zeng, Xuanyu Zhu, Yige Xu · 2026

Read Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks by Jing Jin, Hao Liu, Yan Bai, Yihang Lou, Zhenke Wang, Tianrun Yuan, Juntong Chen, Yongkang Zhu, Fanhu Zeng, Xuanyu Zhu, Yige Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19697.txt.

Introduction

Jin and 11 colleagues introduce StepSTEM, a graduate-level testbed of 283 problems in mathematics, physics, chemistry, biology, and engineering for judging cross-modal reasoning in multimodal models. Existing tests often let models take unimodal shortcuts when text and images repeat each other, and they score only the final answer, ignoring the reasoning path itself. StepSTEM enforces strict complementarity between text and visuals through a rigorous curation pipeline, so each input carries information the other lacks.

Catalog record

CREATORJing Jin, Hao Liu, Yan Bai, Yihang Lou, Zhenke Wang, Tianrun Yuan, Juntong Chen, Yongkang Zhu, Fanhu Zeng, Xuanyu Zhu, Yige Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55055
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19697

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks Jing Jin, Hao Liu, Yan Bai, Yihang Lou, Zhenke Wang, Tianrun Yuan, Juntong Chen, Yongkang Zhu, Fanhu Zeng, Xuanyu Zhu, Yige Xu Abstract Multimodal large language models (MLLMs) have shown promising reasoning abilities, yet evaluating their performance in specialized domains remains challenging. STEM reasoning is a particularly valuable testbed because it provides highly verifiable feedback, but existing benchmarks often permit unimodal shortcuts due to modality redundancy and focus mainly on final-answer accuracy, overlooking the reasoning process itself. To address this challenge, we introduce StepSTEM : a graduate-level benchmark of 283 problems across mathematics, physics, chemistry, biology, and engineering for fine-grained evaluation of cross-modal reasoning in MLLMs. StepSTEM is constructed through a rigorous curation pipeline that enforces strict complementarity between textual and visual inputs. We further propose a general step-level evaluation framework for both text-only chain-of-thought and interleaved image-text reasoning, using dynamic programming to align predicted reasoning steps with multiple reference solutions. Experiments across a wide range of models show that current MLLMs still rely heavily on textual reasoning, with even Gemini 3.1 Pro and Claude Opus 4.6 achieving only 38.29% accuracy. These results highlight substantial headroom for genuine cross-modal STEM reasoning and position StepSTEM as a benchmark for fine-grained evaluation of multimodal reasoning. Source code is available at https://github.com/lll-hhh/STEPSTEM . 1 Introduction Recent advances in Multimodal Large Language Models (MLLMs) Wang et al. 2025 ; Qwen Team 2026 ; Deng et al. 2025 ; Chen et al. 2025 ; Cui et al. 2025 ; Team et al. 2026 have substantially expanded the scope of multimedia information processing. In addition to understand increasingly complex visual and textual inputs, MLLMs are capable of generating intermediate reasoning trajectories in text, image, or interleaved image-text formats, and derive the final answer based on these trajectories Su et al. 2025 ; Li et al. 2025a ; Chern et al. 2025 . This emerging capability has made MLLMs a promising foundation for solving complex multimodal reasoning problems. Among existing multimodal reasoning scenarios, STEM tasks are particularly attractive because of their strong verifiability: the correctness of the final answer is often objectively checkable, which provides a natural basis for supervising and analyzing the reasoning process itself. Despite the growing attention to multimodal STEM (Science, Technology, Engineering, and Mathematical) reasoning, the field still faces several important challenges. (1) First, current MLLM reasoning often exhibit a form of modality collapse , in which they over-rely on textual cues while under-utilizing visual evidence, leading to weak visual grounding and degraded performance on tasks that require genuine visual reasoning Zhao et al. 2025 ; Wu et al. 2025b ; Yao et al. 2025 . In STEM reasoning, the modality collapse becomes more significant because of the detailed textual description on STEM benchmarks. (2) Second, the generation and understanding of structured STEM visuals remain limited. Prior work shows that scientific figures, diagrams, and plots are poorly covered by existing training data, while current image-generation pipelines are largely optimized for naturalistic imagery rather than structured, logically constrained visuals with precise layouts, symbols, and labels Li et al. 2024b ; Wei et al. 2025 . (3) Third, current evaluation and post-training practices still emphasize final-answer correctness more than intermediate-step quality. The broader reasoning literature has shown that chain-of-thought rationales are not always faithful to the model’s actual reasoning process, while both LLM and MLLM studies suggest that step-wise supervision can better support reliable reasoning than outcome-only signals Turpin et al. 2023 ; Lightman et al. 2024 ; Zhang et al. 2025b . As a result, it remains difficult to determine whether a model truly performs grounded cross-modal reasoning or merely reaches the correct answer through superficial shortcuts. To address these limitations, we introduce StepSTEM , a new multimodal STEM benchmark with fine-grained reasoning step evaluation that includes engineering, physics, mathematics, chemistry, and biology subset. StepSTEM contains graduate-level multimodal STEM problems that require both textual information and visual evidence to derive the correct solution, thereby discouraging unimodal shortcuts and encouraging genuine cross-modal reasoning. As illustrated in Figure 1 , during data construction, we deliberately curate problems whose solutions cannot be recovered from text alone or vision alone. More importantly, for each problem we provide one or more ground-truth solution trajectories that contain both textual intermediate steps and visual intermediate steps. Built upon StepSTEM , we further introduce a step-level evaluation framework for multimodal reasoning. Instead of evaluating only the final answer, our framework first measures the similarity between each predicted reasoning step and the annotated ground-truth solution steps, and then aligns the predicted trajectory with one or multiple reference ground truth trajectories through dynamic programming to obtain an overall step-level reasoning score. In this way, the proposed protocol captures not only whether a model reaches the correct answer, but also whether it follows a plausible, grounded, and interpretable multimodal reasoning path. We evaluate a broad range of representative MLLMs on StepSTEM . Experimental results show that StepSTEM remains highly challenging for existing models, revealing substantial room for improvement in genuine multimodal STEM reasoning. Moreover, the proposed step-level scores provide a much more fine-grained view of model behavior than answer-only evaluation, enabling us to identify differences in reasoning quality, modality usage, and grounding ability that are otherwise obscured by final accuracy alone. In summary, our contributions are three-fold: (1) A new multimodal STEM reasoning benchmark that requires cross-modality reasoning : We introduce StepSTEM , consisting of 283 carefully curated problems that enforce strict complementarity to prevent unimodal shortcuts. (2) A fine-grained evaluation framework for intermediate reasoning step evaluation : We propose a step-level evaluation framework for systematically assessing intermediate multimodal reasoning steps beyond final-answer accuracy. (3) A comprehensive empirical analysis : We evaluate representative MLLMs on StepSTEM and provide empirical insights into their capabilities and limitations in cross-modal STEM reasoning. We further show that step-level scores strongly correlate with final-answer accuracy, highlighting the importance of evaluating reasoning processes in addition to final outcomes. 2 Related Works Figure 1: Overview of the StepSTEM construction workflow. The left panel illustrates the data curation pipeline. The right panel illustrates the multi-solution construction pipeline. 2.1 Multimodal Chain-of-Thought Reasoning Traditional Chain-of-Thought (CoT) techniques have achieved remarkable success in pure text logical deduction; however, when confronted with complex scenarios highly dependent on spatial perception and geometric intuition, pure text derivations often lack cognitively meaningful physical anchors. Consequently, multimodal interleaved CoT has emerged, with the core objective of decoupling the complex problem-solving process into a series of intermediate steps where textual deduction and visual evidence are deeply interwoven. Current explicit multimodal CoT methods primarily evolve along two technical trajectories. The first entails tool-augmented designs Li et al. 2024a ; Zhang et al. 2025a ; Hu et al. 2024 ; Duan et al. 2025 ; Zhu et al. 2026 ; Li et al. 2025d , which leverage external visual modules or code interpreters to assist reasoning; nevertheless, the interleaving process of such methods often exhibits high fragility and low flexibility. The second trajectory focuses on intrinsic visual CoT (Intrinsic VCoT) under unified model architectures. For instance, THINKMORPH Gu et al. 2025 reveals advanced reasoning characteristics that emerge under the multimodal interleaved training paradigm, while MathCanvas Shi et al. 2025a constructs an

Read the complete text