MMLU-Reason: Benchmarking Multi-Task Multi-modal Language Understanding and Reasoning
Guiyao Tie, Xueyang Zhou, Tianhe Gu, Ruihang Zhang, Chaoran Hu, Sizhe Zhang, Mengqu Sun, Yan Zhang, Pan Zhou, Lichao Sun · 2025
Read MMLU-Reason: Benchmarking Multi-Task Multi-modal Language Understanding and Reasoning by Guiyao Tie, Xueyang Zhou, Tianhe Gu, Ruihang Zhang, Chaoran Hu, Sizhe Zhang, Mengqu Sun, Yan Zhang, Pan Zhou, Lichao Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16459.txt.
Introduction
Guiyao Tie, Xueyang Zhou, Tianhe Gu, Ruihang Zhang, Chaoran Hu, Sizhe Zhang, Mengqu Sun, Yan Zhang, Pan Zhou, and Lichao Sun wrote this paper in two thousand twenty-five. It presents MMLU-Reason, a benchmark for how well multimodal large language models reason, not just what they answer.
The set holds over a thousand hard questions across six kinds of reasoning, each needing steps that hop between text, image, and form. A scoring pipeline grades the thought traces themselves for fit, coherence, and kinds of error. Models that show their thought beat those that do not, yet even the best, named in the paper, show flaws such as drift and overthinking. The gap between right answers and sound thought is the point.
Catalog record
| CREATOR | Guiyao Tie, Xueyang Zhou, Tianhe Gu, Ruihang Zhang, Chaoran Hu, Sizhe Zhang, Mengqu Sun, Yan Zhang, Pan Zhou, Lichao Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16459
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMLU-Reason: Benchmarking Multi-Task Multi-modal Language Understanding and Reasoning Guiyao Tie, Xueyang Zhou, Tianhe Gu, Ruihang Zhang, Chaoran Hu, Sizhe Zhang, Mengqu Sun, Yan Zhang, Pan Zhou, Lichao Sun Abstract Recent advances in Multi-Modal Large Language Models (MLLMs) have enabled unified processing of language, vision, and structured inputs, opening the door to complex tasks such as logical deduction, spatial reasoning, and scientific analysis. Despite their promise, the reasoning capabilities of MLLMs—particularly those augmented with intermediate thinking traces (MLLMs-T)—remain poorly understood and lack standardized evaluation benchmarks. Existing work focuses primarily on perception or final answer correctness, offering limited insight into how models reason or fail across modalities. To address this gap, we introduce the MMLU-Reason , a new benchmark designed to rigorously evaluate multi-modal reasoning with explicit thinking. The MMLU-Reason comprises 1) a high-difficulty dataset of 1,083 questions spanning six diverse reasoning types with symbolic depth and multi-hop demands and 2) a modular Reasoning Trace Evaluation Pipeline (RTEP) for assessing reasoning quality beyond accuracy through metrics like relevance, consistency, and structured error annotations. Empirical results show that MLLMs-T overall outperform non-thinking counterparts, but even top models like Claude-3.7-Sonnet and Gemini-2.5 Pro suffer from reasoning pathologies such as inconsistency and overthinking. This benchmark reveals persistent gaps between accuracy and reasoning quality and provides an actionable evaluation pipeline for future model development. Overall, the MMLU-Reason offers a scalable foundation for evaluating, comparing, and improving the next generation of multi-modal reasoning systems. Project Page: https://mmmr-benchmark.github.io/ . Figure 1: Overview of the MMLU-Reason . Left: Six reasoning tasks. Middle: Thinking Judgements assessing alignment for relevance (RTQ, RTA), consistency (RSC), and error (Err-T). Right: Accuracy distribution shows a gap between humans and state-of-the-art MLLMs-T. 1 Introduction The rapid advancement of Multi-Modal Large Language Models (MLLMs) has significantly enhanced unified reasoning capabilities across language, vision, and structured data modalities. Early MLLMs such as Qwen-VL [ 8 ] , LLaVA [ 24 ] , and GPT-4 Vision [ 35 ] have demonstrated impressive performance in perception-centric tasks, including visual question answering [ 7 ] , image captioning [ 38 ] , and grounded retrieval [ 37 ] . However, their proficiency remains limited in tasks necessitating structured reasoning, symbolic abstraction, or sequential multi-step inference. To address this limitation, a new paradigm—MLLMs incorporating explicit intermediate reasoning (MLLMs-T)—has emerged. Representative models such as Gemini-2.5 Pro [ 13 ] and Claude-3.7-Sonnet [ 6 ] leverage Chain-of-Thought-style reasoning, decomposing complex problems into interpretable intermediate steps, thereby emulating human-like structured problem-solving in domains such as logical deduction, scientific analysis, and code reasoning. Despite significant progress, rigorously evaluating the reasoning capabilities of MLLMs-T remains challenging. Current benchmarks, including MMBench [ 30 ] , MME-CoT [ 18 ] , and MMMU [ 47 ] , predominantly emphasize broad coverage of tasks and perceptual understanding, offering limited insights into the reasoning process itself. These benchmarks primarily focus on answer correctness without assessing the underlying reasoning’s consistency, coherence, or cognitive alignment. Consequently, a critical research question arises: To what extent do MLLMs-T reliably generate coherent, interpretable, and cognitively aligned reasoning traces in complex multi-modal tasks? Addressing this research question demands a benchmark emphasizing reasoning depth rather than breadth, evaluating not only final predictions but also the intermediate reasoning processes explicitly. Such a benchmark requires 1) challenging multi-modal reasoning tasks explicitly designed to probe structured inference capabilities, and 2) robust criteria for systematically evaluating intermediate reasoning quality. Currently, no benchmark adequately fulfills these requirements [ 30 , 18 , 47 , 31 ] , leaving a significant gap in diagnosing and attributing reasoning failures at the trace level. To bridge this gap, we introduce MMLU-Reason , a comprehensive benchmark explicitly designed to evaluate the multi-modal reasoning capabilities of both MLLMs and MLLMs-T. As shown in Figure 1 , our benchmark comprises 1,083 rigorously curated high-difficulty tasks spanning six reasoning domains: logical reasoning [ 18 , 43 ] , mathematical problem-solving [ 31 , 15 ] , spatio-temporal understanding [ 18 , 44 ] , code reasoning [ 22 , 21 ] , map-based planning [ 29 , 28 ] , and scientific analysis [ 18 , 47 ] . Each task integrates diverse modalities, including text, images, tables, and diagrams, carefully designed to require structured, symbolic, and abstract reasoning beyond mere perception. Unlike prior benchmarks [ 30 , 47 , 31 ] , MMLU-Reason introduces a structured Reasoning Trace Evaluation Pipeline (RTEP) , capturing reasoning trace relevance, logical consistency, and frequent error types. This structured evaluation identifies key reasoning pitfalls such as overthinking, trace inconsistency, and logical errors. The right panel of Figure 1 reveals a pronounced performance gap between state-of-the-art MLLMs-T and human-level expert reasoning. Specifically, while the best-performing model, Gemini-2.5 Pro, achieves a test accuracy of 42.45%, human experts assisted by GPT-4o reach 52.85%. This 10.3% margin underscores the persistent challenge in closing the reasoning gap, even with advanced architectures and explicit thinking mechanisms. In summary, our contributions include: • A comprehensive benchmark for multi-modal reasoning. We introduce MMLU-Reason , the first benchmark that systematically targets multi-modal reasoning across six domains—Logic, Math, Code, Map, Science, and Space-Time. Unlike prior datasets, MMLU-Reason emphasizes hard question solutions and cross-modal alignment to increase reasoning complexity. • The first evaluation pipeline for thinking of MLLMs-T. We propose the Reasoning Trace Evaluation Pipeline (RTEP), the first framework to incorporate intermediate thinking trace analysis into multi-modal reasoning evaluation. RTEP assesses reasoning relevance, stepwise consistency, and alignment, which enables deeper diagnostic insight beyond accuracy. • Insights into reasoning capabilities and failures. Through extensive evaluation on MMLU-Reason , we find that state-of-the-art MLLMs-T achieve high answer accuracy across tasks, yet frequently produce flawed reasoning traces—exhibiting logical inconsistency or overthinking. These findings expose a critical misalignment between surface-level correctness and reasoning fidelity, offering new evaluation directions for future multi-modal model architecture. 2 MMMR: Benchmarking Massive Multi-Modal Reasoning Tasks The MMLU-Reason is a challenging benchmark dataset meticulously crafted to evaluate the reasoning capabilities of Multi-modal Large Language Models with intermediate Thinking (MLLMs-T). Unlike previous benchmarks [ 7 , 16 , 19 , 33 ] which predominantly measure perception or general knowledge, MMLU-Reason emphasizes complex reasoning tasks requiring deep integration across diverse modalities, such as text, images, and structured data. Motivated by recent advancements in MLLMs-T, exemplified by Gemini-2.5 Pro [ 12 ] , which leverage intermediate reasoning processes to enhance performance, we propose a rigorous three-stage evaluation pipeline. This pipeline is specifically designed to evaluate multi-modal reasoning quality and, crucially, assess the effectiveness and robustness of intermediate thinking. As illustrated in Figure 2 , our evaluation pipeline comprises three core stages: (I) Reasoning Dataset Construction, (II) Thinking Quality Assessment, and (III) Reasoning Insights Synthesis. Figure 2: Overview of the MMLU-Reason evaluation pipeline. Stage I involves the creation of a challenging multi-modal reasoning benchmark dataset. Stage II evaluates the quality and structural integrity of intermediate reasoning generated by MLLMs-T. Stage III synthesizes insights regarding reasoning strategies, effectiveness, and common failure patterns across different tasks and models. 2.1 Stage I: Reasoning Dataset Construction In this initial stage, we construct the MMLU-Reason dataset to thoroughly evaluate MLLMs-T across a wide spectrum of reasoning scenarios. The MMLU-Reason comprises 1,083 carefully curated multi-modal tasks, systematically categorized into six distinct reasoning types: Logic (16.8%), Math (19.5%), Space-Time (18.5%), Code (13.0%), Map (13.8%), and