Evaluating MLLMs with M ultimodal M ulti-image R easoning B enchmark DOI: XXXXXXX.XXXXXXX Conference: ACM International Conference on Multimedia (Dataset Track); October 2025; Lisbon, Portugal ISBN: 978-1-4503-XXXX-X/2018/06 CCS: General and reference Evaluation CCS: Computing methodologies Natural
Ziming Cheng 1,†,§ , Binrui Xu 1,§ , Lisheng Gong 1,§ , Zuhe Song 1,§ , Tianshuo Zhou 1,* , Shiqi Zhong 1,* , Siyu Ren 1,* , Mingxiang Chen 1,* , Xiangchao Meng 1,* , Yuxin Zhang 2,* , Yanlin Li 3,* , Lei Ren 4 , Wei Chen 4 , Zhiyuan Huang 5 , Mingjie Zhan 5 , Xiaojie Wang 1 , Fangxiang Feng 1 · 2025
Read Evaluating MLLMs with M ultimodal M ulti-image R easoning B enchmark DOI: XXXXXXX.XXXXXXX Conference: ACM International Conference on Multimedia (Dataset Track); October 2025; Lisbon, Portugal ISBN: 978-1-4503-XXXX-X/2018/06 CCS: General and reference Evaluation CCS: Computing methodologies Natural by Ziming Cheng 1,†,§ , Binrui Xu 1,§ , Lisheng Gong 1,§ , Zuhe Song 1,§ , Tianshuo Zhou 1,* , Shiqi Zhong 1,* , Siyu Ren 1,* , Mingxiang Chen 1,* , Xiangchao Meng 1,* , Yuxin Zhang 2,* , Yanlin Li 3,* , Lei Ren 4 , Wei Chen 4 , Zhiyuan Huang 5 , Mingjie Zhan 5 , Xiaojie Wang 1 , Fangxiang Feng 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04280.txt.
Introduction
Ziming Cheng, Binrui Xu, and their co-authors built a benchmark that tests how well multimodal models reason across several images at once. Their 2025 paper notes that older tests cover either single images or many images with only a final answer checked. The reasoning steps in multi image tasks were largely untested.
Their benchmark holds 92 sub tasks spanning space, time, and meaning based reasoning. Each item carries step by step notes drafted by a large model and refined by human experts, and many items allow more than one right answer. A smaller set tests reward models that grade other models' answers. A sentence level matching method makes scoring fast. Tests of current models show where multi image reasoning still falls short.
Catalog record
| CREATOR | Ziming Cheng 1,†,§ , Binrui Xu 1,§ , Lisheng Gong 1,§ , Zuhe Song 1,§ , Tianshuo Zhou 1,* , Shiqi Zhong 1,* , Siyu Ren 1,* , Mingxiang Chen 1,* , Xiangchao Meng 1,* , Yuxin Zhang 2,* , Yanlin Li 3,* , Lei Ren 4 , Wei Chen 4 , Zhiyuan Huang 5 , Mingjie Zhan 5 , Xiaojie Wang 1 , Fangxiang Feng 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-04280
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating MLLMs with M ultimodal M ulti-image R easoning B enchmark DOI: XXXXXXX.XXXXXXX Conference: ACM International Conference on Multimedia (Dataset Track); October 2025; Lisbon, Portugal ISBN: 978-1-4503-XXXX-X/2018/06 CCS: General and reference Evaluation CCS: Computing methodologies Natural language generation CCS: Computing methodologies Scene understanding Ziming Cheng 1,†,§ , Binrui Xu 1,§ , Lisheng Gong 1,§ , Zuhe Song 1,§ , Tianshuo Zhou 1,* , Shiqi Zhong 1,* , Siyu Ren 1,* , Mingxiang Chen 1,* , Xiangchao Meng 1,* , Yuxin Zhang 2,* , Yanlin Li 3,* , Lei Ren 4 , Wei Chen 4 , Zhiyuan Huang 5 , Mingjie Zhan 5 , Xiaojie Wang 1 , Fangxiang Feng 1 Abstract . With enhanced capabilities and widespread applications, Multimodal Large Language Models (MLLMs) are increasingly required to process and reason over multiple images simultaneously. However, existing MLLM benchmarks focus either on single-image visual reasoning or on multi-image understanding tasks with only final-answer evaluation, leaving the reasoning capabilities of MLLMs over multi-image inputs largely underexplored. To address this gap, we introduce the Multimodal Multi-image Reasoning Benchmark (MMRB) , the first benchmark designed to evaluate structured visual reasoning across multiple images. MMRB comprises 92 sub-tasks covering spatial, temporal, and semantic reasoning, with multi-solution, CoT-style annotations generated by GPT-4o and refined by human experts. A derivative subset is designed to evaluate multimodal reward models in multi-image scenarios. To support fast and scalable evaluation, we propose a sentence-level matching framework using open-source LLMs. Extensive baseline experiments on 40 MLLMs , including 9 reasoning-specific models and 8 reward models, demonstrate that open-source MLLMs still lag significantly behind commercial MLLMs in multi-image reasoning tasks. Furthermore, current multimodal reward models are nearly incapable of handling multi-image reward ranking tasks. Figure 1. Overview of the MMRB benchmark, which evaluates MLLMs on 92 multi-image-only sub-tasks annotated with reasoning steps. Datasets Year Domain Num Sub-tasks Num Samples Total Reasoning Steps Avg Reasoning Steps Avg Images Multiple Solutions CoMT ( Cheng et al. 2025 ) 2024 Math, General 4 3,853 29,706 7.71 3.84 \times WorldQA ( Zhang et al. 2024b ) 2024 General 1 1,007 4,481 4.45 2.41 \times MiCEval ( Zhou et al. 2024 ) 2024 General 8 643 2,889 4.49 1 \times MME-CoT ( Jiang et al. 2025 ) 2025 Science, Math, General 23 1,130 3,865 3.42 2.95 \times VisualProcessBench ( Wang et al. 2025 ) 2025 Math, Science 5 2,866 26,950 9.40 1 \times SVIP-Test ( Gao et al. 2025 ) 2025 General 10 1,934 5,509 2.85 1 \times M3CoT ( Chen et al. 2024a ) 2025 Science, Math, General 17 11,459 124,903 10.90 0.98 \times VRC-Bench ( Thawakar et al. 2025 ) 2025 Science, Math, General 8 1000 4173 4.17 1 \times MMRB (Ours) 2025 General 92 4750 68,882 4.83 6.17 Avg 1.93 Table 1. Comparison of MMRB with recently published visual reasoning benchmarks. 1. Introduction Multimodal Large Language Models (MLLMs), represented by OpenAI o1 ( Jaech et al. 2024 ) and Gemini 2.5 Pro ( Anil et al. 2023 ) , exhibit extraordinary performance in visual reasoning tasks and have fueled a boom in research on open-source visual reasoning models based on Reinforcement Learning (RL). By incorporating Chain-of-Thought (CoT) ( Wei et al. 2022 ) or Multimodal-CoT (MCoT) ( Zhang et al. 2023 ) techniques, models such as LLaVA-o1 ( Xu et al. 2024 ) and LlamaV-o1 ( Thawakar et al. 2025 ) , which adopt a systematic and structured multimodal reasoning process, demonstrate promising potential in approaching the capabilities of OpenAI o1. To thoroughly and systematically evaluate the performance of these MLLMs, a number of benchmarks involving image contexts have been published in parallel. These evaluative efforts fall into two main directions. One focuses on complex visual reasoning problems such as science-related question answering (e.g., ScienceQA ( Saikh et al. 2022 ) , MMMU ( Yue et al. 2024 ) ) and mathematical reasoning, particularly in algebra (e.g., MathVista ( Lu et al. 2023 ) , MathVision ( Wang et al. 2024c ) ), featuring extensive annotations of various intermediate reasoning steps and evaluation protocols supported by LLMs. Their evaluation domains have recently been extended to more general scenarios by works such as M3CoT ( Chen et al. 2024a ) , MME-CoT ( Jiang et al. 2025 ) , and CoMT ( Cheng et al. 2025 ) , although they still primarily focus on a single-image setting. Another direction expands the domain by introducing multi-image understanding tasks. Works such as BLINK ( Fu et al. 2024 ) and MuirBench ( Wang et al. 2024b ) collected multi-image tasks from publicly available sources, MIBench ( Liu et al. 2024 ) introduced image-attached In-Context Learning (ICL) tasks, and MMIU ( Meng et al. 2024 ) further aggregated such tasks from existing benchmarks, covering a total of 52 tasks. However, all existing MLLM benchmarks in the literature involve either single-image visual reasoning tasks or multi-image understanding tasks that include only answer annotations without reasoning steps, largely leaving the evaluation of visual reasoning in multi-image scenarios underexplored and making the reasoning quality of MLLMs with multi-image inputs an open question. Motivated by this observation, we introduce a novel Multimodal Multi-image Reasoning Benchmark (MMRB) , which is the first to combine visual reasoning and multi-image understanding tasks to evaluate MLLMs’ reasoning capability. Specifically, we collect a total of 92 reasoning-specific sub-tasks from prior multi-image datasets and benchmarks that cover spatial, temporal, and high-level semantic reasoning dimensions ( Meng et al. 2024 ) . We then annotate the intermediate reasoning paths using GPT-4o and refine them through expert human corrections. All reasoning paths involved follow structured and systematic MCoT steps, and additionally retain multiple possible thinking and solutions. As shown in Table 1 , MMRB stands out as the largest benchmark by sub-task count and image density, the only one to offer multiple-solution annotations. Using reasoning steps initially annotated by GPT-4o and subsequently corrected by humans, we further construct a subset for evaluating the performance of multimodal reward models in multi-image contexts—the first of its kind in the literature. To comprehensively evaluate MLLMs’ performance on MMRB and democratize the evaluation process, we developed an open-sourced LLM-based evaluator that matches CoT answers with annotated ground-truth reasoning steps at the sentence level. This approach avoids expensive and time-consuming GPT-based evaluation while retaining high evaluation quality. We evaluated a diverse set of 40 commercial and open-source MLLMs with support for multi-image input, including 9 reasoning-specific models and 8 multimodal reward models on our MMRB benchmark, yielding several key findings: (1) Model, data, and inference-time scaling all improve performance on multi-image reasoning tasks. (2) CoT prompting boosts accuracy more substantially for non-reasoning models, while still being beneficial for reasoning-specific models. (3) Multimodal reward models exhibit instability in multi-image reward tasks. 2. Related Work 2.1. Visual Reasoning Benchmarks Visual reasoning is the cognitive process to solve multi-modal problems through processing and manipulating visual information, and is the core capability of both human and advanced MLLMs ( Xu et al. 2025 ) . In recent years, the evaluation of MLLMs’ performance on visual reasoning tasks has gained attention. This line of research initially focused on mathematical problem-solving, such as algebra ( Wang et al. 2024c ) , and has since expanded to the domains including scientific reasoning