MANBench: Is Your Multimodal Model Smarter than Human?

Han Zhou, Qitong Xu, Yiheng Dong, Xin Yang · 2025

Read MANBench: Is Your Multimodal Model Smarter than Human? by Han Zhou, Qitong Xu, Yiheng Dong, Xin Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11080.txt.

Introduction

Han Zhou and co-authors introduce MANBench, a test of multimodal smarts for AI and humans, in a 2025 paper. The benchmark asks whether today's multimodal models can beat people. It holds over a thousand questions in English and Chinese across nine tasks. Some tasks need stored knowledge, and others rely on quick intuitive reasoning.

Human volunteers of varied backgrounds took the same tests as leading models. Models did well on knowledge and text-image reading tasks. They lagged on deeper cross-modal reasoning, like tracking change across images. Both humans and models struggled with puzzles and spatial imagination. The authors hope the benchmark pushes models closer to human-level skill. Its code and data are public.

Catalog record

CREATORHan Zhou, Qitong Xu, Yiheng Dong, Xin Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-11080

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MANBench: Is Your Multimodal Model Smarter than Human? Han Zhou, Qitong Xu, Yiheng Dong, Xin Yang Abstract The rapid advancement of Multimodal Large Language Models (MLLMs) has ignited discussions regarding their potential to surpass human performance in multimodal tasks. In response, we introduce MANBench (Multimodal Ability Norms Benchmark), a bilingual benchmark (English and Chinese) comprising 1,314 questions across nine tasks, spanning knowledge-based and non-knowledge-based domains. MANBench emphasizes intuitive reasoning, seamless cross-modal integration, and real-world complexity, providing a rigorous evaluation framework. Through extensive human experiments involving diverse participants, we compared human performance against state-of-the-art MLLMs. The results indicate that while MLLMs excel in tasks like Knowledge and Text-Image Understanding, they struggle with deeper cross-modal reasoning tasks such as Transmorphic Understanding, Image Consistency, and Multi-image Understanding. Moreover, both humans and MLLMs face challenges in highly complex tasks like Puzzles and Spatial Imagination. MANBench highlights the strengths and limitations of MLLMs, revealing that even advanced models fall short of achieving human-level performance across many domains. We hope MANBench will inspire efforts to bridge the gap between MLLMs and human multimodal capabilities. The code and dataset are available at https://github.com/micdz/MANBench . 1 Introduction Recent advancements in Large Language Models (LLMs) have demonstrated exceptional capabilities in text comprehension, reasoning, and generation ( Liu et al. 2024a ; Dubey et al. 2024 ; Jaech et al. 2024 ) . Building on this foundation, Multimodal Large Language Models (MLLMs) have emerged, significantly enhancing the ability to process and generate multimodal content ( OpenAI 2024b ; Team 2025 ; Anthropic 2024 ; Team et al. 2023 ) . These models have achieved substantial success in tasks such as image captioning, visual question answering, and visual dialog, showcasing their potential for multimodal understanding and generation. As artificial intelligence continues to evolve rapidly, a critical question arises: Can the general multimodal capabilities of MLLMs eventually surpass those of humans? While humans possess sophisticated multimodal abilities, existing benchmarks for evaluating these capabilities exhibit several significant limitations. First, many benchmarks fail to comprehensively assess the full spectrum of multimodal reasoning, often focusing on narrow tasks or requiring domain-specific knowledge ( Lu et al. 2024 ; Chen et al. 2024b ; Zhang et al. 2024b ; Wang et al. 2024c ) . Second, human evaluations in these benchmarks are frequently inadequate, relying on small sample sizes and limited diversity. These shortcomings hinder robust conclusions about human performance ( Zhang et al. 2024b ; Wang et al. 2024c ; Yue et al. 2024 ; Russakovsky et al. 2014 ; Radford et al. 2021 ) . To address these issues, we introduce MANBench (Multimodal Ability Norms Benchmark), a comprehensive benchmark designed to evaluate the multimodal capabilities of both humans and MLLMs. MANBench comprises nine tasks, encompassing 1,314 questions and 2,231 images. It aims to provide a fair and rigorous assessment framework, ensuring equitable comparisons between human and MLLM performance. Specifically, our benchmark addresses the following key limitations of existing benchmarks: (1) it separates questions requiring prior knowledge from those that do not, (2) it ensures that all questions demand reasoning rather than simple retrieval, (3) it mandates the integration of textual and visual information, and (4) it includes a wide range of reasoning difficulties to challenge both humans and MLLMs. We conducted a large-scale human evaluation using the MANBench dataset, systematically comparing the performance of humans and MLLMs across a variety of multimodal tasks. Twelve MLLMs, including state-of-the-art models such as GPT-4o ( OpenAI 2024b ) , Qwen2.5-VL-72B-Instruct ( Team 2025 ) , and Gemini-1.5 Pro ( Team et al. 2023 ) , were evaluated on MANBench. Additionally, we recruited 575 participants from diverse backgrounds to complete the tasks. Our results reveal that while MLLMs excel in tasks such as Knowledge and Text-Image Understanding, they struggle with tasks requiring deeper cross-modal reasoning, including Transmorphic Understanding, Image Consistency, and Multi-image Understanding. Notably, both humans and MLLMs encounter challenges in highly complex tasks such as Puzzles and Spatial Imagination. In summary, our contributions are as follows: (1) We introduce MANBench, a bilingual benchmark with 1,314 questions across nine tasks. (2) We conduct extensive human experiments with 575 diverse participants. (3) We evaluate 12 state-of-the-art MLLMs. (4) We provide insights into the strengths and limitations of MLLMs, highlighting the challenges in bridging the gap between MLLMs and human multimodal abilities. 2 Related Work 2.1 Multimodal Large Language Models As LLMs advance, integrating visual content has become a critical step in the development of interactive intelligent assistants. MLLMs focus on aligning vision and language modalities. CLIP ( Radford et al. 2021 ) is foundational in this field, using contrastive learning on image-text pairs. Approaches like MiniGPT-4 ( Zhu et al. 2023 ) and LLaVA ( Liu et al. 2024b ; Liu et al. 2024c ) , connect vision encoders and LLMs using techniques like Q-Former and MLP, leveraging frozen language models with a limited number of trainable parameters, demonstrating promising results. Furthermore, a range of MLLMs trained on vast datasets are now accessible to the public, pushing the boundaries of visual comprehension and generalization, including InternVL1.5, InternVL2.5 series ( Chen et al. 2024c ) and Qwen-VL ( Wang et al. 2024e ) . 2.2 Evaluations of MLLMs Reasoning ability is a fundamental indicator of intelligence. Numerous benchmarks have been developed to evaluate reasoning in text-only modalities, including MathQA ( Amini et al. 2019 ) , GSM1k ( Zhang et al. 2024a ) , and LiveBench ( White et al. 2024 ) . These benchmarks present questions that require advanced reasoning and knowledge, making them difficult for most people to solve. To comprehensively assess the multimodal reasoning capabilities of both humans and MLLMs, we propose the development of a novel text-image benchmark that addresses certain limitations in existing evaluation frameworks. As illustrated in Figure 1 , current benchmarks exhibit four primary areas for improvement. Figure 1: Limitations of existing text-image benchmarks. (a) Some samples necessitate prior knowledge rather than reasoning capabilities ( Liu et al. 2024d ) . (b) Certain samples can be answered accurately without utilizing the image content ( Lu et al. 2022 ) . (c) Other samples lack sufficient reasoning complexity ( Fu et al. 2024 ) . (d) In contrast, our proposed questions require comprehensive reasoning, effective integration of textual and visual information, and a diverse range of reasoning difficulties. Limitation 1: Overreliance on Prior Knowledge in Certain Samples Existing benchmarks frequently emphasize prior knowledge over reasoning capability ( Lu et al. 2024 ; Chen et al. 2024b ; Zhang et al. 2024b ; Wang et al. 2024c ) , leading to variability in evaluating human performance. To address this issue, MANBench introduces a clear distinction between Knowledge-based and non-knowledge-based tasks, facilitating a more balanced evaluation of cognitive processes. Limitation 2: Potential for Knowledge Retrieval Without Reasoning Current benchmarks sometimes enable solutions through direct knowledge retrieval ( Chen et al. 2024a ) , which limits their ability to effectively evaluate reasoning skills. Our proposed dataset mitigates this with carefully constructed questions that explicitly require reasoning processes, while clearly distinguishing knowledge-based tasks. Limitation 3: Insufficient Integration of Multimodal Information Some existing benchmarks fail to effectively utilize the integration potential of textual and visual data ( Chen et al. 2024a ) . Our framework addresses this by introducing

Read the complete text