Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning

Haozhen Gong, Xiaozhong Ji, Yuansen Liu, Wenbin Wu, Xiaoxiao Yan, Jingjing Liu, Kai Wu, Jiazhen Pan, Bailiang Jian, Jiangning Zhang, Xiaobin Hu, Hongwei Bran Li, [3pt] National University of Singapore · 2025

Read Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning by Haozhen Gong, Xiaozhong Ji, Yuansen Liu, Wenbin Wu, Xiaoxiao Yan, Jingjing Liu, Kai Wu, Jiazhen Pan, Bailiang Jian, Jiangning Zhang, Xiaobin Hu, Hongwei Bran Li, [3pt] National University of Singapore (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00818.txt.

Introduction

Haozhen Gong and a large team introduce Med-CMR in a 2025 paper on medical AI. Large multimodal models are entering clinical work, where they must reason over images, text, and patient facts together. How well they handle complex medical reasoning is still unclear.

Med-CMR is a benchmark built to test that reasoning in fine detail. It joins visual evidence from medical images with clinical logic, the chain of thinking a doctor follows. A fine-grained test can show where a model reasons soundly and where it guesses. Hospitals and builders can use such results before trusting these systems near patients.

Catalog record

CREATORHaozhen Gong, Xiaozhong Ji, Yuansen Liu, Wenbin Wu, Xiaoxiao Yan, Jingjing Liu, Kai Wu, Jiazhen Pan, Bailiang Jian, Jiangning Zhang, Xiaobin Hu, Hongwei Bran Li, [3pt] National University of Singapore
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00818

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning Haozhen Gong, Xiaozhong Ji, Yuansen Liu, Wenbin Wu, Xiaoxiao Yan, Jingjing Liu, Kai Wu, Jiazhen Pan, Bailiang Jian, Jiangning Zhang, Xiaobin Hu, Hongwei Bran Li, [3pt] National University of Singapore Abstract MLLMs are beginning to appear in clinical workflows, but their ability to perform complex medical reasoning remains unclear. We present Med-CMR, a fine-grained Med ical C omplex M ultimodal R easoning benchmark. Med-CMR distinguishes from existing counterparts by three core features: 1) Systematic capability decomposition , splitting medical multimodal reasoning into fine-grained visual understanding and multi-step reasoning to enable targeted evaluation; 2) Challenging task design , with visual understanding across three key dimensions (small-object detection, fine-detail discrimination, spatial understanding) and reasoning covering four clinically relevant scenarios (temporal prediction, causal reasoning, long-tail generalization, multi-source integration); 3) Broad, high-quality data coverage , comprising 20,653 Visual Question Answering (VQA) pairs spanning 11 organ systems and 12 imaging modalities, validated via a rigorous two-stage (human expert + model-assisted) review to ensure clinical authenticity. We evaluate 18 state-of-the-art MLLMs with Med-CMR, revealing GPT-5 as the top-performing commercial model: 57.81 accuracy on multiple-choice questions (MCQs) and a 48.70 open-ended score, outperforming Gemini 2.5 Pro (49.87 MCQ accuracy, 45.98 open-ended score) and leading open-source model Qwen3-VL-235B-A22B (49.34 MCQ accuracy, 42.62 open-ended score). However, specialized medical MLLMs do not reliably outperform strong general models, and long-tail generalization emerges as the dominant failure mode. Med-CMR thus provides a stress test for visual–reasoning integration and rare-case robustness in medical MLLMs, and a rigorous yardstick for future clinical systems. Project page: https://github.com/LsmnBmnc/Med-CMR . 1 Introduction Multimodal large language models (MLLMs) Saab et al. 2024 ; Moor et al. 2023 ; Li et al. 2023a ; Chen et al. 2024 ; Lin et al. 2025 ; Nath et al. 2024 ; Tian et al. 2023 ; Chen et al. 2023 ; Wei Zhu and Wang 2023 ; Wang et al. 2025a are rapidly moving from proof-of-concept demos into tools that clinicians can actually touch. Before they are trusted in practice, we need to understand not only how often they are right, but how they reach decisions: can they detect subtle findings, integrate multiple images, track disease evolution, and reason about rare but critical scenarios? Concretely, we ask: to what extent can current MLLMs integrate medical images and clinical context to answer multi-step, reasoning-intensive questions, beyond basic VQA? Most existing multimodal medical benchmarks answer only a small part of this question Bitton et al. 2023 ; Zhu et al. 2024 ; Li et al. 2025 ; Ray et al. 2024 ; Lim et al. 2024 ; Yu et al. 2023 ; Yu et al. 2024 ; Xu et al. 2023 ; Lee et al. 2024 ; Roberts et al. 2024 ; Lau et al. 2018 ; He et al. 2020 ; Zhang et al. 2023 ; Hu et al. 2024 . They are dominated by perception-level visual question answering, where the model describes an image or retrieves an obvious fact from a short context. This setup hides many of the hard cases that shape clinical decisions: tiny, low-contrast lesions; cross-modal comparisons; temporal change; causal chains linking symptoms, imaging, and outcomes; and long-tailed distribution in textbooks. As a result, today’s benchmarks provide limited visibility into the complex medical reasoning capabilities that matter in real workflows. Evaluating such capabilities requires three ingredients that current benchmarks largely lack. First, systematic capability decomposition : instead of treating “medical multimodal reasoning" as a single score, we must separate fine-grained visual understanding from downstream reasoning, and further break both into clinically meaningful sub-dimensions. Second, clinically aligned and deliberately challenging tasks : questions should be built around real cases and explicitly target difficult settings such as temporal prediction, causal reasoning, long-tail generalization, and integration of multiple sources. Third, broad and well-curated coverage across organs, modalities, and disease processes, with expert review to ensure that questions remain realistic and clinically interpretable. To address these gaps, we introduce Med-CMR, a comprehensive benchmark that systematically evaluates MLLMs across multiple dimensions. Specifically, we categorizes reasoning complexity into three visual dimensions, small-object detection, fine-detail discrimination, and spatial understanding ; and four reasoning complexity dimensions, temporal prediction, causal reasoning, long-tail generalization, and multi-source integration . Each dimension corresponds to a distinct capability of MLLMs, enabling detailed diagnosis of model strengths and weaknesses. We collect data from authentic journal case reports and clinical research articles, followed by a two-stage filtering process combining human expert validation and model-assisted screening to ensure both quality and difficulty. Ultimately, Med-CMR achieves comprehensive coverage across 11 organ systems and 12 modalities, establishing a diverse foundation for assessing complex clinical reasoning (Figure 1 ). Figure 1 : Overview of Med-CMR. Med-CMR decomposes medical multimodal reasoning complexity into visual complexity (i.e., small-object detection, fine-detail discrimination, and spatial understanding) and reasoning complexity (i.e., temporal prediction, causal reasoning, long-tail generalization, and multi-source integration). Each dimension corresponds to a specific task designed to evaluate the model’s capability in that dimension. During the evaluation phase, we assess models using both multiple-choice and open-ended questions. While multiple-choice questions measure factual correctness, open-ended responses evaluate reasoning and generation quality. The latter are graded by an external, standard-aligned LLM based on four complementary criteria: Consistency, Coherence, Visual accuracy, and Ground-truth correctness, offering a holistic view of model performance. As validated in Sec. 6.1 , the LLM-based scoring shows high reliability and strong consistency with human judgment. We evaluate 18 proprietary and open-source MLLMs on Med-CMR to systematically compare their specific capabilities in complex clinical reasoning. GPT-5 achieves the best overall performance, reaching 57.81% accuracy on MCQs and a score of 48.70 on open-ended tasks. We conduct a detailed error analysis of GPT-5, the current upper-bound model, to identify the main causes of its failures and suggest directions for advancing real-world complex clinical reasoning. Through experiments, we analyze and validate the potential reasons why medical MLLMs do not consistently outperform general models in complex reasoning. Our main contributions are three-fold: • We propose Med-CMR, the first benchmark that enables fine-grained evaluation of clinical complex reasoning in MLLMs. Med-CMR decomposes reasoning into seven tasks that capture distinct sources of medical complexity, providing a structured and clinically grounded assessment of models’ capabilities. • We design a unified evaluation protocol combining multiple-choice and open-ended questions, where the latter are graded by an external, standard-aligned LLM along consistency, coherence, visual accuracy, and ground-truth correctness. This framework yields a holistic view of both answer correctness and reasoning quality. • We benchmark 18 proprietary and open-source vision–language models, including medical MLLMs and general-purpose MLLMs, and conduct detailed error and ablation analyses. We show that medical MLLMs do not consistently outperform general models in complex clinical reasoning, revealing concrete failure modes and directions for future improvement. Table 1 : Comparison of Med-CMR with prior multimodal medical benchmarks. The background colors denote benchmark categories: purple for early Medical VQA datasets focusing on basic understanding, orange for benchmarks that start to touch complex reasoning, and light yellow for our Med-CMR with fine-grained complex medical reasoning evaluation. Benchmark # Images # Size QA Type Annotation Type Broad Coverage Challenging Tasks Fine-grained Eval. VQA-RAD Lau et al. 2018 204 451 Open-ended Human \bm{\times} \bm{\times} \bm{\times} VQA-Med Abacha

Read the complete text