Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks
Miao Jing, Mengting Jia, Junling Lin, Zhongxia Shen, Lijun Wang, Yuanyuan Peng, Huan Gao, Mingkun Xu, Shangyang Li · 2025
Read Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks by Miao Jing, Mengting Jia, Junling Lin, Zhongxia Shen, Lijun Wang, Yuanyuan Peng, Huan Gao, Mingkun Xu, Shangyang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22258.txt.
Introduction
Vision-language models score well on standard medical tests, yet Miao Jing and six co-authors doubt those scores reflect real clinical reasoning. Most medical benchmarks reward classification accuracy, which creates a false sense of skill while models still fail at high-stakes diagnosis. Their 2025 paper introduces Neural-MedBench, a compact but reasoning-heavy benchmark for neurology. It combines multi-sequence MRI scans, structured health records, and clinical notes in each case.
The benchmark's tasks fall in three families: telling apart possible diagnoses, recognizing lesions, and writing out the reasoning behind an answer. A hybrid scoring pipeline mixes model-based grading, clinician validation, and meaning checks. When leading models, including well-known commercial and medical ones, were evaluated, their performance dropped sharply compared with conventional data sets. The failures were mostly reasoning failures, not failures to see the images.
Catalog record
| CREATOR | Miao Jing, Mengting Jia, Junling Lin, Zhongxia Shen, Lijun Wang, Yuanyuan Peng, Huan Gao, Mingkun Xu, Shangyang Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-22258
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks Miao Jing, Mengting Jia, Junling Lin, Zhongxia Shen, Lijun Wang, Yuanyuan Peng, Huan Gao, Mingkun Xu, Shangyang Li Abstract Recent advances in vision-language models (VLMs) have achieved remarkable performance on standard medical benchmarks, yet their true clinical reasoning ability remains unclear. Existing datasets predominantly emphasize classification accuracy, creating an evaluation illusion in which models appear proficient while still failing at high-stakes diagnostic reasoning. We introduce Neural-MedBench , a compact yet reasoning-intensive benchmark specifically designed to probe the limits of multimodal clinical reasoning in neurology. Neural-MedBench integrates multi-sequence MRI scans, structured electronic health records, and clinical notes, and encompasses three core task families: differential diagnosis, lesion recognition, and rationale generation. To ensure reliable evaluation, we develop a hybrid scoring pipeline that combines LLM-based graders, clinician validation, and semantic similarity metrics. Through systematic evaluation of state-of-the-art VLMs, including GPT-4o, Claude-4, and MedGemma, we observe a sharp performance drop compared to conventional datasets. Error analysis shows that reasoning failures, rather than perceptual errors, dominate model shortcomings. Our findings highlight the necessity of a Two-Axis Evaluation Framework: breadth-oriented large datasets for statistical generalization, and depth-oriented, compact benchmarks such as Neural-MedBench for reasoning fidelity. We release Neural-MedBench at https://neuromedbench.github.io/ as an open and extensible diagnostic testbed, which guides the expansion of future benchmarks and enables rigorous yet cost-effective assessment of clinically trustworthy AI. 1 Introduction Recent advances in vision-language models (VLMs) have led to striking improvements across a wide range of medical AI tasks. On standard benchmarks such as MedMNIST v2 Yang et al. 2023 and MultiMedQA Zhou et al. 2024 , state-of-the-art models achieve near-human or even superhuman performance in label prediction and image–text alignment. These results have created an impression that medical VLMs are nearing clinical readiness. Yet, as the clinical reasoning literature has recently underscored Schwartzstein 2024 , safe and effective diagnostic practice, especially in high-stakes fields such as neurology, demands more than classification accuracy: it requires multimodal synthesis, ambiguity resolution, and the capacity to justify conclusions in a manner consistent with clinical logic. Current benchmarks, despite their scale, rarely capture these aspects. We argue that this discrepancy creates an evaluation illusion , a misleading sense of model capability, where strong performance on shallow benchmarks obscures deep weaknesses in reasoning fidelity and clinical trustworthiness. To address this gap, we propose a new perspective on evaluation: a Two-Axis Evaluation Framework for medical AI. Along the first axis, Breadth , large-scale datasets assess statistical generalization and coverage across populations and conditions. This is where almost all existing benchmarks operate. However, recent work such as DiagnosisArena Zhu et al. 2025 has shown that relying solely on such breadth-oriented evaluations leads to overestimation of model capability, as they fail to capture true clinical reasoning challenges. Along the second axis, Depth , compact but diagnostically complex, expert-curated benchmarks assess reasoning fidelity, forcing models to integrate multimodal signals, reason under uncertainty, and provide structured justifications, an approach that echoes trends in MedAgentsBench Tang et al. 2025 . Importantly, we hypothesize that these axes are largely uncorrelated: success on breadth benchmarks does not guarantee competence in depth scenarios. This hypothesis is supported by findings that even expert LLMs struggle with self-awareness and metacognitive reasoning on challenging cases, as shown in MetaMedQA Griot et al. 2025 . A complete picture of model readiness therefore requires evaluation along both dimensions. Figure 1: Conceptual Positioning of Neural-MedBench and Empirical Evidence of the Evaluation Illusion. (Left) An illustrative mapping of existing medical VLM benchmarks along the proposed axes of Breadth (dataset size and diversity) and Depth (reasoning complexity). Most benchmarks occupy the high-Breadth, low-Depth space, leaving a critical gap in high-Depth evaluation, which Neural-MedBench is designed to fill. (Right) Performance of leading VLMs on two Breadth-Axis benchmarks (MMLU-Pro and DrVD-Bench) versus our Depth-Axis benchmark ( Neural-MedBench ). The stark performance drop on Neural-MedBench provides clear empirical evidence of the ”evaluation illusion” and the disconnect between the two evaluation axes. Table. 2 shows full results. To operationalize the Depth axis in clinical neurology, we introduce Neural-MedBench , a diagnostic benchmark deliberately designed as a “stress test.” Neural-MedBench comprises 120 expert-annotated multimodal cases, including multi-sequence MRI scans, structured electronic health records (EHRs), and clinical narratives, yielding 200 reasoning-intensive tasks. Its design mirrors practices in medical education, such as Objective Structured Clinical Examinations (OSCEs), where carefully constructed cases are used to assess reasoning skills under uncertainty Geathers et al. 2025 ; Siegelman et al. 2024 . Unlike prior resources focused on breadth, Neural-MedBench is compact by design, prioritizing reasoning density over volume, enabling cost-effective yet high-signal evaluation of clinical reasoning. Our evaluation of leading models, including GPT-4o, Claude 4, and MedGemma, reveals a striking pattern: models that excel on large-scale benchmarks fail systematically on Neural-MedBench . Error analysis shows that these failures stem not from perception or lexical mismatch, but rather from breakdowns in clinical reasoning, limitations that remain hidden under existing evaluation paradigms. This disconnect provides the first empirical evidence for the independence of the two axes, underscoring the urgent need for benchmarks that explicitly capture reasoning depth. This work makes the following contributions: • We propose the Two-Axis Evaluation Framework, arguing that trustworthy clinical AI requires complementary assessments of both breadth (statistical generalization) and depth (reasoning fidelity). • We release Neural-MedBench , the first neurology-focused benchmark explicitly designed to operationalize the Depth axis, comprising 120 multimodal, expert-curated diagnostic cases with 200 reasoning-intensive tasks. • We provide empirical evidence of the disconnect between breadth and depth, showing that state-of-the-art VLMs fail primarily at reasoning, despite strong performance on existing large-scale datasets. • We present a systematic error analysis and a human performance baseline to contextualize model results, and we release Neural-MedBench as an open, extensible resource with a public leaderboard and roadmap for expansion. 2 Related Work 2.1 Medical VLM Benchmarks A wide range of benchmarks have been developed for evaluating medical VLMs. Early datasets such as ChestX-ray14 Wang et al. 2017 , CheXpert Irvin et al. 2019 , and MIMIC-CXR Johnson et al. 2019 primarily enabled large-scale classification and report generation. Organ- or disease-specific challenges—including LUNA for pulmonary nodules Setio et al. 2017 , LiTS for liver tumors Bilic et al. 2019 , BraTS for brain tumors Menze et al. 2015 , and ISLES for stroke Winzeck et al. 2017 —expanded evaluation to segmentation and lesion characterization. More recent multimodal benchmarks target richer tasks: ROCO v2 Rückert et al. 2024 for radiology captioning, RadGraph Jain et al. 2021 for entity–relation extraction, and MedFMC Zhang et al. 2023 for federated multi-center classification. VQA-style benchmarks, such as VQA-RAD Lau et al. 2018 , PathVQA He et al. 2020 , and OmniMedVQA Hu et al. 2024 , test text–image reasoning but often rely on templated questions, limiting their coverage of authentic diagnostic challenges. Recent meta-benchmarks (e.g., Rad-Bench Kuo et al. 2024 , DiagnosisArena Zhu et al. 2025 , MedAgentsBench Tang et al. 2025 ) have emphasized comprehensiveness, yet remain breadth-oriented: they focus on scale and diversity rather than probing reasoning under uncertainty. In contrast, Neural-MedBench explicitly operationalizes the Depth axis of evaluation. By curating 120 multimodal, diagnostically complex neurology cases into 200 reasoning-intensive tasks, it emphasizes reasoning fidelity over volume. This design mirrors clinical examinations such