OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks
Zhihao Peng, Cheng Wang, Shengyuan Liu, Zhiying Liang, Yixuan Yuan · 2025
Read OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks by Zhihao Peng, Cheng Wang, Shengyuan Liu, Zhiying Liang, Yixuan Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00846.txt.
Introduction
Zhihao Peng and a large team introduce OmniBrainBench, a benchmark for testing how well multimodal language models read brain images. Existing question answering benchmarks cover few imaging types or only coarse descriptions of disease, so they miss much of the clinical workflow.
The benchmark spans fifteen imaging modalities from thirty verified medical sources, with more than nine thousand validated question pairs and over thirty thousand images. Its fifteen tasks follow clinical stages and were checked by a professional radiologist. Testing twenty four leading models, the team finds the best proprietary model scores far below physicians. All models struggle with complex reasoning before surgery, exposing a gap between seeing an image and clinical judgment.
Catalog record
| CREATOR | Zhihao Peng, Cheng Wang, Shengyuan Liu, Zhiying Liang, Yixuan Yuan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-00846
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks Zhihao Peng, Cheng Wang, Shengyuan Liu, Zhiying Liang, Yixuan Yuan Abstract Brain imaging analysis is vital for diagnosing and treating brain disorders, and multimodal large language models (MLLMs) increasingly assisting in that analysis. However, current brain-oriented visual question-answering (VQA) benchmarks either cover a few imaging modalities or are limited to coarse-grained pathological descriptions, hindering a comprehensive assessment of MLLMs throughout the full clinical continuum. To address these, we introduce OmniBrainBench, the first comprehensive multimodal VQA benchmark specifically designed to assess the multimodal comprehension capabilities of MLLMs in brain imaging analysis. OmniBrainBench consists of 15 distinct brain imaging modalities collected from 30 verified medical sources, yielding 9,527 validated VQA pairs and 31,706 images. It simulates clinical workflows and encompasses 15 multi-stage clinical tasks rigorously validated by a professional radiologist. Evaluation of 24 state-of-the-art models, including open-source, medical and proprietary MLLMs, highlights the substantial challenges posed by OmniBrainBench. Our experiments reveal: (1) proprietary MLLMs (e.g., GPT-5) beat open-source and medical models but lag physicians; (2) medical MLLMs vary widely in performance; (3) open-source MLLMs trail overall but excel in specific tasks; (4) MLLMs underperform sharply in complex preoperative tasks, revealing a visual-to-clinical reasoning gap. OmniBrainBench sets a new standard for evaluating and advancing MLLMs in brain imaging analysis, highlighting gaps compared to expert clinical reasoning. We release it at benchmark & code . 1 Introduction Brain imaging analysis has become a cornerstone of modern diagnostic and therapeutic decision-making by visualizing structural and functional abnormalities Abulnaga et al. 2025 ; Shin et al. 2025 , detecting early pathological changes Zhang et al. 2025 ; Huang and Shu 2025 , and supporting longitudinal monitoring of neurological diseases Lachinov et al. 2023 ; Gutbrod et al. 2025 . In routine practice, traditional brain imaging analysis heavily relies on the radiologist’s subjective expertise, causing variability and delays. Recently, multimodal large language models (MLLMs) Team 2025 ; Guo et al. 2025 ; xAI 2025 ; OpenAI 2025 ; Anthropic 2025 ; Comanici et al. 2025 have demonstrated attractive promise in multimodal perception, contextual understanding, and cross-modal reasoning, significantly impacting brain imaging analysis across diverse modalities. However, applying these models to brain imaging diagnosis presents domain-specific challenges. For instance, the scarcity of brain-specific expertise and the necessity to account for clinical-specific anatomical variations pose significant hurdles Liu et al. 2025a ; Johnson et al. 2019 ; Irvin et al. 2019 . A subsequent natural question is how to design a specialized brain imaging benchmark that aligns with clinical multi-stage analysis workflows to evaluate the comprehension capability of MLLMs in brain imaging analysis Shehzad et al. 2025 ; Bercea et al. 2025 ; Hamada 2025 . A central challenge in evaluating MLLMs is the limited modality coverage in existing brain imaging benchmarks Lau et al. 2018 ; Liu et al. 2021 ; Ray et al. 2024 ; Seenivasan et al. 2023 ; Bai et al. 2024 ; Johnson et al. 2019 ; Kucs et al. 2024 ; Wardlaw et al. 2019 . Most benchmarks emphasize a small set of radiological modalities and fail to fully cover the commonly used spectrum of structural, functional, and molecular neuroimaging Thomalla et al. 2018 . For example, Brain Tumor VQA is restricted to structural magnetic resonance imaging (sMRI) volumes, omitting functional series such as functional MRI (fMRI), diffusion imaging, and molecular techniques like positron emission tomography (PET) Shehzad et al. 2025 . Likewise, NOVA focuses on anatomical brain MRI and offers no coverage of nuclear medicine or other metabolically oriented modalities Ray et al. 2024 . In practice, clinical reality instead demands modality diversity Mahler et al. 2020 ; Tang et al. 2024 : stroke evaluation typically begins with non-contrast CT to rule out hemorrhage and adds diffusion-weighted imaging (DWI), susceptibility-weighted imaging (SWI), and fluid-attenuated inversion recovery (FLAIR) to map the damaged tissue and adjacent swelling Wardlaw et al. 2019 ; Parkinson’s disease management frequently integrates structural MRI sequences, diffusion MRI, resting-state fMRI, and dopaminergic imaging via SPECT or PET to track brain degeneration and functional network changes El Zahraa El Said et al. 2023 ; and Alzheimer’s disease assessment relies on T1-weighted MRI (T1W) together with PET to capture both structural atrophy and protein deposition Jack et al. 2024 ; Liu et al. 2025b . There is a clear gap between the design of existing benchmarks and clinical practice in neuroimaging, underscoring the need for a comprehensive evaluation framework that supports truly multimodal assessment. Another challenge is that existing brain imaging benchmarks focus on isolated tasks rather than the full clinical continuum for patients. A complete clinical workflow typically starts with basic identification of anatomical structures, moves on to disease detection and precise lesion localization, proceeds to formulation of a treatment plan, and prognostic assessment. However, current benchmarks tend to cover only a subset of these stages. For example, the Brain Tumor MRI benchmark is limited to questions about tumor type and other basic attributes, while leaving out other tumor-related objectives such as prognosis estimation and treatment planning Shehzad et al. 2025 . VQA-RAD emphasizes VQA about basic findings without requiring integrated evaluation across localization, diagnosis, and prognosis Lau et al. 2018 . NOVA concentrates on lesion localization in brain MRI without linking results to outcome prediction and management Ray et al. 2024 . In clinical practice, end-to-end competence is necessary, e.g., assessment in high-grade gliomas is standardized by criteria that consider tumor components, clinical status, and medication use, so that radiographic changes can be interpreted reliably for trial endpoints and patient management Wen et al. 2010 . Presurgical evaluation of drug-resistant epilepsy leverages multimodal imaging and analytic algorithms to identify the brain regions where seizures originate, assess surgical risk, and plan interventions Antel et al. 2024 . These examples show that evaluating MLLMs on only a subset of tasks underestimates the broad skills needed for comprehensive brain imaging analysis. Therefore, we need a benchmark that aligns with the full range of clinical needs and can verify how models will perform across real-world tasks. To address these challenges, we introduce OmniBrainBench , the comprehensive multimodal benchmark for evaluating MLLMs in brain imaging analysis, as shown in Fig. 1 . To the best of our knowledge, OmniBrainBenchis the most extensive multimodal brain imaging benchmark to date, drawn from 30 rigorously validated sources, yielding 259,628 VQA pairs comprising our OmniBrainVQAdataset, the current largest brain imaging instruction-tuning collection. From OmniBrainVQA, we extract representative pairs that undergo rigorous clinical review, resulting in our final OmniBrainBenchof 9,527 clinically verified VQA pairs and 31,706 images, as detailed in Fig. 2 . To further distinguish the difference between OmniBrainBench and other existing ones, we elaborate the benchmark details as detailed in Table 1 . For clinical modality coverage , the benchmark provides 15 imaging modalities , including the coarse-grained ones include CT, MRI, PET, SPECT, anatomical diagram (ADiag), histopathology imaging (HI) modalities, and the fine-grained ones include DWI, SWI, FLAIR, T1W, T1-weighted contrast-enhanced (T1CE), T2-weighted imaging (T2W), magnetic resonance angiography (MRA), proton density weighted imaging (PD), fMRI. With its extensive scope and multi-dimensional evaluation criteria, OmniBrainBench is positioned to comprehensively assess the effectiveness of MLLMs in diverse brain imaging modality data. For clinical tasks grouping ,