Performance of GPT-5 in Brain Tumor MRI Reasoning

Mojtaba Safari, Shansong Wang, Mingzhe Hu, Zach Eidex, Qiang Li, Xiaofeng Yang · 2025

Read Performance of GPT-5 in Brain Tumor MRI Reasoning by Mojtaba Safari, Shansong Wang, Mingzhe Hu, Zach Eidex, Qiang Li, Xiaofeng Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10865.txt.

Introduction

Mojtaba Safari and five co-authors test how well new chat models read brain scans in this 2025 arXiv paper. Sorting brain tumors on MRI scans guides how doctors plan care, so the skill matters. The team built a question set from three known tumor data sets, covering glioblastoma, meningioma, and spread from cancers elsewhere in the body.

Each case joined scan images with plain clinical facts, phrased as set questions. Four models, from GPT-4o to the GPT-5 family, answered with no prior training on the task. The mid-size GPT-5-mini scored best overall, at about forty-four percent right. No model led on every tumor type, and none came close to a level safe for clinic use. The study gives a sober measure of what these models can and cannot do with scans.

Catalog record

CREATORMojtaba Safari, Shansong Wang, Mingzhe Hu, Zach Eidex, Qiang Li, Xiaofeng Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-10865

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Performance of GPT-5 in Brain Tumor MRI Reasoning Mojtaba Safari, Shansong Wang, Mingzhe Hu, Zach Eidex, Qiang Li, Xiaofeng Yang Abstract Accurate differentiation of brain tumor types on magnetic resonance imaging (MRI) is critical for guiding treatment planning in neuro-oncology. Recent advances in large language models (LLMs) have enabled visual question answering (VQA) approaches that integrate image interpretation with natural language reasoning. In this study, we evaluated GPT-4o, GPT-5-nano, GPT-5-mini, and GPT-5 on a curated brain tumor VQA benchmark derived from 3 Brain Tumor Segmentation (BraTS) datasets - glioblastoma (GLI), meningioma (MEN), and brain metastases (MET). Each case included multi-sequence MRI triplanar mosaics and structured clinical features transformed into standardized VQA items. Models were assessed in a zero-shot chain-of-thought setting for accuracy on both visual and reasoning tasks. Results showed that GPT-5-mini achieved the highest macro-average accuracy (44.19%), followed by GPT-5 (43.71%), GPT-4o (41.49%), and GPT-5-nano (35.85%). Performance varied by tumor subtype, with no single model dominating across all cohorts. These findings suggest that GPT-5 family models can achieve moderate accuracy in structured neuro-oncological VQA tasks, but not at a level acceptable for clinical use. Keywords: Brain tumor MRI, Large language models, Glioblastoma, GPT-5, Meningioma, Metastases 1 Introduction Brain tumors, including glioblastoma multiforme (GBM), meningioma, and brain metastases, represent a diverse group of intracranial neoplasms with distinct radiological appearances, biological behaviors, and prognostic implications mcfaline2018brain . Accurate and timely diagnosis is critical, as early differentiation among these tumor types can significantly influence surgical planning, adjuvant therapy, and patient outcomes 10.1093/neuonc/noab106 . Magnetic resonance imaging (MRI) remains the clinical standard for brain tumor evaluation due to its superior soft tissue contrast and multiparametric capabilities. However, interpretation of MRI scans requires expert neuroradiological knowledge, and diagnostic variability can arise from differences in training, experience, and fatigue laustsen2025interobserver ; crowe2017expertise . As such, the integration of AI, particularly large language and multi-modal models, into the diagnostic workflow has emerged as a promising avenue to enhance efficiency and standardization. The visual question answering (VQA) paradigm in medical imaging aims to answer clinically relevant, natural-language questions about medical images, combining visual understanding with medical reasoning. In neuro-oncology, VQA systems must not only identify radiological features such as contrast enhancement, necrosis, and peritumoral edema, but also contextualize these findings to provide accurate localization, differential diagnoses, and treatment implications. This makes brain tumor VQA a high-complexity task that demands both robust visual grounding and domain-specific reasoning. Recent advances in large language models (LLMs) have accelerated a shift away from narrowly trained, task-specific systems toward frameworks in which LLMs function as central reasoning engines liu2024deepseek ; achiam2023gpt . In clinical applications, relevant information often arises from diverse modalities. For instance, clinical texts and benchmarks assessing medical knowledge in LLMs singhal2023large , structured representations and grounded reporting frameworks guo2023gpt4graphlargelanguagemodels , and imaging data wang2025unifying ; wang2025triad . GPT-4 demonstrated strong general medical reasoning abilities and outperformed prior models across multiple medical benchmarks hu2024advancing ; nori2023capabilitiesgpt4medicalchallenge . However, its performance on brain tumor MRI remained constrained, particularly in fine-grained localization and tumor-type differentiation. The introduction of AutoRG-Brain lei2024autorgbraingroundedreportgeneration highlighted the importance of pixel-level grounding and report generation for brain MRI, providing a structured evaluation framework using BraTS-like datasets that explicitly link visual features to textual outputs. Building on these foundations, GPT-5 and its lightweight variants (GPT-5-mini, GPT-5-nano) have introduced further improvements in multi-modal reasoning wang2025capabilitiesgpt5multimodalmedical * * * https://github.com/wangshansong1/GPT-5-Evaluation † † † https://github.com/TsinghuaC3I/MedXpertQA , including gains in clinical VQA accuracy, reasoning consistency, and domain adaptability over GPT-4o. In this study, we evaluated GPT-4o-2024-11-20, GPT-5, GPT-5-mini, and GPT-5-nano on a curated brain tumor VQA benchmark derived from the BraTS datasets. Each case’s multi-modal MRI sequences, T1-contrast (T1c), T1-weighted (T1w), T2-fluid-attenuated inversion recovery (T2-FLAIR), T2-weighted (T2w) are processed to extract centroid-aligned triplanar mosaics. The associated radiology findings are parsed into structured clinical concepts including lesion location, enhancement pattern, boundary definition, edema, midline shift, maximum dimension, and ventricle compression, which are then converted into standardized yes/no and multiple-choice questions with distractors, answer keys, and linked imaging. This ensures that every model is evaluated on identical, clinically grounded prompts, combining both image and text input, and that performance can be directly compared across models and question types. By aligning the dataset construction with wang2025capabilitiesgpt5multimodalmedical prompting and evaluation protocol, we ensure reproducibility and isolate model-specific differences in multi-modal reasoning on glioblastoma, meningioma, and metastasis cases. 2 Methodology This study used the BraTS dataset, which contains multi-parametric MRI scans of glioblastoma (GLI) menze2014multimodal , meningioma (MEN) labella2023asnr , and brain metastases (MET) moawad2024brain . For each case, four standard MRI sequences were available: T1-contrast enhanced (T1c), T1-weighted (T1w), T2-weighted (T2w), and T2-fluid-attenuated inversion recovery (T2-FLAIR). Tumor segmentation masks provided with the dataset were used to localize lesions, and each case was also paired with radiology-style textual findings generated in the style of AutoRG-Brain or obtained from equivalent clinical reporting sources. All MRI volumes were converted to a standard RAS orientation prior to analysis to ensure spatial consistency across subjects. Tumor centroids were calculated directly from the segmentation masks. From each volume, axial, sagittal, and coronal slices passing through the tumor centroid were extracted. Intensities were normalized using percentile-based clipping (2nd to 98th percentile) to suppress outliers and enhance lesion visibility. The three slices were concatenated to form a single triplanar mosaic image, providing a compact but information-rich visual representation for subsequent evaluation. The accompanying clinical text for each case was processed using a custom parsing pipeline to extract structured descriptors of tumor appearance. This process identified the primary lesion location, enhancement pattern, boundary definition, the presence of peritumoral edema, the presence or absence of midline shift, maximum lesion dimension in millimeters, involvement across the midline, and compression of any ventricular structures. The parser combined rule-based term matching, synonym normalization, and regular expressions for numerical values to ensure consistent extraction across reports. These structured features were then transformed into VQA items following the style protocol described in wang2025capabilitiesgpt5multimodalmedical . Each feature was mapped to a natural-language question designed to assess either visual understanding or clinical reasoning. For example, enhancement features yielded yes/no questions such as “Does the lesion show ring enhancement?”, location features were converted into multiple-choice questions with plausible distractors, and lesion size measurements were framed as numeric multiple-choice questions. Each VQA item included the question text with labeled answer choices, the correct answer, the associated triplanar mosaic, and metadata specifying the medical task, body system, and question type. Four large multi-modal language models were evaluated on the resulting VQA dataset: GPT-4o, GPT-5, GPT-5-mini, and GPT-5-nano. Each model received the triplanar mosaic and the question text with answer choices as a single multi-modal input (see Figure 1 ). The model’s predicted choice was compared with the reference answer, and accuracy was calculated as the proportion of correctly answered questions. Figure 1 : An example BraTS-GLI case containing axial, coronal, and sagittal slices chosen to depict the extent of the tumor volume for (a) T1c, (b) T1w, (c) T2-FLAIR, and (d) T2w MRI. A single sequence was provided to the GPT models for VQA. Zero-shot + CoT for VQA [ { "role": "system", "content": "You are a helpful medical assistant." }, { "role": "user", "content": [ { "type": "text", "text": "Q: {QUESTION_TEXT}\nA: Let’s think step by step."

Read the complete text