Benchmarking GPT-5 in Radiation Oncology: Measurable Gains, but Persistent Need for Expert Oversight
Ugur Dinc 1,2,3∗ , Jibak Sarkar 1,2,3 , Philipp Schubert 1,2,3 , Sabine Semrau 1,2,3 , Thomas Weissmann 1,2,3 , Andre Karius 1,2,3 , Johann Brand 1,2,3 , Bernd-Niklas Axer 1,2,3 , Ahmed Gomaa 1,2,3 , Pluvio Stephan 1,2,3 , Ishita Sheth 1,2,3 , Sogand Beirami 1,2,3 , Annette Schwarz 1,2,3 , Udo Gaipl · 2025
Read Benchmarking GPT-5 in Radiation Oncology: Measurable Gains, but Persistent Need for Expert Oversight by Ugur Dinc 1,2,3∗ , Jibak Sarkar 1,2,3 , Philipp Schubert 1,2,3 , Sabine Semrau 1,2,3 , Thomas Weissmann 1,2,3 , Andre Karius 1,2,3 , Johann Brand 1,2,3 , Bernd-Niklas Axer 1,2,3 , Ahmed Gomaa 1,2,3 , Pluvio Stephan 1,2,3 , Ishita Sheth 1,2,3 , Sogand Beirami 1,2,3 , Annette Schwarz 1,2,3 , Udo Gaipl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21777.txt.
Introduction
The authors benchmark a new language model, GPT-5, on radiation oncology tasks in a study from 2025. The model is marketed for cancer care support, so the team tested it on a 300-item training exam for the field and on sixty real-style patient cases. Four board-certified specialists scored its treatment plans for correctness, completeness, and made-up claims.
The model scored about ninety-three percent on the exam, well above two older models. Its case plans scored high for completeness and fairly for correctness, and clear fabrications were rare. Rater agreement was low, and errors clustered in complex cases. Expert oversight remains needed, the authors say.
Catalog record
| CREATOR | Ugur Dinc 1,2,3∗ , Jibak Sarkar 1,2,3 , Philipp Schubert 1,2,3 , Sabine Semrau 1,2,3 , Thomas Weissmann 1,2,3 , Andre Karius 1,2,3 , Johann Brand 1,2,3 , Bernd-Niklas Axer 1,2,3 , Ahmed Gomaa 1,2,3 , Pluvio Stephan 1,2,3 , Ishita Sheth 1,2,3 , Sogand Beirami 1,2,3 , Annette Schwarz 1,2,3 , Udo Gaipl |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-21777
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking GPT-5 in Radiation Oncology: Measurable Gains, but Persistent Need for Expert Oversight Ugur Dinc 1,2,3∗ , Jibak Sarkar 1,2,3 , Philipp Schubert 1,2,3 , Sabine Semrau 1,2,3 , Thomas Weissmann 1,2,3 , Andre Karius 1,2,3 , Johann Brand 1,2,3 , Bernd-Niklas Axer 1,2,3 , Ahmed Gomaa 1,2,3 , Pluvio Stephan 1,2,3 , Ishita Sheth 1,2,3 , Sogand Beirami 1,2,3 , Annette Schwarz 1,2,3 , Udo Gaipl 1,2,3 , Benjamin Frey 1,2,3 , Christoph Bert 1,2,3 , Stefanie Corradini 4,3 , Rainer Fietkau 1,2,3 and Florian Putz 1,2,3 Abstract 1 Introduction: Large language models (LLM) have shown great potential in clinical decision support and medical education. GPT-5 is a novel LLM system that has been specifically marketed towards oncology use. This study comprehensively benchmarks GPT-5 for the field of radiation oncology. Methods: Performance was assessed using two complementary benchmarks: (i) the American College of Radiology Radiation Oncology In-Training Examination (TXIT, 2021), comprising 300 multiple-choice items, and (ii) a curated set of 60 authentic radiation oncologic vignettes representing diverse disease sites and treatment indications. For the vignette evaluation, GPT-5 was instructed to generate structured therapeutic plans and concise two-line summaries. Four board-certified radiation oncologists independently rated outputs for correctness, comprehensiveness, and hallucinations. Inter-rater reliability was quantified using Fleiss’ κ \kappa . GPT-5 results were compared to published GPT-3.5 and GPT-4 baselines. Results: On the TXIT benchmark, GPT-5 achieved a mean accuracy of 92.8%, outperforming GPT-4 (78.8%) and GPT-3.5 (62.1%). Domain-specific gains were most pronounced in dose specification and diagnosis. In the vignette evaluation, GPT-5’s treatment recommendations were rated highly for correctness (mean 3.24/4, 95% CI: 3.11–3.38) and comprehensiveness (3.59/4, 95% CI: 3.49–3.69). Hallucinations were rare (mean 10.0%), and no case reached majority consensus for their presence. Inter-rater agreement was low (Fleiss’ κ \kappa 0.083 for correctness), reflecting inherent variability in clinical judgment. Errors clustered in complex scenarios requiring precise trial knowledge or detailed clinical adaptation. Discussion: GPT-5 clearly outperformed prior model variants on the radiation oncology multiple-choice benchmark. Although GPT-5 exhibited favorable performance in generating real-world radiation oncology treatment recommendations, correctness ratings indicate room for further improvement. While hallucinations were infrequent, the presence of substantive errors underscores that GPT-5-generated recommendations require rigorous expert oversight before clinical implementation. Keywords: GPT-5, Artificial Intelligence, Radiation Oncology, Large Language Models, Treatment Recommendation, Oncologic Decision Support, Hallucination, Real-World Evaluation \correspondance \extraAuth † † firstpage: 1 2 Introduction Large language models (LLMs) have advanced rapidly in recent years, driven by scaling of parameters ( 1 ) , reinforcement-learning–based alignment ( 2 , 3 ) , and the development of modular architectures such as Mixture-of-Experts (MoE) ( 4 , 5 ) . These innovations have enabled broad use of LLMs across scientific and clinical domains ( 6 , 7 , 8 ) . In biomedicine, domain-specific pretraining and clinical fine-tuning have enhanced representation of medical terminology and workflows ( 9 , 10 , 11 ) , while general-purpose models have achieved exam-level performance in several evaluations ( 12 , 13 , 14 ) . Nonetheless, accuracy remains heterogeneous across specialties and problem types ( 15 , 16 , 17 , 18 ) . Current consensus emphasizes transparent communication of model limitations and the need for sustained human oversight ( 17 , 18 ) . Within radiation oncology, deep learning methods are established for tasks such as segmentation, image enhancement, dose estimation, and outcome prediction ( 19 , 20 , 21 , 22 , 23 , 24 , 25 ) . LLMs extend this toolkit with text-centric applications such as guideline summarization, structured rationale generation, question answering, and automated documentation ( 26 , 27 , 28 ) . Evaluation of such models now include physics-focused question sets ( 29 , 30 ) as well as surgical and board-style assessments ( 31 , 32 ) . Radiation oncology-specific studies underscore both the promise of LLMs and the persistence of domain-specific limitations (e.g., dose prescription, differentiation between percutaneous and interstitial techniques) as well as challenges in keeping pace with evolving trial evidence ( 16 , 17 ) . A key area of ongoing research is the use of large language models (LLMs) as clinical decision support (CDS) systems, with prominent initiatives including Med-PaLM, Med-PaLM 2, and Google AMIE ( 33 , 13 , 34 ) . LLM agents like AMIE illustrate how LLM-based assistants may retrieve, synthesize, and contextualize medical evidence for patient-specific recommendations under expert supervision ( 33 ) . Early evaluations report promising accuracy in case-based reasoning and treatment planning, while underscoring the necessity of explicit uncertainty handling and clinician oversight ( 16 , 18 , 35 ) . GPT-5, the latest generation of OpenAI’s foundation models, represents a fundamental shift compared to GPT-3.5 and GPT-4 by explicitly incorporating reasoning-focused reinforcement learning reward models ( 36 ) . In combination with a larger MoE backbone and improved calibration of probabilistic outputs, GPT-5 achieves stronger logical consistency, longer-context reasoning, and higher factual accuracy. Importantly, GPT-5 is a major OpenAI model explicitly positioned as a reasoning model, designed to generate structured, interpretable rationales in addition to predictions. These advances have translated into improved performance across biomedical benchmarks, USMLE-style exams, radiology case reasoning, and oncology-specific tasks, while also reducing hallucination rates ( 37 , 38 , 17 , 18 ) . Despite these improvements, supervised use remains essential, particularly in high-stakes oncology settings. Building on this progress, the present work provides the first comprehensive evaluation of GPT-5 in radiation oncology. We investigate two complementary settings: (i) a benchmark against the American College of Radiology Radiation Oncology In-Training Examination (ACR TXIT) subset using an automated Responses API pipeline directly comparable to GPT-3.5/4 results, and (ii) a novel real-world scenario dataset comprising 60 complex clinical cases without a single established standard of care ( 39 ) . By jointly analyzing standardized benchmarking and novel scenario-based evaluation, we assess GPT-5’s accuracy, comprehensiveness and hallucination frequency. This dual design allows rigorous quantification of performance while also examining GPT-5’s practical usability and failure modes in clinically ambiguous situations. Given the explicit positioning of GPT-5 as a reasoning model for scientific and medical tasks, our study provides a timely and domain-specific benchmark of its potential and limitations in radiation oncology. 3 Materials and methods This study comprised two complementary evaluations of GPT-5 in radiation oncology: (i) performance on a standardized multiple-choice knowledge benchmark, and (ii) structured decision-support recommendations on real-world oncologic case vignettes. All analyses were performed on de-identified data, using isolated sessions without cross-case information transfer. No protected health information was processed. 3.1 GPT-5 model and prompting framework The large language model (LLM) under test corresponds to the GPT-5 family as characterized in the publicly released system card ( 37 ) . GPT-5 is a transformer-based model trained on large text corpora with mixture-of-experts routing and reinforcement learning from human and artifical intelligence (AI) feedback. For reproducibility, standardized instructions were used for all experiments, and prompts/outputs were logged. Each API call was executed in a fresh session to avoid context leakage between cases. Automation was implemented in Python. 3.2 ACR Radiation Oncology In-Training Examination benchmark Knowledge-based performance was assessed using the 2021 American College of Radiology (ACR) Radiation Oncology In-Training Examination (TXIT) ( 40 , 39 ) , which comprises 300 multiple-choice questions spanning statistics, physics, biology,