RadOnc-GPT: An Autonomous LLM Agent for Real-Time Patient Outcomes Labeling at Scale
Jason Holmes, Yuexing Hao, Mariana Borras-Osorio · 2025
Read RadOnc-GPT: An Autonomous LLM Agent for Real-Time Patient Outcomes Labeling at Scale by Jason Holmes, Yuexing Hao, Mariana Borras-Osorio (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25540.txt.
Introduction
Jason Holmes and co-authors presented RadOnc-GPT in 2025, an AI agent that labels patient outcomes for radiation oncology research. Hand labeling is slow and caps how much outcomes work can scale. Their agent pulls a patient's data on its own, weighs the proof in loops, and returns tidy labels. The team grade it in two tiers of rising hardness.
The first tier checks plain retrieval: ages, doses, and plan details from structured records. The second tier asks for true judgment, such as spotting jaw bone death after radiation, or cancer's return, in prostate and head-and-neck groups. Those calls need structured and free-text data read together. The first tier builds trust for the second. The authors pitch the agent as a way to run outcomes studies at a scale hand work cannot reach.
Catalog record
| CREATOR | Jason Holmes, Yuexing Hao, Mariana Borras-Osorio |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-25540
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RadOnc-GPT: An Autonomous LLM Agent for Real-Time Patient Outcomes Labeling at Scale Jason Holmes, Yuexing Hao, Mariana Borras-Osorio Abstract Purpose: Manual labeling limits the scale, accuracy, and timeliness of patient outcomes research in radiation oncology. We present RadOnc-GPT , an autonomous large language model (LLM)–based agent capable of independently retrieving patient-specific information, iteratively assessing evidence, and returning structured outcomes. Our evaluation explicitly validates RadOnc-GPT across two clearly defined tiers of increasing complexity: (1) a structured quality assurance (QA) tier, assessing the accurate retrieval of demographic and radiotherapy treatment plan details, followed by (2) a complex clinical outcomes labeling tier involving determination of mandibular osteoradionecrosis (ORN) in head-and-neck (H&N) cancer patients and detection of cancer recurrence in independent prostate and head-and-neck cancer cohorts requiring combined interpretation of structured and unstructured patient data. The QA tier establishes foundational trust in structured-data retrieval, a critical prerequisite for successful complex clinical outcome labeling. Methods: A wrapper program called LLM Task Streaming sequentially supplied patient IDs and specific task instructions to RadOnc-GPT. Initially, RadOnc-GPT completed the structured QA task using demographic and radiotherapy treatment plan course data exclusively. In the second tier, RadOnc-GPT autonomously combined structured electrical health record (EHR) data with unstructured clinical notes, radiology, and pathology reports to determine clinical outcomes. Ground-truth labels for ORN and cancer recurrence were retrospectively generated by expert radiation oncologists. Discrepancies between RadOnc-GPT’s outputs and ground-truth labels underwent independent adjudication by independent radiation oncologists, classifying each discrepancy into one of three categories: model error, ground-truth error, or indeterminate. Results: Across 895 unique patients, RadOnc-GPT matched all six demographic fields for all 500 patients in the QA tier (100%) and accurately reproduced radiation-course counts in 497/500 (99.4%), demonstrating robust retrieval fidelity. In the complex patient outcomes labeling tier, initial accuracy improved significantly post-adjudication: for ORN determination (233 patients) accuracy rose from 84.5% to 95.2%, prostate cancer recurrence detection (80 patients) improved from 92.5% to 95.0%, and H&N recurrence detection (82 patients) improved from 92.7% to 96.3%. Among 48 initial discrepancies, adjudication revealed 30 (63%) previously unrecognized ground-truth errors, 13 genuine model errors, and 5 indeterminate cases. Conclusions: RadOnc-GPT reliably retrieves foundational structured data and subsequently generalizes complex clinical outcome labeling tasks effectively, notably using a single cancer recurrence detection prompt across multiple disease sites. Its high recall performance minimizes clinically critical false negatives, while its dual capacity as labeler and auditor significantly enhances registry data integrity. Thus, RadOnc-GPT enables scalable, trustworthy, and real-time curation of radiation-oncology research datasets. 1 Introduction Large-language models (LLMs) have progressed rapidly from general-purpose chatbots to highly specialized systems capable of matching or surpassing human experts across multiple clinical tasks [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 ] . Specifically, in radiation oncology GPT-4 has attained expert-level performance on radiation-oncology physics exams [ 11 , 12 ] , broad clinical knowledge assessments [ 13 , 14 , 15 ] , clinical task management such as in-basket response [ 16 ] , automated drafting of payer-appeal documents [ 17 ] , and harmonizing naming conventions of anatomical structures [ 18 ] . In addition, recently open source LLMs have been fine-tuned to improve the performance of important clinical tasks in radiation oncology [ 19 ] . Further improvements have been demonstrated through retrieval-augmented generation (RAG) approaches, which significantly enhance accuracy in clinical trial screening applications [ 20 , 21 , 22 , 23 , 24 ] . Parallel studies in oncology-specific natural language processing (NLP) have demonstrated the successful extraction of detailed clinical information such as pathology findings [ 25 ] , computed tomography (CT) imaging phenotypes [ 26 ] , key entities from discharge letters [ 27 ] , toxicity assessments [ 28 ] , patient comorbidities [ 29 ] , outcomes from lung-ablation procedures [ 30 ] , and patient-reported outcomes [ 31 ] . Recent multi-modal models that integrate imaging and textual data have further expanded capabilities to tasks like tumor contouring [ 32 ] and survival prediction [ 33 ] . However, a significant limitation of most existing NLP approaches and registry curation efforts is their inherent assumption that institutional registry labels represent reliable gold standards. Recent audits, such as a multi-year review of an academic head-and-neck cancer registry, suggest otherwise, revealing significant latent errors, with up to half of true recurrence events potentially missing from automated flags [ 34 ] . Correcting these errors via traditional manual review would require substantial staffing resources, highlighting the urgent need for more efficient, scalable methods for auditing and curating clinical outcomes data. To address these critical limitations, we developed RadOnc-GPT , a GPT-4o–based autonomous agent capable of retrieving structured and unstructured clinical data directly from institutional databases, iteratively assessing the evidence, and synthesizing structured clinical outcomes. In order to evaluate RadOnc-GPT, we designed a two-tier strategy differing by their level of complexity: first, a relatively simple structured data retrieval task aimed at establishing the agent’s structured information retrieval capability. Because this task involves reproducing structured data, manual review is not required and therefore makes this evaluation step a good candidate for quality assurance purposes. After demonstrating reliable structured-data reproduction, we increase the complexity of the tasks, evaluating RadOnc-GPT’s ability to mix structured and unstructured patient data towards performing the task. These high-complexity tasks included determination of mandibular osteoradionecrosis (ORN) in head-and-neck cancer patients and cancer recurrence detection in separate prostate and head-and-neck cancer patient cohorts. A noteworthy design feature of our evaluation was using an identical cancer recurrence detection method (identical input prompt) across two disease sites—prostate cancer and head-and-neck cancer—testing RadOnc-GPT’s ability to generalize clinical reasoning across disease sites. Through this staged evaluation approach, we aim to answer three key questions: 1. Can an autonomous LLM-based agent reliably retrieve and interpret structured clinical data from institutional databases? 2. Once baseline structured retrieval accuracy is confirmed, can the agent successfully perform complex clinical outcomes labeling using both structured and unstructured data? 3. Can such an agent simultaneously identify latent errors in existing institutional registry labels, thus serving a dual function of patient outcome labeling and real-time data auditing? Answering these questions clearly and rigorously establishes autonomous LLM agents as a potentially transformative solution for near-real-time, scalable and accurate clinical outcomes labeling. 2 Methods 2.1 RadOnc-GPT System Design Figure 1 illustrates the architecture of RadOnc-GPT, which integrates both internal and external data resources. Internal connections include Mayo Clinic’s radiation oncology–specific database system (Minnesota, Arizona, Florida), Aria (Varian Medical Systems, Palo Alto, CA), and enterprise electronic health record (EHR) database systems interfacing with Epic (Epic Systems, Verona, WI). RadOnc-GPT further connects with public data resources including PubMed, ClinicalTrials.gov, and the National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE), each connected via their public APIs. RadOnc-GPT is designed as an LLM-based agent, defined here as a system capable of conducting multi-turn conversations while autonomously determining which functions to call and when to stop responding. The RadOnc-GPT system makes use of the many data sources by calling upon whitelisted functions that retrieve, process, and return data to GPT-4o (OpenAI) or Gemini 2.5 Pro (Google). The specifications of these functions—including their inputs and outputs—are explicitly detailed within the system prompt so as to