Intelligent Documentation in Medical Education: Can AI Replace Manual Case Logging?
Nafiz Imtiaz Khan, MSc, Kylie Cleland, BSc, Vladimir Filkov, PhD, Roger Eric Goldman, PhD · 2026
Read Intelligent Documentation in Medical Education: Can AI Replace Manual Case Logging? by Nafiz Imtiaz Khan, MSc, Kylie Cleland, BSc, Vladimir Filkov, PhD, Roger Eric Goldman, PhD (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12648.txt.
Introduction
Nafiz Imtiaz Khan, Kylie Cleland, Vladimir Filkov, and Roger Eric Goldman ask whether AI can log training cases for doctors. Radiology trainees must log each procedure case they perform, by hand. Manual logging eats time and invites errors and gaps. The team test whether large language models can automate that documentation.
They evaluate how well models extract case details for the log. They also identify which procedure types are hardest to extract. A final question is how such a tool would fit clinical workflows. If AI logging proves reliable, trainees could spend more time learning. If certain case types fail often, those need human checks. The study weighs feasibility before any replacement of manual work.
Catalog record
| CREATOR | Nafiz Imtiaz Khan, MSc, Kylie Cleland, BSc, Vladimir Filkov, PhD, Roger Eric Goldman, PhD |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 159 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-12648
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Intelligent Documentation in Medical Education: Can AI Replace Manual Case Logging? Nafiz Imtiaz Khan, MSc, Kylie Cleland, BSc, Vladimir Filkov, PhD, Roger Eric Goldman, PhD Abstract Objective: This study investigates the feasibility of using large language models (LLMs) to automate procedural case log documentation in radiology training. We evaluate whether AI can replace manual logging, identify which procedure types are most challenging for extraction, and assess integration into clinical workflows. Materials and Methods: We retrospectively analyzed 414 curated radiology reports authored by nine interventional radiology residents between 2018 and 2024. A set of candidate models, including local (Qwen-2.5) and commercial (Claude-3.5), were tested under instruction and chain-of-thought prompting. Performance was measured by sensitivity, specificity, and F1-score, along with inference time and token efficiency to estimate operational cost. Results: Both local and commercial LLMs outperformed the standard benchmark. Qwen-2.5 achieved F1-scores of 86.66 with chain-of-thought prompting, while Claude-3.5-Haiku reached an F1-score of 86.89%. Commercial inference delivered sub-2s latency and concise outputs, while local deployment traded speed for lower recurring cost. Automation could save over 35 hours of manual annotation per resident annually. Discussion: LLMs can provide a scalable and accurate solution for radiology case log documentation. Optimizing for procedure-specific challenges and ensuring seamless integration with existing systems will be essential. Future work should validate across larger, multi-institution datasets and explore additional prompting strategies. Conclusion: LLMs show promise for automating radiology case log documentation, potentially reducing resident clerical burden. However, this single-institution feasibility study underscores the need for broader validation across diverse institutions, assessment of real-world workflow integration, and safeguards against misclassification before clinical adoption. Keywords: artificial intelligence, large language models, radiology, case logs, medical education Corresponding Author: Nafiz Imtiaz Khan ( [email protected] ) 1 Introduction Accurate and comprehensive clinical activity documentation is paramount for medical education and professional practice [ 1 ] . Among the most critical artifacts in this process are physician case logs, which serve as structured records that detail the types and frequencies of procedures performed by individual clinicians. These logs are integral to multiple high-stakes functions. They provide evidence for board certification, serve as official documentation for credentialing and privileging within hospitals, and support quality assurance and performance evaluations [ 2 , 3 , 4 , 5 ] . During medical training, case logs are especially vital, enabling program directors to monitor resident progress, ensure exposure to a sufficient variety and volume of procedures, and identify areas needing remediation. Importantly, these logs also underpin institutional compliance with accreditation requirements, such as those mandated by the Accreditation Council for Graduate Medical Education (ACGME) 1 1 1 https://www.acgme.org/ . However, the current reality of case log documentation in many specialties, particularly radiology, is that the process remains highly manual, fragmented, and inefficient. Radiology trainees are typically required to self-report the procedures they perform by entering data into spreadsheets or institutional logging systems [ 6 ] . Each entry often includes detailed metadata, such as the type of imaging modality, the anatomical site, the nature of the procedure, and whether it was performed independently or under supervision. According to Cox et al. [ 7 ] , general residents spend a median of 23.7 hours per week—nearly 38% of their duty hours—engaged with the electronic health record (EHR). While this reflects overall documentation burden rather than case log entry specifically, it highlights how clerical tasks consume significant trainee time. Given that a radiology resident may perform well over 1600 (typically 1500 to 1800) procedures throughout their residency, the cumulative documentation burden is substantial. This manual process consumes valuable time that could otherwise be devoted to clinical learning or scholarly activity. In addition, manual effort introduces considerable potential for errors and inconsistencies in the documentation. Moreover, the absence of real-time feedback mechanisms limits the utility of these logs as tools for formative assessment. In an era where natural language processing (NLP) and machine learning (ML) technologies—particularly large language models (LLMs)—have made significant advancements [ 8 , 9 , 10 ] , there exists a compelling opportunity to reimagine how procedural case logs are created [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ] . Radiology reports, routinely authored by radiology residents and attendings, provide comprehensive narrative summaries of diagnostic imaging encounters and are inherently rich in procedural detail. These reports are routinely produced as part of standard clinical documentation and are stored in digital or semi-structured formats within Electronic Health Records (EHRs) [ 20 ] , Radiology Information Systems (RIS) [ 21 ] , or Picture Archiving and Communication Systems (PACS) [ 22 ] . Using LLMs to automatically extract structured procedural data from these unstructured narratives could generate accurate procedural case logs with minimal human intervention. Such automation promises not only to alleviate administrative burden but also to enhance the accuracy, completeness, and timeliness of documentation. At the same time, prior experience with clinical AI tools demonstrates the risks of overclaiming performance without rigorous validation [ 23 , 24 ] , underscoring the importance of cautious evaluation in this domain. While the theoretical potential of LLMs in this space is enticing, several important objectives motivate this study. First, we assess whether AI can effectively automate procedural case log documentation in radiology training, examining the feasibility and performance of using LLMs to convert narrative radiology reports into structured, loggable entries. Second, we investigate whether performance varies across procedure categories, identifying which categories are difficult to classify, and explored the linguistic or structural reasons behind these disparities. Finally, we examine the translational challenges of real-world deployment, including cost, and processing time. In sum, this is the first work that explores the intersection of artificial intelligence and procedural documentation in medical education by systematically evaluating the feasibility of automation, the variation in accuracy across procedure categories, and the practical considerations for integration into clinical and educational workflows. Ultimately, we envision a future in which intelligent systems reduce administrative burden, improve documentation quality, and allow medical trainees to devote more attention to patient care and professional development. 2 Methodology This retrospective study was approved by the institutional review board. The requirement of informed consent for inclusion in the study was waived. No protected health information is included in the reported results. 2.1 Data Collection We retrospectively collected a dataset of raw radiology reports from a tertiary-care academic medical center, spanning the period from October 2018 through September 2024. These reports were authored by one of nine current or former Interventional Radiology (IR) residents. From this larger pool (36,659 reports across 301 unique exam codes, including 162 distinct procedural codes), we curated a subset of 414 reports corresponding to 39 representative IR procedures. For each procedural exam code, at least one corresponding radiology report was selected for manual labeling. If a code was associated with two reports, both were labeled. For codes with three or more associated reports, a random sample of three reports was selected for labeling. This ensured inclusion across the full range of required categories, but we acknowledge the small sample size per category as a limitation. 2.2 Data Annotation Following data collection, we conducted a structured annotation process to determine which unique radiology procedures were mentioned in each report. This task involved