Efficacy of large language models in detecting postoperative delirium from unstructured clinical notes: A retrospective cohort study Authors: Da-In Eun, Hyung-Chul Lee, Gang Heo, Jae Hoon Jeong, Jipyeong Lee, Hyeonhoon Lee, Sang-Youn Park, Karam Nam, Ho-Jin Lee, Jae-Woo Ju Published: 2025 ABSTRACT AbstractEarly identification of postoperative delirium (POD) remains challenging. This retrospective observational study compared the performance of large language models (LLMs), Llama-3-70B and GPT-4o, and physicians in predicting clinically significant POD, defined as either requiring antipsychotics or diagnosis of delirium by neurologists following consultation for delirium-related symptoms. The c-statistics of Llama-3-70B and GPT-4o were 0.74 and 0.76, respectively. LLMs showed higher sensitivity (Llama-3-70B, 0.900; GPT-4o, 0.868; physicians, 0.723) and lower specificity (0.463, 0.547, and 0.814, respectively) than physicians. Inter-rater agreement was almost perfect for both Llama-3-70B and GPT-4o (Fleiss’ kappa = 0.852 and 0.854, respectively) but fair for physicians (0.219). Both LLMs detected clinically significant POD approximately one day earlier than physicians (Kaplan-Meier analysis, median time to diagnosis: Llama-3-70B, 34.5 h; GPT-4o, 37.5 h; physicians, 62.9 h; log-rank P < 0.001). The integration of LLMs as a complementary screening tool under physician supervision may improve the early, reproducible diagnosis of clinically significant POD.Subject terms: Medical research, Outcomes research Introduction Postoperative delirium (POD) is a common complication affecting approximately 5–52% of surgical patients1,2. POD is associated with prolonged hospitalization, intensive care unit (ICU) readmission, and mortality3–5. Additionally, prolonged duration of delirium is associated with increased risk of mortality and long-term cognitive decline6,7. Therefore, early detection and appropriate intervention of POD are critical for improving postoperative outcomes. The established standard for screening delirium in the general ward includes tools such as the Confusion Assessment Method (CAM), 4 A’s Test, and the Nursing Delirium Screening Scale8. However, in clinical practice, these tools are often underutilized and are frequently overlooked during the acute onset of delirium. Instead, abnormal behaviors are recorded as free text in clinical notes, leaving other healthcare providers to inevitably assess delirium retrospectively based on these records. Inouye and colleagues introduced a chart-based method to retrospectively assess delirium, which screens clinical notes for delirium-related keywords, followed by a manual review by clinicians9. This method showed an overall agreement of 82% with CAM7 and has been validated by multiple studies10–12. However, manual chart reviews by clinicians are labor-intensive and prone to inter-rater variability, thereby making it difficult to be a routine screening tool in clinical practice. Natural language processing (NLP) and large language models (LLMs) have recently emerged as powerful tools for analyzing clinical notes and predicting various medical conditions13–15. However, studies directly applying these methods to unstructured clinical notes are limited16,17. Although several studies have shown that NLP- and LLM-based approaches may be useful in detecting behavioral disturbances and predicting delirium18–20, their applications have been limited to the ICU setting, leaving the broader spectrum of cognitive and behavioral changes in surgical patients documented in the clinical notes largely unexplored. Therefore, this study presents the first application of LLMs to comprehensively analyze large numbers of unstructured clinical notes from surgical patients to detect POD, with the aim of comparing their performance with physician assessment. Additionally, we sought to compare the inter-rater agreement and the time to detect delirium between LLMs and physicians. Results Patient characteristics and data distribution During the study period, 35,421 patients met the inclusion criteria. After excluding 4117 patients based on the exclusion criteria, 31,304 patients were included in the analysis (Fig. 1). Baseline characteristics and perioperative data are summarized in Table S1. A total of 2,177,729 clinical notes containing 50,248,840 words documented within seven postoperative days were reviewed. After screening the clinical notes containing delirium-related keywords, 7168 records from 1685 patients were identified. Among these, 22.9% (386/1685) were diagnosed with clinically significant POD. Study flowchart. Delirium assessment and inter-rater agreement analysis Among the clinical notes containing delirium-related keywords, physicians identified 20.1% (1437/7168) as delirium-positive. Complete agreement among all three physicians was observed in 3638 (50.8%) cases, of which 744 (10.4%) were delirium-positive and 2894 (40.4%) were delirium-negative (Fig. S1). The overall inter-rater agreement among the three physicians was fair (Fleiss’ kappa = 0.219). The proportion of records identified as delirium-positive by Llama-3-70B and GPT-4o was 37.1% (2656/7168) and 33.9% (2431/7168), respectively. In terms of consensus across 20 independent assessments, Llama-3-70B demonstrated complete agreement in 77.4% (5551/7168) of cases, with 26.3% (1888/7168) as delirium-positive and 51.1% (3663/7168) as delirium-negative (Fig. 2). GPT-4o achieved complete agreement in 79.1% (5673/7168) of cases, with 23.5% (1687/7168) being delirium-positive and 55.6% (3986/7168) delirium-negative. Llama-3-70B and GPT-4o achieved a Fleiss’ kappa of 0.852 and 0.854, respectively, both indicating an almost perfect agreement between its predictions. Distribution of delirium-positive assessments at the clinical note level by Llama-3-70B (left panel) and GPT-4o (right panel). Diagnostic performance of LLMs and physicians A comparison of the performance metrics for the detection of clinically significant POD at the individual patient level among physicians, Llama-3-70B, and GPT-4o is summarized in Table 1. Physicians achieved an accuracy of 0.793, whereas the GPT-4o and Llama-3-70B achieved accuracies of 0.621 and 0.563, respectively. Llama-3-70B demonstrated the highest sensitivity (0.900), followed by GPT-4o (0.868), and physicians (0.723). Physicians showed the highest specificity (0.814), followed by GPT-4o (0.547), and Llama-3-70B (0.463). The positive predictive values were 0.536, 0.332, and 0.363; the negative predictive values were 0.908, 0.939, and 0.933; and the F1-scores were 0.615, 0.485, and 0.512 for the physicians, Llama-3-70B, and GPT-4o, respectively. Performance metrics for individual physicians are presented in Table S2. Table 1Performance metrics comparison among physicians, Llama-3-70B, and GPT-4o for detecting clinically significant postoperative deliriumPhysiciansLlama-3-70BGPT-4oAccuracy0.7930.5630.621Sensitivity0.7230.9000.868Specificity0.8140.4630.547PPV0.5360.3320.363NPV0.9080.9390.933F1-Score0.6150.4850.512PPV positive predictive value, NPV negative predictive value. Performance metrics comparison among physicians, Llama-3-70B, and GPT-4o for detecting clinically significant postoperative delirium PPV positive predictive value, NPV negative predictive value. Figure 3 illustrates the confusion matrices for the agreement between clinically significant POD, primary outcome, and physician- and LLM-assessed POD. Among the 386 patients with clinically significant POD, 279 (72.3%), 347 (90.0%), and 335 (86.8%) were assessed as POD-positive by physicians, Llama-3-70B, and GPT-4o, respectively. Specifically, among 107 patients who had clinically significant POD but were assessed as delirium-negative by physicians, which can be classified as a missed diagnosis, 72 (67.3%) and 63 (58.9%) patients were assessed as delirium-positive by Llama-3-70B and GPT-4o, respectively. Confusion matrices of the human physicians (left panel), Llama-3-70B (middle panel), and GPT-4o (right panel) for clinically significant postoperative delirium classification at the patient level. Each cell presents the number of patients according to the predicted and true labels. The receiver operating characteristic curves of Llama-3-70B and GPT-4o for predicting clinically significant POD are shown in Fig. 4. The c-statistics of Llama-3-70B and GPT-4o were 0.74 (95% CI, 0.72–0.77) and 0.76 (95% CI, 0.73–0.78), respectively. The differences in the discriminative performance between the two LLMs were not statistically significant (DeLong’s test, P = 0.778). Receiver operating characteristic curves of the Llama-3-70B and GPT-4o for predicting clinically significant postoperative delirium. CI confidence interval. Time-to-diagnosis analysis Kaplan–Meier curves for detecting clinically significant POD are presented in Fig. 5. The median time to delirium detection was shortest with Llama-3-70B (34.5 h), followed by GPT-4o (37.5 h), and physicians (62.9 h). Both LLMs detected clinically significant POD earlier than the physicians (all pairwise log-rank P < 0.001), and no significant difference was observed between the two LLMs (log-rank P = 0.428). Kaplan–Meier curves comparing the time-to-delirium detection among physicians, Llama-3-70B, and GPT-4o in patients with clinically significant postoperative delirium within seven days after major surgery. Sensitivity analysis The results of the sensitivity analysis of 3,032 clinical notes from 30 randomly selected patients whose medical records contained no delirium-related keywords are summarized in Table S3 and Figs. S2, S3, and S4. Physicians identified 47 (1.6%) notes as delirium-positive, which was lower than Llama-3-70B (96, 3.2%) and GPT-4o (61, 2.0%). Inter-rater agreement was almost perfect for Llama-3-70B and GPT-4o (Fleiss’ kappa: 0.834 and 0.956, respectively) but moderate for physicians (0.494). Llama-3-70B demonstrated the greatest sensitivity (0.800), followed by GPT-4o (0.733) and physicians (0.400; see Table S3). Of the 15 patients with clinically significant POD, 9 (60%) were assessed as POD-negative by physicians (missed diagnosis; see Fig. S2). Among the other 9 patients, 6 (66.7%) and 5 (55.6%) patients were considered as delirium-positive by Llama-3-70B and GPT-4o, respectively. Discussion The main finding of this large retrospective cohort study was that LLMs can be a useful complementary tool for diagnosing POD in patients undergoing major noncardiac surgery. Both Llama-3-70B and GPT-4o showed moderate predictive performance for clinically significant POD based on unstructured clinical notes. Although the specificity of identifying clinically significant POD was lower in both LLMs than in physicians, both LLMs exhibited substantially higher sensitivity. Notably, both LLMs successfully identified POD in the majority of patients with clinically significant POD who were diagnosed as POD negative by physicians. Furthermore, compared with physicians, LLMs demonstrated markedly higher inter-rater agreement and identified POD approximately one day earlier in patients who developed clinically significant POD. In this study, we used LLMs to identify POD from unstructured clinical notes in a surgical population and to compare their performance with that of physician assessments. Previous studies have investigated the use of NLP to identify delirium. Young and colleagues reported that behavioral disturbances identified by NLP in critically ill patients captured more patients treated with antipsychotics than those identified by the Confusion Assessment Method for the Intensive Care Unit18. A subsequent study by the same research group successfully employed NLP to characterize various behavioral disturbances in clinical notes19. Nevertheless, these studies were confined to ICU settings, and the performance of NLP was not directly compared with physician assessments18,19. Another multicenter retrospective cohort study of 10,516 patients showed that NLP detected delirium with high accuracy using unstructured clinical notes21. However, they used physician assessment as the reference standard to evaluate NLP performance, rather than objective measures such as antipsychotic use21. Furthermore, these studies generally relied on extensive manual labeling to train NLP models, a labor-intensive process that makes clinical implementation challenging18,19,21. In contrast, our study utilized LLMs via a few-shot prompting approach, which is an efficient technique that avoids extensive manual labeling and can be directly incorporated into clinical practice. A prolonged duration of delirium has consistently been linked to worse clinical outcomes6,7. A prospective cohort study involving critically ill patients reported a 10% higher risk of 1-year mortality with every additional day of delirium6. Similarly, another study on critically ill patients with respiratory failure or shock indicated that prolonged delirium is associated with long-term cognitive dysfunction7. Therefore, current guidelines recommend routine assessment of delirium at least once per nursing shift in at-risk patients to facilitate early detection22. However, while two or three daily delirium assessments may capture most cases of delirium, a typical assessment interval of 8–12 h suggests that even with routine screening, delirium diagnosis could be delayed by many hours, potentially up to half a day, leading to missed opportunities for timely intervention23. Moreover, conducting more frequent routine assessments is often impractical because of substantial staffing requirements and high workloads. In our study, we found that LLMs detected delirium approximately one day earlier than physicians. Additionally, LLMs identified the majority of delirium cases overlooked by physicians. The improved sensitivity and earlier identification provided by LLMs could substantially mitigate the harmful consequences of delayed or missed diagnosis of delirium. Integrating LLM-based continuous real-time analysis of clinical notes may further enhance the early detection and diagnosis of delirium, overcome the practical limitations of conventional periodic assessments, and potentially improve patient outcomes. Although previous studies have demonstrated the high reliability of chart-based delirium assessments24,25, the agreement among the three physicians in our study was poor (Fleiss’ kappa = 0.219), and complete agreement was achieved in only about half of the cases. Possible explanations for this discrepancy include differences in the clinical settings, documentation styles, and assessor competence. Earlier studies focused on emergency departments or ICUs24,25, where delirium is routinely assessed and clinicians are more familiar with the typical manifestations of delirium, leading to clearer and more consistent documentation. In contrast, our study utilized unstructured notes primarily from general wards across multiple surgical specialties, where clinician awareness and systematic delirium assessment are relatively low and documentation of delirium-related symptoms tends to be ambiguous and less systematic. Our findings suggest that the previously reported high inter-rater agreement in chart-based delirium assessments in specialized clinical settings may not extend to broader clinical contexts. Additionally, given the non-deterministic nature of LLMs, we performed 20 independent assessments for each model and observed almost perfect agreement, highlighting the potential role of LLMs as adjunctive tools for enhancing consistency in delirium recognition. Among automated systems that utilize electronic medical records, the rapid response system is currently used in clinical practice. This system detects predefined deteriorations in vital signs or mental status in real-time and automatically alerts healthcare providers, thereby reducing cardiopulmonary arrest and in-hospital mortality26. Recently, machine learning-based early warning systems have been introduced to monitor patients at risk of sepsis, offering potential benefits such as earlier identification of sepsis and reduction in the length of hospital stay and inpatient mortality27. Notably, although early evidence on the diagnostic accuracy of LLMs is promising28,29 the integration of LLMs into routine clinical practice to support clinical decision-making has not yet been reported, possibly because of ethical concerns such as accuracy, generalizability, patient privacy, and liability30. Therefore, clear regulatory standards and practical guidelines must be established for the clinical application of LLMs to ensure patient safety and ethical integrity. This study has several limitations. First, this study was conducted at a single tertiary university hospital in Korea. While we demonstrated the potential of LLMs in a non-English clinical context, performance may be affected by factors unique to our institution, such as Korean language structure, the unique cultural expression of cognitive symptoms (e.g., colloquialisms, euphemisms, and metaphors), and different documentation practices. Further research in multicenter, multilingual cohorts is required for external validation. Second, the high sensitivity observed in LLM-based assessments might be attributed to the prior screening of medical records for delirium-related keywords. However, our sensitivity analysis demonstrated that LLMs outperformed physician assessments, even in patients whose clinical records did not include these keywords. Third, direct comparison of LLMs with validated delirium screening tools, such as the CAM, 4 A’s Test, or the Nursing Delirium Screening Scale, was not feasible because routine delirium assessments were inconsistently performed at our institution. Consequently, we used clinically significant POD as the primary outcome, which may have identified only a subset of delirium cases, making hypoactive or mild forms potentially underrecognized. Further research should incorporate validated delirium screening tools as the reference standard. Fourth, both LLMs and physicians assessed each clinical note independently, which may not fully capture the cumulative patient-level context. However, providing LLMs with all clinical notes for each patient would require high computational and financial resources, making it impractical for routine use. Importantly, both LLMs and physicians were evaluated under identical conditions to minimize any potential influence on our overall findings. Finally, although the LLMs were instructed to provide clinical reasoning, we did not perform a systematic qualitative review of discordant cases or evaluate the validity of these explanations. Further studies incorporating structured case reviews and evaluation of LLM-generated rationales are necessary to establish their clinical applicability. In conclusion, compared with physicians, LLMs identified POD in patients undergoing major noncardiac surgery with greater sensitivity and reproducibility and at an earlier stage. Our findings indicate that the integration of LLMs into perioperative care may facilitate the early detection and treatment of POD. Further research is needed to explore the applicability of LLM-based assessments across institutions with different documentation practices and languages. Methods Study design and population The protocol for this retrospective observational study was approved by the Institutional Review Board of the Seoul National University Hospital (approval no. 2410-183-1582). The requirement for written informed consent was waived because all data were de-identified. The study protocol conformed to the principles of the Declaration of Helsinki and the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD)-LLM statements31. The study included adult patients (≥18 years) who underwent major noncardiac surgery under general anesthesia for ≥2 h between January 2019 and December 2021 at a tertiary university hospital. The exclusion criteria were outpatient surgery, brain surgery, American Society of Anesthesiologists physical status VI, and preoperative antipsychotic use. If the patient underwent multiple surgeries during the study period, only the first case was included. Data collection and study outcomes All data used in this study were pseudonymized and extracted by reviewing the electronic medical records from the institutional clinical data warehouse, the Seoul National University Hospital Patients Research Environment (SUPREME) system. The extracted data included age, sex, type of surgery, duration of surgery, length of postoperative hospital stay, American Society of Anesthesiologists physical status, and perioperative antipsychotic use such as including haloperidol, olanzapine, quetiapine, risperidone, and ziprasidone. All available clinical notes, including physician notes, nursing notes, and discharge summaries recorded within seven days of surgery were also obtained. The primary outcome was the predictive performance of the physician and LLM assessments for clinically significant POD within the first seven postoperative days. Clinically significant POD was defined as either requiring antipsychotic medication or a diagnosis of delirium by a neurologist following consultation for delirium-related symptoms. The secondary outcomes were 1) inter-rater agreement at the level of individual clinical notes and 2) time-to-diagnosis from the end of anesthesia at the patient level among those confirmed with clinically significant POD. Assessment of delirium Delirium was assessed for each clinical note using a dual assessment approach: physician-based and LLM-based. Clinical notes from the first seven postoperative days were initially screened for the following delirium-related keywords and their derivatives, including Korean equivalents: delirium, mental status change, inattention, disorientation, hallucination, agitation, and inappropriate behavior10. For the physician-based assessment of delirium, clinical notes were independently reviewed by three physicians: one board-certified anesthesiologist and two second-year anesthesiology residents. The final assessment of whether each clinical note indicated delirium (delirium-positive or delirium-negative) was based on the majority of the assessments by the three physicians. For the LLM-based assessment of delirium, we used two LLMs: Llama-3-70B and GPT-4o. The detailed configuration settings of both LLMs are provided in Supplementary Notes. These LLMs were provided with the same screened clinical notes and instructed to assess whether each clinical note indicated delirium. Each clinical note was subjected to 20 independent assessments using both LLMs, which would provide an adequate number of assessments to reliably assess inter-rater agreement and discriminative performance32. A few-shot prompting approach with structured prompts was used, and each LLM was provided with four example cases (two delirium-positive and two delirium-negative). These cases were selected by an experienced physician and incorporated into the prompts, along with the clinical rationale for determining delirium in each case (Table S4). Both models were instructed to provide a binary classification (“yes” or “no” for delirium) and a clinical rationale for each decision following the format shown in the example cases. Similar to the physician assessment, the final assessment by each LLM of whether each clinical note indicated delirium was determined based on the decision of the majority of the 20 independent assessments and labeled as delirium-positive or delirium-negative. The probability of each clinical note being classified as delirium-positive was also calculated as the proportion of delirium-positive assessments out of 20 LLM-based independent assessments. At the patient level, patients were considered to have POD (POD-positive) if at least one clinical note was identified as delirium-positive during the study period for each physician- and LLM-based assessment. The probability of being POD-positive at the patient level was calculated as 1 minus the product of the probabilities that all notes were classified as delirium-negative. Statistical analysis All analyses were performed using Python version 3.9, with the packages NumPy (version 1.24.3), lifelines (version 0.27.4), and StatsModels (version 0.14.0). P values less than 0.05 were considered statistically significant. The predictive ability of physicians, Llama-3-70B, and GPT-4o for clinically significant POD was compared using metrics including sensitivity, specificity, precision, recall, and F1-score. For both LLMs, discriminative performance was evaluated using receiver operating characteristic curves for the probability that each case would be classified as POD-positive. C-statistics with 95% confidence intervals (CI) were calculated using a bootstrapping method with 2000 replicates, which were compared using DeLong’s test. Additionally, confusion matrices were used to compare the accuracy of POD detection at the patient level. Inter-rater agreement was evaluated using the Fleiss’ kappa coefficient for overall agreement among the three physicians as well as for each LLM’s 20 independent answers33. For patients with clinically significant POD, the Kaplan-Meier method and log-rank test were used to compare the detection of POD between LLM- and physician-based assessments. For each assessment method, the time-to-diagnosis was defined as the interval from the end of anesthesia to the first medical record assessed as delirium-positive. The Bonferroni correction was used for pairwise comparisons. To examine whether keyword-based note screening influences the performance of LLM-based assessments, we performed sensitivity analyses using the notes of 30 patients whose medical records did not contain any delirium-related keywords. Fifteen patients with clinically significant POD and 15 patients without were randomly selected. All notes from these 30 patients were assessed independently by the physicians and LLMs, and the analyses were repeated as previously described. Supplementary information Supplemental Material (revised, clean) Acknowledgements This study used clinical data retrieved from Seoul National University Hospital Patients Research Environment (SUPREME) system. Author contributions D.I.E. and J.W.J. designed the study. D.I.E., H.C.L., G.H., J.H.J., J.L., H.L., S.Y.P., and J.W.J. collected data. All authors contributed to the data analysis. D.I.E., H.L., and K.N. prepared the manuscript. H.C.L., H.J.L., and J.W.J. revised the manuscript. All authors approved the final manuscript as submitted and agreed to be accountable for all aspects of the work. Funding This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number : RS-2024-00439677). Data availability The data that support the findings of this study are available from Seoul National University Hospital but restrictions apply to the availability of these data, which were used under licence for the current study, and so are not publicly available. Data are however available from the corresponding author upon reasonable request and with permission of the Institutional Review Board of the Seoul National University Hospital. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Supplementary information The online version contains supplementary material available at 10.1038/s41746-025-02231-8. References Associated Data Supplementary Materials Supplemental Material (revised, clean) Data Availability Statement The data that support the findings of this study are available from Seoul National University Hospital but restrictions apply to the availability of these data, which were used under licence for the current study, and so are not publicly available. Data are however available from the corresponding author upon reasonable request and with permission of the Institutional Review Board of the Seoul National University Hospital.