Validation of natural language processing for surgical complication surveillance: detecting 11 postoperative complications from electronic health records Authors: Emilie Even Dencker, Alexander Bonde, Anders Troelsen, Martin Sillesen Published: 2026 ABSTRACT AbstractINTRODUCTIONPostoperative complications (PCs) rates are crucial quality metrics in surgery, as they reflect both patient outcomes, perioperative care effectiveness and healthcare resource strain. Despite their importance, efficient, accurate and affordable methods for tracking PCs are lacking. This study aimed to evaluate whether natural language processing (NLP) models could detect 11 PCs from surgical electronic health records at a level comparable to human curation.RESEARCH AND DESIGN METHODSRetrospective study in 18 hospitals across two regions in Denmark. A total of 17 486 surgical cases spanning 6 years were included. The dataset was divided into training, validation and test sets for NLP-model development and evaluation (50.2%/33.6%/16.2%). Model performance was compared against the current method of PC monitoring (International Classification of Diseases, 10th Revision (ICD-10) codes) and manual curation, the latter serving as the gold standard.17 486 surgical cases from spanning 6 years were included. The dataset was divided into training, validation and test sets for NLP-model development and evaluation (50.2%/33.6%/16.2%). Model performance was compared against the current method of PC monitoring (International Classification of Diseases, 10th Revision (ICD-10) codes) and manual curation, the latter serving as the gold standard.RESULTSThe NLP-models had a receiver operating characteristic area under the curve between 0.901 and 0.999 for the test set and significantly outperformed ICD-10 coding in detecting PCs. Sensitivity of the models when compared with manual curation ranged from 0.701 to 1.00, except for myocardial infarction (0.500). Positive predictive value (PPV) ranged from 0.0165 to 0.947, and negative predictive value from 0.995 to 1.00. Using a Human-in-the-Loop approach, only 16.3% of cases required manual review to reach a PPV of 1.00.CONCLUSIONSThe NLP models alone were able to detect PCs at an acceptable level and outperformed ICD-10 codes. While combining NLP with manual review (Human-in-the-Loop) improved overall accuracy and reduced workload, the models still failed to identify some complications. Therefore, NLP algorithms may support (but not replace) manual surveillance and present a potential solution for more scalable PC monitoring.Keywords: Acute Care Surgery, Emergency Surgery, General Surgery, Health Information Technology, Quality of Health Care WHAT IS ALREADY KNOWN ON THIS TOPIC Postoperative complications are crucial quality measures, but current monitoring methods, including International Classification of Diseases, 10th Revision coding and manual review, have significant limitations. Postoperative complications are crucial quality measures, but current monitoring methods, including International Classification of Diseases, 10th Revision coding and manual review, have significant limitations. WHAT THIS STUDY ADDS This study demonstrates that natural language processing (NLP) can detect eleven postoperative complications from electronic health records with accuracy comparable to human curation, offering a possible solution to the existing monitoring methods. This study demonstrates that natural language processing (NLP) can detect eleven postoperative complications from electronic health records with accuracy comparable to human curation, offering a possible solution to the existing monitoring methods. HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY NLP-based monitoring could enable real-time and scalable complication detection, improving surgical quality surveillance and healthcare resource allocation. NLP-based monitoring could enable real-time and scalable complication detection, improving surgical quality surveillance and healthcare resource allocation. Introduction Postoperative complications (PCs) are widely recognised as a critical measure of surgical quality.1 PCs reflect not only the surgical technical performance but also the quality of the perioperative and postoperative care, impacting patient outcomes, including quality of life, functional level and life expectancy. Additionally, at a systemic level, PCs demand substantial resources, both in terms of clinical staffing and economic costs.2,5 Despite their importance, methods for detecting and monitoring PCs remain debated. Many healthcare systems use administrative data sources, such as International Classification of Diseases, 10th Revision (ICD-10) codes and insurance claims, to track PC rates. However, these data sources are often criticised for limited reliability,6,10 with the most notable issues being underreporting and inaccurate documentation of PCs.511,13 To overcome these limitations, some healthcare systems have developed quality improvement programmes relying on manual data extraction from electronic healthcare records (EHRs). The American College of Surgeons’ National Surgical Quality Improvement Program (ACS NSQIP) being a prominent example. While effective and cost-efficient, these programmes have limitations: they are resource-intensive, can introduce selection bias and lack real-time monitoring capacity.14 15 16 Considering the limitations of the current monitoring methods and the resource strain associated with PCs, there is an urgent need for a real-time, high-quality monitoring approach. Here, artificial intelligence (AI), particularly natural language processing (NLP), offers a promising opportunity to improve surgical quality by analysing text from EHRs with minimal cost and time.617,19 However, the range of PCs that NLP can accurately detect remains unclear. This study addresses this gap by evaluating NLP algorithms for detecting a wide spectrum of PCs and assessing their potential as a universal monitoring tool across various surgical specialties. Specifically, we aimed to determine whether NLP models could identify eleven target PCs (deep surgical site infection, organ/space site infection, wound disruption, pulmonary embolism, stroke, cardiac arrest, myocardial infarction (MI), deep vein thrombosis, unplanned readmission, unplanned reoperation and unplanned intubation) from EHR data, including free medical text, with accuracy comparable to human curation. We hypothesised that NLP-models would reliably detect these PCs up to 30 days postoperatively, achieving performance levels equivalent to human curation. Methods Patient and public involvement Due to the retrospective study design and the de-identified nature of the data, no patient or public involvement was possible. Data source Current capture of PCs in Denmark relies on ICD-10 coding, as no national or local surgical quality database exists. We obtained data from EHRs of surgical patients in Denmark’s Capital and Zealand regions over 5 years (2016–2021). Denmark operates a tax-funded universal healthcare system providing comprehensive surgical care to all citizens at no personal expense. The two regions encompass 18 hospitals with access to all surgical subspecialities and postsurgical care, serving approximately 2.7 million residents.20 The dataset included all surgical cases within the 5-year period, covering a range of surgeries (elective and emergency) across various specialties, including orthopaedic, general, plastic, gynaecology and obstetrics, urology, ophthalmic, cardiothoracic, vascular, neurological, oral and otorhinolaryngology surgery. The two regions use the EPIC EHR system (EPIC, Verona, Wisconsin, USA). EHR data comprised clinical notes (eg, admission notes, progress notes, procedure notes, discharge notes), biochemical values and vital signs recorded up to 30 days postoperatively. All notes are mandated by law to be in Danish, but the structure follows international chart notes structures. Additionally, ICD-10 diagnosis codes for comorbidities, acute conditions and PCs were included along with demographic data (age, sex, weight, etc). The study adhered to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines (online supplemental figure S3). Training set The training set includes cases from 1 May 2016 to 30 October 2021, and was originally developed for a previous study on superficial surgical site infections (SSSIs).17 Cases with an ICD-10 code indicating an SSSI were included. (Illustrated in online supplemental figure S1 ‘case flow diagram’). The included cases were manually labelled for various PCs, making the training set suitable for multiple studies on detection of different PCs. When using NLP, the first step is to train the model. This involves using a portion of the data (the training set) for the model to learn how to classify information. This process requires the data to be labelled, so that the model learns which features (words or phrases) are important for detecting an outcome. The training data also teaches the model to connect input (medical free text) with output (PCs). Finally, the training data helps the model generalise from specific examples to predictions/classifications on unseen data. After training, classifiers were added—one for each PC. A classifier determines the probability threshold at which a given PC is classified as present. These first steps of model development are illustrated in figure 1, step 1. This figure outlines the step-by-step development of the NLP models and the specific roles of the three data subsets. The training set is used to train the models, after which classifiers are added. The validation set is used to validate the models and fine-tune the thresholds. Finally, the test set is used to evaluate the models’ performance, where the previously set thresholds are applied. NLP, natural language processing. Validation set The validation set included cases from 1 November through 20 November 2021 (see figure 1). The validation set is used to fine-tune model configuration, optimise hyperparameters and determine thresholds used for classifying PCs (illustrated in figure 1, step 2). Since classification is expressed as a probability rather than a binary output (yes/no), the threshold defines the probability level at which a PC is flagged as present. Adjusting these thresholds affects model sensitivity and specificity. Test set The test set included cases from 21 November through 30 November 2021 (see figure 1). The test set is used to test the model’s performance on a portion of yet unseen data. Therefore, a ‘ground truth’ must be established, and for this we chose manual curation of the EHRs. The thresholds determined during the validation process are then applied to the test set (figure 1, step 3). Validation review processes To prepare data for use in an NLP context, the total dataset was split into three separate subsets: a training set, a validation set and a test set (see above). All cases in the subsets (training, validation and test sets) were initially reviewed once by a member of a team of medical reviewers including medical doctors and trained medical students for the presence of any of the studied complications. To ensure consistency, all positively labelled cases were subsequently rereviewed by a medical doctor (coauthor AB). Cases without any positive labels were not double reviewed. While no formal discrepancy resolution protocol was used, the positive case confirmation step was included to improve labelling reliability for the gold standard. The 2018 ASC NSQIP definitions were used for manual labelling and are available in online supplemental table S2. ICD-10 codes To compare the NLP model’s performance with conventional ICD-10-based monitoring, relevant codes were defined for each PC (online supplemental table S1). ICD-10-codes were documented by treating physicians during treatment or at time of discharge. All codes within 30 days postoperatively were included. For the PCs ‘unplanned readmission’, ‘unplanned reoperation’ and ‘unplanned intubation’ no relevant ICD-10 codes exist, hence comparison was limited to NLP-models versus manual curation. Data leakage and missing data To reduce data leakage and missing data, several steps were taken. Cases not marked as ‘complete’ and cases with missing values for procedure start or completion times were excluded. Only cases from relevant surgical specialties (as determined by Sundhedsvæsenets Klassifikations system-code) were included. If a patient had multiple unrelated procedures within 30 days, only the first was included, except in cases of reoperation. To further prevent temporal leakage, no chart notes dated after 30 October 2021, were included in the training set, ensuring complete separation from the validation and test sets, which were temporally adjacent in time. Machine learning modelling The overall modelling approach has been detailed previously,6 but is summarised here for clarity. For each case, all clinical notes recorded within 30 days postoperatively (eg, admission, progress, operative, anaesthesia, consult and discharge notes) were concatenated chronologically into a single text document. Text preprocessing included lowercasing, removal of punctuation and whitespace-based tokenisation. A Term Frequency–Inverse Document Frequency (TF-IDF) matrix was then generated using word 1–3 grams, retaining terms that appeared in at least five documents. The resulting TF-IDF vectors served as the sole input features to the Light Gradient Boosting Machine (LightGBM) classifiers. The LightGBM is a fast, memory-efficient gradient boosting framework that builds an ensemble of decision trees using a leaf-wise growth strategy and histogram-based splitting. For NLP tasks, it is typically applied to structured representations of text such as TF-IDF vectors, bag-of-words features or engineered linguistic features, rather than raw text sequences. This approach was chosen to serve as a benchmark due to its trade-off between implementation efficiency, performance on modest data sizes and strong performance on tabularised text features. Structured EHR data (eg, vitals, lab values, ICD-10 codes) were not included as input features. However, structured information was used to assign negative labels for three complications: specifically, cases with no readmission were automatically labelled as no unplanned readmission; cases with no reoperation were labelled as no unplanned reoperation. Additionally, cases with index procedure for MI-related surgery (procedure-code prefix KFN) were labelled as no MI complication before text classification. Threshold selection Since the model outputs probabilities for each PC rather than binary classifications, selecting the optimal threshold for these probabilities is crucial. The thresholds directly affect the sensitivity and specificity of the model, influencing rates of false positives and false negatives. To address this, we assess two approaches: a Stand-Alone-Model (SAM) and a Human-In-The-Loop (HITL). The SAM classifies cases based on probability thresholds alone, while the HITL-approach flags higher-probability cases for human review, while low-probability cases were classified as negative and not reviewed by human. Human review was used to assess cases exceeding the selected probability threshold. Thresholds were selected to optimise the F2 score, prioritising sensitivity over positive predictive value (PPV), in order to minimise the rate of false negatives. Performance Model performance was assessed using receiver operator area under the curve (ROC AUC) and area under the precision–recall curve (AUPRC), alongside sensitivity, specificity, PPV and negative predictive value (NPV). Additionally, thresholds for both the SAM and HITL approach are provided. For comparison, performance metrics for ICD-10 codes are also included. Data presentation and implementation Results are presented as counts, medians (IQR), percentages and performance metrics as stated above where appropriate. Data processing was conducted within a secure cloud environment on the Windows Azure platform. Python V.3.8.13 was used to implement the NLP algorithms. Results The total dataset comprised 17 486 surgical cases, corresponding to 17 925 surgical procedures and 182 293 EHR chart notes. Data preprocessing steps are outlined in figure 1 and exclusion criteria are detailed in online supplemental table S3. The dataset was split into a training, validation and test set on a time-based approach (online supplemental figure S2). The training set comprised 5352 cases and 91 435 EHR notes, the validation set comprised 8329 and 61 335 EHR notes and the test set comprised 3805 cases and 29 523 EHR notes. Based on EHR notes the split was 50.2%/33.6%/16.2% (training/validation/test). Demographic results are presented in table 1. Of the study population, 52.8% were female, with a median age of 59 years. The median surgery duration was 52 min, with the most common surgical specialties being orthopaedic (24.4%) and general surgery (22.9%). Table 1Demographics of the total datasetPatients in dataset (>18 years)n=17 486Surgical procedures in datasetn=17 925EHR notes in datasetn=91 435Age, median (Q1–Q3)59 years (39–72 years)Sex, femalen=9238 (52.8%)Surgery duration in minutes, median (Q1–Q3)52 min (27–91 min)BMI (kg/m2), median (Q1–Q3)25.8 kg/m2 (22.6–30.0 kg/m2)ASA score, median (Q1–Q3)2 (2–3)Surgical specialty (by procedure code for planned procedure): Orthopaedic surgery4267 (24.4%) General surgery3997 (22.9%) Plastic surgery2102 (12.0%) Gynaecology and obstetric surgery1769 (10.1%) Urology1206 (6.90%) Ophthalmic surgery1150 (6.58%) Cardiothoracic surgery843 (4.82%) Vascular surgery682 (3.90%) Neurological surgery675 (3.86%) Oral/maxillofacial surgery431 (2.46%) Otorhinolaryngology364 (2.08 %)ASA, American Society of Anesthesiologists; BMI, body mass index; EHR, electronic healthcare record. Demographics of the total dataset ASA, American Society of Anesthesiologists; BMI, body mass index; EHR, electronic healthcare record. Table 2 presents the distribution of the 11 PCs in the validation and test sets, respectively, as determined by both manual curation and ICD-10 coding. Manual curation consistently detected significantly more PCs than ICD-10 code tracking, with overall incidence rates over 10 times higher. The incidence of complications by manual curation was 11.1% in the validation set and 13.5% in the test set. For ICD-10 coded PCs, only around one-third were confirmed as actual complications by manual review (presented in the column ‘matching’). Specifically, 36.0% of the ICD-10 coded PCs in the validation set and 26.1% of the ICD-10 coded PCs in the test set were confirmed. Table 2Manual label distribution and ICD-10 coding for each studied complication in the validation and test setPostoperative complicationManual pos.(n,%)Manual neg.(n,%)ICD-code pos.(n,%)ICD-code neg.(n,%)Matching cases between ICD-10 and manual review (positive)Incidence of postoperative complications in validation set (8329 cases) Deep SSI12 (0.144)8317 (99.9)10 (0.120)8319 (99.9)2 Organ space SSI40 (0.480)8289 (99.5)25 (0.300)8304 (99.7)11 Wound disruption22 (0.264)8307 (99.7)2 (0.0240)8327 (99.9)2 Pulmonary embolism10 (0.120)8319 (99.9)5 (0.0600)8324 (99.9)0 Stroke18 (0.216)8311 (99.8)8 (0.0960)8321 (99.9)6 Cardiac arrest24 (0.288)8305 (99.7)0 (0)8329 (100)0 Myocardial infarction4 (0.0480)8325 (99.9)19 (0.228)8310 (99.8)1 Deep vein thrombosis22 (0.264)8307 (99.7)20 (0.240)8309 (99.8)10 Unplanned readmission359 (4.31)7970 (95.7)NANANA Unplanned reoperation348 (4.18)7981 (95.8)NANANA Unplanned intubation62 (0.744)8267 (99.3)NANANA Total 921 (11.1) 7408 (88.9) 89 (1.07) 8240 (98.9) 32 (36.0) Incidence of postoperative complications in test set (3805 cases) Deep SSI4 (0.105)3801 (99.9)4 (0.105)3801 (99.9)1 Organ space SSI20 (0.526)3785 (99.5)6 (0.156)3799 (99.8)3 Wound disruption5 (0.131)3800 (99.9)0 (0)3805 (100)0 Pulmonary embolism4 (0.105)3801 (99.9)4 (0.105)3801 (99.9)0 Stroke9 (0.237)3796 (99.8)5 (0.131)3800 (99.9)3 Cardiac arrest20 (0.526)3785 (99.5)2 (0.0526)3803 (99.9)1 Myocardial infarction4 (0.105)3801 (99.9)16 (0.420)3789 (99.6)1 Deep vein thrombosis8 (0.210)3797 (99.8)9 (0.237)3796 (99.8)3 Unplanned readmission242 (6.36)3563 (93.6)NANANA Unplanned reoperation168 (4.41)3637 (95.6)NANANA Unplanned intubation31 (0.814)3774 (99.2)NANANA Total 515 (13.5) 3290 (86.5) 46 (1.21) 3764 (98.9) 12 (26.1) The table presents the number and percentage of cases manually labelled as positive or negative for each complication (ie, the ground truth used for model training and evaluation), as well as the corresponding ICD-10 coded cases. The final column (‘Matching’) indicates how many ICD-10 coded cases were confirmed as true complications by manual review.ICD-10, International Classification of Diseases, 10th Revision; NA, not assessed; neg., negative; pos., positive; SSI, surgical site infection. Manual label distribution and ICD-10 coding for each studied complication in the validation and test set The table presents the number and percentage of cases manually labelled as positive or negative for each complication (ie, the ground truth used for model training and evaluation), as well as the corresponding ICD-10 coded cases. The final column (‘Matching’) indicates how many ICD-10 coded cases were confirmed as true complications by manual review. ICD-10, International Classification of Diseases, 10th Revision; NA, not assessed; neg., negative; pos., positive; SSI, surgical site infection. Table 3 presents performance metrics for both the NLP algorithms and ICD-10 tracking compared with manual curation, which served as gold standard. Performance metrics are shown for both the SAM and the HITL approaches, as well as the thresholds. Table 3Performance metrics of the three different methods of identifying PCsROC AUCAUPRCSensitivitySpecificityPPVNPVThreshold*% Needing manual reviewICD-10-tracking DSSI0.6250.0630.25000.99920.25000.992NANA OSSI0.7480.0790.15000.99920.50000.9955NANA Wound disruptionNA0.0010.00001.0000NA0.9989NANA Pulmonary embolism1.0000.5630.75000.99970.75000.9997NANA Stroke0.8740.2020.37500.99950.60000.9987NANA Cardiac arrestNA0.0300.05261.00001.00000.9952NANA Myocardial infarction0.5260.0160.25000.99600.06250.9992NANA Deep vein thrombosis0.7490.1260.37500.99840.33330.9987NANA Unplanned reoperationNANANANANANANANA Unplanned readmissionNANANANANANANANA Unplanned intubationNANANANANANANANASAM approach DSSI0.9990.3921.00000.95990.02601.00000.0181NA OSSI0.9710.2590.75000.96350.09930.99870.0066NA Wound disruption0.9980.6631.00000.93640.01651.00000.0476NA Pulmonary embolism0.9980.3121.00000.99150.11111.00000.0067NA Stroke0.9950.2321.00000.98450.12120.99970.0045NA Cardiac arrest0.9560.1980.84210.94370.07080.99920.0014NA Myocardial infarction0.9900.4430.50000.99570.11110.99950.8152NA Deep vein thrombosis0.9930.1881.00000.97490.07841.00000.3176NA Unplanned reoperation0.9900.8780.89630.98740.76560.99530.4243NA Unplanned readmission0.9010.8000.70130.99740.94740.98080.6530NA Unplanned intubation0.9990.8940.91670.99790.73330.99950.1516NAHITL approach DSSI0.9990.3921.00001.00001.00001.00000.01814.11 OSSI0.9710.2590.75001.00001.00000.99870.00664.03 Wound disruption0.9980.6631.00001.00001.00001.00000.04766.43 Pulmonary embolism0.9980.3121.00001.00001.00001.00000.00670.96 Stroke0.9950.2321.00001.00001.00001.00000.00451.76 Cardiac arrest0.9560.1980.84211.00001.00000.99920.00146.03 Myocardial infarction0.9900.4430.50001.00001.00000.99950.81520.48 Deep vein thrombosis0.9930.1881.00001.00001.00001.00000.31762.72 Unplanned reoperation0.9900.8780.89631.00001.00000.99530.42435.12 Unplanned readmission0.9010.8000.70131.00001.00000.98080.65304.56 Unplanned intubation0.9990.8940.91671.00001.00000.99950.15160.80 TotalNANANANANANANA 16.3 The SAM approach represents the NLP algorithms’ performance by themselves with no human interaction. The HITL approach represents the performance of the NLP algorithms’ in combination with human review, where high-probability cases identified by the NLP models were forwarded to human review. For the HITL approach, we assumed that all cases flagged by the model and subsequently reviewed by a human were correctly classified, as manual review serves as the consistent gold standard throughout the study.*Threshold was determined using the F2 score to optimise the sensitivity over PPV. This was done to minimise the false negative rate.AUPRC, area under the precision–recall curve; DSSI, deep surgical site infection; HITL, Human-In-The-Loop; ICD-10, International Classification of Diseases, 10th Revision; NA, not assessed; NLP, natural language processing; NPV, negative predictive value; OSSI, organ/space surgical site infection; PCs, postoperative complications; PPV, positive predictive value; ROC AUC, receiver operating characteristic area under the curve; SAM, Stand-Alone Model. Performance metrics of the three different methods of identifying PCs The SAM approach represents the NLP algorithms’ performance by themselves with no human interaction. The HITL approach represents the performance of the NLP algorithms’ in combination with human review, where high-probability cases identified by the NLP models were forwarded to human review. For the HITL approach, we assumed that all cases flagged by the model and subsequently reviewed by a human were correctly classified, as manual review serves as the consistent gold standard throughout the study. Threshold was determined using the F2 score to optimise the sensitivity over PPV. This was done to minimise the false negative rate. AUPRC, area under the precision–recall curve; DSSI, deep surgical site infection; HITL, Human-In-The-Loop; ICD-10, International Classification of Diseases, 10th Revision; NA, not assessed; NLP, natural language processing; NPV, negative predictive value; OSSI, organ/space surgical site infection; PCs, postoperative complications; PPV, positive predictive value; ROC AUC, receiver operating characteristic area under the curve; SAM, Stand-Alone Model. For ICD-10 tracking, ROC AUC values ranged from 0.531 (MI) to 0.875 (pulmonary embolism). Wound disruption and cardiac arrest had an insufficient number of ICD-10 codes for ROC AUC calculations, and therefore these results are listed as NA. Sensitivity values ranged from 0.00 (wound disruption) to 0.375 (stroke and deep vein thrombosis), except for pulmonary embolism, which had a sensitivity of 0.750. Specificity values ranged between 0.996 and 1.00, PPV from 0.0625 to 1.00 and NPV from 0.992 to 0.999. For the SAM approach, ROC AUC values ranged from 0.901 to 0.999, with sensitivity values from 0.701 to 1.00, except for MI (0.500). Specificity ranged from 0.936 to 0.997, PPV from 0.0165 to 0.947 and NPV from 0.995 to 1.00. To account for class imbalance, we additionally calculated the AUPRC for each complication (table 3). As expected, AUPRC values were lower than ROC AUC values, particularly for rare complications such as MI (0.443 for the SAM model) and wound disruption (0.663 for the SAM model). These results illustrate the limitations of ROC AUC in highly imbalanced datasets and show that, although the NLP models achieved high sensitivity, precision varied substantially across complications. In the HITL approach, ROC AUC, AUPRC, sensitivity and NPV were identical to those in the SAM approach. All cases flagged for human review were false positives, and no additional true positives were identified through manual evaluation. Consequently, the HITL approach did not increase sensitivity or true positive detection but improved specificity and PPV by converting false positive predictions into true negatives, as flagged cases were assumed correctly identified by the reviewer. Comparative bar graphs (figure 2A) illustrate the methods for PC identification (ICD-tracking vs SAM vs manual curation). PCs with no ICD-10 codes available, SAM versus manual curation is shown in figure 2B. True positives and false positives were presented for each PC as well. (A) Comparative bar graphs of number of PCs identified by ICD-10 codes, manual curation and the ‘stand-alone model’ (SAM) model for the test dataset. For each PC the number of both true positives and false positives are shown. (B) Comparative bar graphs showing the number of postoperative complications identified by manual curation and the stand-alone NLP model (SAM) for complications without available ICD-10 codes. For each PC, both true positives and false positives are shown. DSSI, deep surgical site infection; ICD-10, International Classification of Diseases, 10th Revision; OSSI, organ/space surgical site infection; PCs, postoperative complications; SAM, Stand-Alone-Model. Using the HITL-approach, between 0.48% (MI) and 6.43% (wound disruption) of cases required manual review to identify and remove false positive predictions and thereby achieve the listed specificity and PPV. To detect all eleven PCs, 16.3% of all cases needing manual review. Discussion This study validated the use of NLP algorithms to detect a broad spectrum of PCs. We deliberately chose a simple, interpretable model architecture for this proof-of-concept study, aiming to establish a clear performance baseline for detecting a broad range of PCs. Establishing a baseline is essential before exploring more complex models, which may introduce additional technical and interpretability challenges without guaranteeing improved performance. While the NLP models independently outperformed ICD-10 codes in sensitivity, the fully automated approach requires refinement due to a relatively high rate of false positives. We found that incorporating minimal human interaction produced a more accurate hybrid system (HITL), where only 16.3% of cases required manual review to correct false positive predictions and thereby achieve high specificity and PPV, while sensitivity and NPV remained unchanged. This reduced the manual workload by over 80% compared with fully manual quality improvement programme, such as ACS NSQIP. It should, however, be noted that the benefit of the HITL over the SAM approach is due to the conversion of false positives to true negatives, whereas other metrics remain largely unchanged. Our findings suggest that when using NLP classification on unbalanced datasets (ie, low incidence PCs), relying on ROC AUC alone may misrepresent model performance and NLP remains insufficient for completely autonomous classification without sacrificing key quality metrics. As such, it is important to not only carefully consider AUC performance scores, but also how the thresholding affects the trade-off between sensitivity and PPVs. The optimal trade-off will be dependent on the use case at hand. This is in line with findings from previous studies.6 17 To further address this limitation, we also calculated the AUPRC for each complication. AUPRC provides a more informative measure of model performance in the context of class imbalance by focusing on precision and recall rather than false positive rate. As expected, AUPRC values were consistently lower than ROC AUC, especially for rare complications. However, the NLP models still outperformed ICD-10 tracking by a wide margin across both ROC AUC and AUPRC metrics, supporting their utility for real-world complication surveillance. Even though prior studies have examined NLP for PC detection, most focus on a single complication, often surgical site infections.17,1921 22 In comparison, our study evaluates a broad range of PCs. Previous studies report ROC AUC values between 0.855 and 0.921, with sensitivities ranging from 0.580 to 0.849. In contrast, our results show ROC AUC values ranging from 0.901 to 0.999, indicating superior performance, with the previously mentioned limitation in mind. Our sensitivities, ranging from 0.500 to 1.000 with an average of 0.873, are comparable to those reported in the literature.17,1921 22 Other studies assessing NLP for multiple PCs report similar trends, with sensitivities from 0.560 to 0.950 for nine PCs,23 while another study achieved an average sensitivity of 0.830 for detecting 18 PCs.24 A 2021 systematic review and meta-analysis further that NLP models tend to outperform non-NLP models in sensitivity.25 It should be noted that this approach used here differs from how large language models (LLMs) could be used. While LLMs have shown impressive performance in similar tasks, they are in essence generative models whereas this approach is aimed at classification. As such, the NLP approach used here enables uncertainty measurements (eg,75% likelihood of the PC being present). This can be used for thresholding and thereby ‘calling’ complications, critically for adjusting the sensitivity/specificity and NPV/PPV required for the HITL approach. This is not easily achievable using LLMs. While there are model differences between this NLP approach and the use of LLMs, this thus also extends into how these models can be used in the clinical abstraction workflow. Conversely, the LightGBM model is a simple NLP approach, which may fail to capture complex clinical reasoning required for PC abstraction. As such, performance for some PCs such as the unplanned readmission were only moderate, likely reflecting the shortcomings in this modelling approach when clinical domain knowledge is required. This study offers several strengths compared with the 45 studies included in the meta-analysis,25 our dataset is among the largest. Moreover, using a diverse, multi-specialty dataset increases robustness and generalisability. Additionally, comparing NLP with both the gold standard for PC-detection (manual curation) and the widely used ICD-10 codes tracking provided a comprehensive, multidimensional perspective on quality improvement initiatives. Furthermore, by covering a wide spectrum of PCs, the study offers a more realistic reflection of the diverse clinical setting relevant for future quality improvement strategies. However, this study has limitations. First, not all EHR data types were included (eg, imaging, nurse chart entries, microbiology results), which may have limited the SAM’s performance. In this respect, this study should be seen as a baseline for how well a simple NLP approach can capture PCs on the presented dataset alone. This benchmark can likely be surpassed by increasing the available data types and using more advanced modelling approaches. The quality of ICD-10 codes, which we based our training set on,17 presents another limitation. Recent findings from our research group indicate that ICD-10 codes not only under-report, but also misreport PCs. Using possibly low-quality ICD-10 codes for training the NLP-models may not be optimal. The use of NLP and machine learning have key limitations. Although our models were trained and tested using multihospital data, their generalisability to healthcare systems outside Denmark with different patient populations, clinical documentation practices and EHR infrastructures remains undetermined. External validation in diverse settings will be essential before broader implementation. Dataset biases (eg, age, gender or race) may also affect model performance and potentially result in reduced accuracy for under-represented subgroups. To maximise the detection of true PCs, we optimised for sensitivity. Yet, this came at the expense of a lower PPV, leading to a higher rate of false positive cases. As a result, the models cannot function autonomously. Therefore, we implemented the HITL approach with manual review for high-probability cases. This ensured greater accuracy but limits the models’ efficiency. An important limitation of the HITL implementation is that cases flagged as high probability by the NLP classifier were not enriched for true PCs. All threshold-exceeding cases were ultimately classified as false positives, indicating limited calibration of predicted probabilities for true positive identification. As a result, the HITL approach functioned solely as a false positive reduction mechanism and did not contribute to the detection of additional true positives. Furthermore, it should be noted that the HITL implementation assumed that manual abstraction served as the gold standard, indicated by 1.0 PPV in table 3. It should be underlined that this does not imply perfect abstraction by the human reviewer, but rather that the abstraction would be in line with whether a single human abstractor classified the case as describing the PC in question or not. It could be argued that differences in human abstraction performances and perceptions could affect this. We did, however, choose this approach to mimic the single-abstractor workflow often used for large scale registry approaches such as NSQIP. For real world implementation in large systems where multiple abstractors assess charts, adding measures for cross abstraction performance, inter-rater reliability and the downstream effect on PPVs would be a required addition. Lastly, the ‘black box’ nature of machine learning models (including NLP models) is a notable limitation. The decision-making process of these models can lack transparency, possibly impacting trust and willingness to implement AI-tools in healthcare settings. In summary, this study establishes NLP algorithms as a promising tool for detecting a wide range of PCs from free medical text, significantly outperforming ICD-10 tracking methods and approaching the performance of human curation. While the HITL approach reduced the manual workload substantially, some complications were still missed, underscoring the need for continued human oversight and model refinement. Although further development and external validation are needed, this approach lays the groundwork for a future real-time, automated system for detecting, monitoring and reducing PCs on an international scale. Future studies should aim to improve model performance and assess generalisability across diverse healthcare settings. Supplementary material Footnotes Funding: Supported by a grant (#NNF19OC0055183) from the Novo Nordisk Fonden to MS. Provenance and peer review: Not commissioned; externally peer reviewed. Patient consent for publication: Not applicable. Data availability free text: Due to GDPR-regulations and as this study uses patient sensitive data, the authors are not permitted to share the data without authorisation. The dataset can be accessed with authorisation from the Danish Patients Safety Board (Styrelsen for Patientsikkerhed) and the Danish Capital Region Data Safety Board (Videnscenter for Dataanmeldelser). Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research. Ethics approval: The study received approval from the Institutional Review Board (IRB) for retrospective patient studies in Denmark, the Danish Patients Safety Board (Styrelsen for Patientsikkerhed, approval #31-1521-182) and the Danish Capital Region Data Safety Board (Videnscenter for Dataanmeldelser, approval #P-2020-180). As it was retrospective, used de-identified data and involved no patient contact, informed consent was not required under Danish law. Data availability statement Data may be obtained from a third party and are not publicly available. References Associated Data Supplementary Materials Data Availability Statement Data may be obtained from a third party and are not publicly available.