Development and validation of interpretable machine learning models for postoperative pneumonia prediction Authors: Bingbing Xiang, Yiran Liu, Shulan Jiao, Wensheng Zhang, Shun Wang, Mingliang Yi Published: 2024 ABSTRACT BackgroundPostoperative pneumonia, a prevalent form of hospital-acquired pneumonia, poses significant risks to patients’ prognosis and even their lives. This study aimed to develop and validate a predictive model for postoperative pneumonia in surgical patients using nine machine learning methods.ObjectiveOur study aims to develop and validate a predictive model for POP in surgical patients using nine machine learning algorithms. By evaluating the performance differences among these machine learning models, this study aims to assist clinicians in early prediction and diagnosis of POP, providing optimal interventions and treatments.MethodsRetrospective data from electronic medical records was collected for 264 patients diagnosed with postoperative pneumonia and 264 healthy control surgical patients. Through correlation screening, chi-square tests, and feature importance ranking, 47 variables were narrowed down to 5 potential predictive factors based on the main cohort of 528 patients. Nine machine learning models, including k-nearest neighbors, support vector machine, random forest, decision tree, gradient boosting machine, adaptive boosting, naive bayes, general linear model, and linear discriminant analysis, were developed and validated to predict postoperative pneumonia. Model performance was evaluated using the area under the receiver operating curve, sensitivity, specificity, accuracy, precision, recall, and F1 score. A distribution plot of feature importance and feature interaction was obtained to interpret the machine learning models.ResultsAmong 17,190 surgical patients, 264 (1.54%) experienced postoperative pneumonia, which resulted in adverse outcomes such as prolonged hospital stay, increased ICU admission rates, and mortality. We successfully established nine machine learning models for predicting postoperative pneumonia in surgical patients, with the general linear model demonstrating the best overall performance. The AUC of the general linear model on the testing set was 0.877, with an accuracy of 0.82, specificity of 0.89, sensitivity of 0.74, precision of 0.88, and F1 score of 0.80. Our study revealed that the duration of bed rest, unplanned re-operation, end-tidal CO2, postoperative albumin, and chest X-ray film were significant predictors of postoperative pneumonia.ConclusionOur study firstly demonstrated that the general linear model based on 5 common variables might predict postoperative pneumonia in the general surgical population. Introduction Currently, postoperative pneumonia (POP) is the most prevalent hospital-acquired pneumonia worldwide, accounting for approximately 50% of all nosocomial pneumonias (1). It is the third most common complication in all surgical procedures, with an incidence ranging from 1.5 to 15.8% (1–3). Postoperative pneumonia can cause significant harm to surgical patients’ prognosis, even jeopardizing their lives. Reported mortality rates associated with POP in surgical patients range from 9 to 50%, with significant variations depending on the type of surgery, and even after risk adjustment, the 5-year survival rate for patients decreases by 66% (1, 2, 4). Furthermore, POP leads to a significant prolongation of hospital stay, a notable increase in postoperative ICU admission rate, readmission rate, reoperation rate, and mortality rate, greatly burdening patients and their families with additional medical expenses, which can increase by an average of 2 to 10 times (1–5).Early detection of POP is crucial for timely intervention to prevent complications. However, predicting POP has remained challenging (6). With the rise of 21st-century computer technology and the advent of the big data era, machine learning (ML) have garnered increasing attention in the medical field (7). Research evidence indicates that ML is extensively employed in medicine to establish algorithms and models that effectively classify patients, thereby providing more accurate predictions and diagnoses for diseases, as well as identifying risk factors (8). Given the adverse outcomes, high mortality rates, and substantial healthcare resource consumption caused by postoperative pneumonia globally, further development and validation of predictive models for POP are clearly warranted. Our study aims to develop and validate a predictive model for POP in surgical patients using nine machine learning algorithms, including K-Nearest Neighbors Classifier (KNN), Support Vector Machine-Radial Basis (SVM-Radial), Random Forest Classifier (RF), Decision Tree Classifier-RPART (RPART), Light Gradient Boosting Machine (GBM), Adaptive Boosting (ADA), Naive Bayes (NB), General Linear Model (GLM), and Linear Discriminant Analysis (LDA). By evaluating the performance differences among these machine learning models, this study aims to assist clinicians in early prediction and diagnosis of POP, providing optimal interventions and treatments. Additionally, it aims to establish a practical foundation for the application of machine learning algorithms in the field of anesthesiology and perioperative medicine. Methods Human subjects and study design The study protocol was approved by the Ethics Committee of the Second Affiliated Hospital of Kunming Medical University (No. PJ202139). As this study design is retrospective, the ethics committee waived the requirement for informed consent and clinical trial registration. This study adhered to the applicable TRIPOD guidelines. The data were derived from 17,190 hospitalized patients who underwent surgical treatment at the Second Affiliated Hospital of Kunming Medical University in 2021. In the electronic patient record (EPR) systems of our hospital, a database platform was established by extracting medical records from hospital information system (HIS), anesthesia information management system (AIMS), nosocomial infections surveillance system (NISS), laboratory information system (LIS), picture archiving and communication system (PACS). The inclusion criteria were used: (1) age over 18 years old; (2) newly developed pneumonia within 30 days after surgery. The patients with the following conditions were excluded from this study: (1) pre-existing pneumonia before operation; (2) tracheal intubation or tracheostomy before operation; (3) procedures outside an operating room; (4) outpatient procedures (hospital stay <24 h); (5) incomplete medical records. Postoperative pneumonia was defined on the basis of US Centers for Disease Control definition of pneumonia (9). Based on the inclusion and exclusion criteria, a total of 264 postoperative pneumonia diagnosed patients and 264 healthy control surgical patients during the same period were included to collect relevant information. Controls were matched by surgical specialty and randomly selected at 1:1 from the remaining surgical patients without pneumonia. Variable selection Based on the review of previous literature and expert recommendations, combined with the actual situation of our hospital, a total of 47 variables were collected through the database platform of the electronic medical record of our hospital: (1) Demographic characteristics: gender, height, weight, gender, and body mass index; (2) Patients’ past medical history: history of smoking, alcohol abuse, hypertension, diabetes, cancer, chronic obstructive pulmonary disease, heart failure, coronary heart disease, stroke, intracerebral hemorrhage, hepatitis, coma and infection; (3) Imaging and laboratory indicators (reviewed as last values before operation or first values after operation): chest X-ray film (mottled or vitreous shadows), pulmonary function test, serum albumin, hemoglobin, blood urea, nitrogen, and creatinine; (4) Surgery-related factors: surgical site, surgical difficulty classification, patient’s operative position (supine or prone), type of surgery (scheduled or emergency), surgery period (day or night), duration of surgery, unplanned re-operation, blood loss, duration of bed rest (duration until patients’ first off-bed activity), and preoperative prophylactic antimicrobial use; (5) Anesthesia-related factors: anesthesia method, ASA physical status, total volume of infusion, red blood cells transfusions, human albumin infusion, duration of mechanical ventilation, mean airway pressure, end tidal CO2, and vasoactive drugs (long-term infusion); (6) Invasive procedures: deep vein catheterization, radial artery cannulation, gastric tube insertion, and indwelling urinary catheter; (7) Outcomes: ICU admission, length of hospital stay, mortality. Statistical analysis All collected raw data was entered into a pre-designed Excel spreadsheet, and statistical analysis were performed using SPSS 23.0. Continuous variables were expressed as mean along with standard deviation or median along with interquartile range. Independent sample t-tests were used for normally distributed data, while Mann–Whitney U tests were used for non-normally distributed data. Categorical variables were expressed as numbers and percentages, and tested using Chi-square tests or Fisher’s exact test. Data preprocessing The original variables contain various data types, therefore, before conducting machine learning, the variables needed to be numerized and categorized, and the variables were analyzed to detect missing values and outliers. In our study, no variable had a missing percentage higher than 1%. For quantitative data, missing values were imputed using the median by group, and for qualitative data, missing values were imputed using the mode by group. The allocation of training and testing sets was randomly divided in a 3:1 ratio using the createDataPartition function in the Rstudio statistical software (R 4.2.2) with the caret package. A predictive model was built based on the training set and externally validated using the testing set. Feature selection Since multicollinearity and confounding variables can affect the performance of the model, various methods were used to select variable features to prevent overfitting and enhance applicability (6). (1) Feature selection based on correlation: Calculate the correlation between features and POP, obtain the indices of feature columns with correlation greater than 0.1, and obtain the corresponding feature names. (2) Feature selection based on statistical analysis: For all categorical data features, calculate the chi-square test for two groups of samples separately, and obtain features with p < 0.05. (3) Feature selection based on feature importance: Use a random forest model to obtain feature importance, where a higher MeanDecreaseGini value indicates a higher importance of the feature in the model. Obtain the top 5 features with the highest MeanDecreaseGini. Take the intersection of the above three selection methods and output a Venn diagram to obtain the optimal predictive features. Development of the ML model The construction and validation of the model primarily rely on the “caret” package and “pROC” package in R. A total of nine machine learning classifiers are selected to initially train the training set data. The trainControl function is used to specify training parameters. To build more superior and stable training models, the tune_model function is employed for automatic parameter tuning. This study utilizes a 10-fold cross-validation method to evaluate the generalization performance of the models. This involves dividing the dataset into 10 equal parts, using 9 parts for model training and the remaining 1 part as the validation set to assess model performance and prevent overfitting. These models are evaluated using the receiver operating characteristic curve (ROC) and the area under the receiver operating characteristic curve (AUC) on the test set. The best-performing predictive model is ultimately selected, and the distribution of feature importance is obtained. The interaction between these features is explored and visualized using the “interaction” package in R. Performance assessment of the ML model The Confusion Matrix is the most basic, intuitive, and computationally simple method for measuring the performance of classification models. It provides a more intuitive observation of the accuracy of different classifications, particularly in cases where there is an imbalance in the sample distribution. For binary classification models, the Confusion Matrix is represented by a 2×2 matrix. Our study adopts a 10-fold cross-validation to evaluate the generalization performance of the model. The probability at which the Youden Index is maximized is used as the threshold for classification. The AUC is the main indicator for evaluating the prediction performance of various classification models. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, recall, F1 score, precision, prevalence, balanced accuracy, and threshold are auxiliary evaluation indicators used for performance comparison among different models. Results Patient characteristics A total of 17,190 surgical patients were selected for the study, of which 457 patients were diagnosed with pneumonia. We excluded 129 patient for pre-existing pneumonia before operation, 52 patients for procedures outside an operating room and 12 patients for tracheal intubation or tracheostomy before operation. 264 patients with postoperative pneumonia were included, along with 264 randomly selected healthy control surgical patients during the same period, resulting in a total of 528 patients finally enrolled in our study. The flowchart of the enrollment was shown in Figure 1. The demographic characteristics, laboratory test results, and clinical features of the enrolled patients are shown in Table 1. Flow chart of patient enrollment in this study. Table 1Preoperative characteristics of the study patients.VariablesPatients with pneumonia (n = 264)Patients without pneumonia (n = 264)p_valueGenderMale155(58.71%)132(50.0%)0.04Female109(41.29%)132(50.0%)Weight (kg)61.12 ± 10.8860.35 ± 10.250.41Age (yr)54.70 ± 14.8548.73 ± 14.86<0.001Body Mass Index≥24 kg/m2103(46.8%)92(35.4%)0.01<24 kg/m2117(53.2%)168(64.6%)Hypertension81(30.7%)52(19.7%)0.004Diabetes mellitus28(10.6%)14(5.3%)0.02Malignancy86(32.6%)52(19.7%)0.001Chronic obstructive pulmonary disease33(12.5%)6(2.3%)<0.001Coronary heart disease7(2.7%)6(2.3%)0.78Stroke15(5.7%)8(3.0%)0.14Intracerebral hemorrhage31(11.7%)14(5.3%)0.008Heart failure3(1.1%)0(0%)0.08Hypohepatia7(2.7%)10(3.8%)0.46Renal dysfunction6(2.3%)2(0.8%)0.15Hepatitis6(2.3%)12(4.5%)0.15Infection of other sites23(8.7%)12(4.5%)0.05Chest X-ray filmAbnormal136(51.5%)85(32.2%)<0.001Normal128(48.5%)179(67.8%)Pulmonary function testAbnormal27(10.2%)8(3.0%)<0.001Normal237(89.8%)256(97.0%)Coma (GCS<8)34(12.9%)5(1.9%)<0.001History of smoking112(42.4%)83(31.4%)0.009Alcohol abuse91(34.5%)66(25.0%)0.02Emergency surgery58(22.1%)19(7.2%)<0.001Preoperative hemoglobin<100 g/L19(7.2%)15(5.7%)0.48≥100 g/L245(92.8%)249(94.3%)Postoperative hemoglobin<100 g/L87(33.0%)44(16.7%)<0.001≥100 g/L177(67.0%)220(83.3%)Preoperative albumin<35 g/L47(17.8%)40(15.2%)0.41≥35 g/L217(82.2%)224(84.8%)Postoperative albumin<35 g/L198(75.0%)117(44.3%)<0.001≥35 g/L66(25.0%)147(55.7%)ASA physical status≥Grade III143(54.2%)106(40.2%)0.001