Early Machine Learning-Based Identification of Hospitalized Patients at Low Risk of Respiratory Deterioration or Mortality in Community-Acquired Pneumonia: External Validation of a Multivariable Model Authors: Claudia Gyimah, Prasamsa Pudasaini, Allison LeMahieu, Phillip Schulte, Yewande E. Odeyemi Published: 2026 ABSTRACT Background/Objective: To externally validate our previously published machine learning model for identifying hospitalized patients with community-acquired pneumonia (CAP) at low risk of respiratory deterioration or death using the Medical Information Mart for Intensive Care IV (MIMIC-IV) database. Methods: This is a retrospective cohort study of adult patients (≥18 years) who were admitted with CAP and acute hypoxemic respiratory failure using the publicly available MIMIC-IV critical care dataset (Beth Israel Deaconess Medical Center, Boston, MA). We conducted an external validation study of a previously developed gradient boosting machine (GBM) model without recalibration using data available within the first 6 h of hospital admission. Results: For the primary composite outcome (need for advanced respiratory support [high flow nasal cannula (HFNC), non-invasive mechanical ventilation (NIMV), invasive mechanical ventilation (IMV)] or in-hospital death), the gradient boosting model demonstrated comparable performance in the derivation and external validation cohorts. The area under the receiver operating characteristic curve (AUC) was 0.713 in the Mayo cohort (n = 4379) and 0.689 in the MIMIC-IV cohort. Accuracy was 0.612 (95% confidence interval [CI], 0.595–0.628) versus 0.606 (95% CI 0.600–0.611), specificity 0.574 versus 0.523, sensitivity 0.723 versus 0.754, NPV 0.860 versus 0.842, and PPV 0.364 versus 0.401, respectively. For secondary outcomes, model discrimination was comparable between cohorts. The AUC for in-hospital mortality was 0.727 in the Mayo Cohort versus 0.733 in MIMIC-IV; for IMV, 0.736 versus 0.724; and for NIMV, 0.732 versus 0.708. Conclusions: Our machine learning algorithm demonstrated good discrimination and a high negative predictive value for identifying low-risk hospitalized CAP patients for respiratory deterioration or death in the external MIMIC-IV dataset, supporting its potential utility for prognostic enrichment in pneumonia clinical trials. 1. Introduction Community-acquired pneumonia (CAP) remains a major cause of hospitalizations among adults and is associated with substantial morbidity, mortality, and healthcare costs worldwide [1,2]. Despite advancements in antimicrobial therapy, vaccination, and supportive care, CAP remains a leading cause of sepsis, acute respiratory failure, and intensive care unit (ICU) admissions [3,4]. The burden is particularly pronounced among individuals with advanced age, chronic lung disease, chronic heart disease, cardiovascular disease, diabetes mellitus, malnutrition, immunocompromising conditions, and those with lifestyle factors such as smoking and excessive alcohol consumption [5,6]. A 2024 review noted that in the United States alone, CAP accounted for approximately 1.4 million emergency department visits, 740,000 hospitalizations, 41,000 deaths, and $7.7 billion in inpatient costs annually [7]. Against this backdrop of high and variable morbidity, the clinical trajectories of hospitalized CAP patients are notably heterogeneous. While many hospitalized patients recover with standard oxygen therapy and antimicrobial treatment, a clinically important subset experiences rapid deterioration requiring escalation to advanced respiratory support, including high-flow nasal cannula (HFNC), non-invasive mechanical ventilation (NIMV), invasive mechanical ventilation (IMV), vasopressor support, or ICU transfer [8,9,10,11]. Early recognition of patients at risk of clinical deterioration remains challenging because physiological instability may emerge after initial presentation despite seemingly reassuring admission findings [10]. Patients may deteriorate rapidly, sometimes within the first hours of hospitalization, making early and reliable identification of high-risk individuals paramount for guiding clinical decisions and allocating resources appropriately [10,12]. Several conventional severity tools are widely used in CAP, most notably the Pneumonia Severity Index (PSI) and confusion, uremia, respiratory rate, blood pressure, and age ≥ 65 years (CURB-65). These instruments were designed to predict mortality risk and inform disposition decisions such as outpatient versus inpatient management rather than the need for advanced oxygen strategies, ICU admission, or organ support, making them suboptimal for identifying patients at risk of deterioration [6,10]. Although both scores have demonstrated reasonable discriminatory performance for short-term mortality, they share fundamental limitations in predicting the dynamic clinical trajectory of hospitalized patients. Specifically, neither tool captures the risk of acute respiratory deterioration during hospitalization—a clinically critical endpoint that is distinct from mortality and often precedes it [13]. Furthermore, these tools were derived predominantly from pre-ICU era cohorts and may not reflect contemporary patterns of care, including the widespread adoption of high-flow nasal cannula and non-invasive ventilation [8,14]. In comparative studies, the Sequential Organ Failure Assessment (SOFA) score has been found most effective at predicting the need for ICU admission, though it often identifies organ failure already existing at the time of evaluation [10]. The American thoracic society/Infectious disease society of America (ATS/IDSA) recommends using the IDSA/ATS 2007 minor severity criteria together with clinical judgment to guide the need for ICU-level care in patients not immediately requiring vasopressors or mechanical ventilation [15]. The 2026 Surviving Sepsis Campaign guidelines recommend the use of screening tools such as the National Early Warning Score (NEWS, NEWS2), Modified Early Warning Score (MEWS), or Systemic Inflammatory Response Syndrome (SIRS) for sepsis identification in hospitalized patients, based on consistent evidence that early warning scores demonstrate higher sensitivity for detecting clinical deterioration [16]. Although these tools are useful for identifying patients at low risk of mortality, they were not developed to predict respiratory deterioration and do not reliably identify patients who will subsequently require advanced respiratory support. Delayed recognition of patients who ultimately require escalation of respiratory support has been associated with worse outcomes, including increased mortality and prolonged hospitalization [17]. These limitations highlight a critical gap in current risk stratification approaches. Improved prognostic tools have the potential to enhance both clinical care and clinical research. In routine practice, accurate early risk stratification may facilitate timely monitoring, escalation of care, and resource allocation. In addition, reliable identification of patients at high risk of respiratory deterioration can support prognostic enrichment in clinical trials by selecting patients most likely to experience clinically relevant endpoints, thereby improving trial efficiency and reducing sample size requirements [18]. These applications are complementary but serve distinct clinical and research objectives. Machine learning approaches may offer advantages over traditional point-based clinical scores [19]. Unlike linear additive systems, machine learning algorithms can capture nonlinear relationships, interactions among variables, and complex patterns within routinely collected electronic health record data [20]. Over the past decade, multiple machine learning models have been proposed for sepsis, deterioration detection, ICU triage, and mortality prediction [21,22]. While these tools show significant potential, most require further external validation before widespread clinical adoption. We previously developed a gradient boosting machine model using variables available within the first six hours of hospital presentation to identify patients with CAP at low risk for respiratory deterioration or death [23]. In the derivation cohort, the model demonstrated moderate discrimination with strong negative predictive value, supporting potential use as a rule-out tool (area under the receiver operating characteristic curve [AUC] 0.713; sensitivity 0.723; negative predictive value [NPV] 0.860) [23]. Such a tool could have direct bedside utility by identifying patients unlikely to require escalation of care and informing early clinical decision-making. Beyond its potential clinical application, accurate risk prediction may also facilitate prognostic enrichment in clinical trials by identifying patients at greater risk for clinically meaningful outcomes. It may also offer research value by enabling prognostic enrichment in clinical trials through the exclusion of individuals unlikely to reach measurable endpoints. The primary objective of the present study was to externally validate the performance of this model in an independent cohort using the Medical Information Mart for Intensive Care IV (MIMIC-IV) database. As part of this validation, we assessed the model’s transportability (generalizability) and its ability to accurately identify patients at low risk for respiratory deterioration or death. We hypothesized that model performance would remain clinically meaningful in an independent cohort despite expected attenuation in discrimination. 2. Materials and Methods 2.1. Study Design and Data Source We conducted an observational retrospective cohort study to externally validate a previously developed gradient boosting machine (GBM) model using the MIMIC-IV database. MIMIC-IV is a large, publicly available, de-identified database containing detailed electronic health record data from patients admitted to Beth Israel Deaconess Medical Center, including demographics, diagnoses, laboratory measurements, and clinical interventions [24,25]. Use of the database was approved under the relevant data use agreements, and the study was exempt from institutional review board oversight due to the use of de-identified data. 2.2. Study Population Adult patients (≥18 years) hospitalized with CAP and acute hypoxemic respiratory failure were identified using methods aligned with the derivation cohort. Inclusion criteria for CAP were ICD diagnosis of pneumonia [International Classification of Diseases (ICD) 9 codes 481–486 and ICD 10 codes J13, J15, J18] and chest radiology report indicating infiltrate. Exclusion criteria were aspiration pneumonia, hospital-acquired pneumonia, ventilator-acquired pneumonia, interstitial lung disease, leukopenia, neutropenia, acquired immunodeficiency syndrome (AIDS), and human immunodeficiency virus (HIV). To preserve the intended prediction window and ensure comparability with the original model, we excluded patients who required advanced respiratory support—defined as high-flow nasal cannula (HFNC), non-invasive mechanical ventilation (NIMV), or invasive mechanical ventilation (IMV)—or who died within the first 6 h of presentation [23]. 2.3. Predictor Variables Predictor variables were selected a priori based on the original model and included data available within the first 6 h of hospital admission [23]. These variables comprised: Demographics: age, sex, and race (categorized as White vs. other/unknown). Comorbidities: congestive heart failure, chronic obstructive pulmonary disease, asthma, chronic liver disease, chronic kidney disease, and active or prior neoplasia. Physiologic and laboratory data: vital signs and laboratory values, including white blood cell count with differential, serum bicarbonate, sodium, and blood urea nitrogen. When multiple values were available within the 6 h window, the earliest recorded value was used to reflect real-time clinical decision-making. 2.4. Outcomes The primary outcome was a composite of need for advanced respiratory support (HFNC, NIMV, or IMV) or in-hospital mortality. Secondary outcomes comprised the individual components to assess model performance across each clinically important endpoint independently and included:In-hospital mortalityRequirement for invasive mechanical ventilation (IMV)Requirement for non-invasive mechanical ventilation (NIMV) In-hospital mortality In-hospital mortality Requirement for invasive mechanical ventilation (IMV) Requirement for invasive mechanical ventilation (IMV) Requirement for non-invasive mechanical ventilation (NIMV) Requirement for non-invasive mechanical ventilation (NIMV) 2.5. Model Application The previously developed GBM model was applied to the external cohort without retraining to assess generalizability. Predicted probabilities were generated for each patient, and a predefined probability threshold of >0.30 was used to classify patients as high risk. This threshold was selected in the derivation study to maximize sensitivity and achieve a NPV of at least 0.85, consistent with the model’s intended use as a screening tool to identify patients at low risk of clinical deterioration for potential prognostic enrichment in clinical trials. 2.6. Statistical Analysis Patient characteristics were summarized using the median [interquartile range (IQR)] for continuous variables and the number (percentage) for categorical variables. These characteristics were summarized for the external validation cohort and the derivation cohort. Model performance was evaluated using standard measures of discrimination and classification. Discrimination was assessed using the AUC. Classification performance metrics included accuracy, sensitivity, specificity, NPV, and positive predictive value (PPV). Further, 95% binomial confidence intervals were calculated for all classification performance metrics. Calibration was assessed by comparing predicted and observed event rates across risk strata; see Figure 1. We observed poor calibration in our model when applied to these external validation data, with the model over-estimating risk at the low end and over-estimating risk among high-risk individuals. We performed a logistic recalibration, which showed good performance [26]. Additionally, calibration was assessed by subgroups of interest (age < 65 vs. age ≥ 65, congestive heart failure [CHF] yes/no, and chronic obstructive pulmonary disease [COPD] yes/no). A decision curve analysis was conducted to evaluate the prediction model. Any missing predictors were treated as a distinct and possibly informative segment of the data, reflecting actual practices where such omissions are common. This allows the resulting GBM prediction model to handle missing (unmeasured) inputs and still produce the predicted probability of an event. There were no missing data for the primary and secondary outcomes or for the following predictors: age, gender, or comorbidities. Data management and analysis were conducted using R version 4.1.2 (RStudio Team 2021, Boston, MA, USA). We used the R packages “gbm” and “caret” for model training. 3. Results In the MIMIC-IV cohort (n = 5438), a total of 4622 patients (85.0%) were free of the primary outcome within the first 6 h of presentation and were therefore included in the validation analysis. Among these patients, 1459 (31.6%) subsequently experienced the primary composite outcome of requiring advanced respiratory support or in-hospital death. This event rate was higher than that observed in the derivation (Mayo) cohort, where 890 of 3528 patients (25.2%) met the primary outcome [23], suggesting a population with greater baseline severity or differing practice patterns in the external dataset (Table 1). At the prespecified probability threshold of ≥0.30 [23], the GBM model demonstrated modest discrimination for the primary outcome, with comparable performance across cohorts (AUC 0.689 in MIMIC vs. 0.713 in Mayo). In the MIMIC cohort, sensitivity was 0.754 (vs. 0.723 in the Mayo cohort), indicating that the model identified approximately three-quarters of patients who went on to experience the outcome, while specificity was 0.523 (vs. 0.574 in the Mayo cohort), reflecting moderate ability to correctly identify those who remained event-free. The NPV remained high at 0.842 (vs. 0.860 in the Mayo cohort), consistent with the model’s intended role as a screening tool to identify low-risk patients. The PPV was 0.401 (vs. 0.364 in the Mayo cohort), and overall accuracy was 0.606 (95% confidence interval [CI], 0.600–0.611), similar to that observed in the Mayo cohort (accuracy 0.612 [95% CI, 0.595–0.628]). The calibration plot (Figure 1) indicates we are over-predicting the risk among low-risk patients and under-predicting risk for high-risk patients. For example, we predict 40% risk when the actual risk for that type of patient is closer to 70%. However, after recalibration, the plot indicates a similar prediction of risk among all risk levels. Subgroup calibration plots by age group, CHF history, and COPD history indicated similar calibration by group, indicating these key factors were not influential on calibration (Supplementary Figures S1–S3). Also, see the figure for variable importance in the external cohort (Supplementary Figure S4). For the secondary outcome of in-hospital mortality, which occurred in 5.1% of patients in the MIMIC cohort compared with 3.5% in the Mayo cohort [22], model discrimination was similar between datasets (AUC 0.733 vs. 0.727). At the predefined threshold, specificity was very high in both cohorts (0.996 in MIMIC vs. 0.995 in Mayo), indicating that the model rarely classified survivors as high risk for death. However, sensitivity was low (0.074 vs. 0.077), reflecting limited ability to identify patients who ultimately died. Despite this, the NPV remained high (0.970 vs. 0.967), and overall accuracy was 0.971 (95% CI, 0.968–0.974) in MIMIC compared with 0.963 (95% CI, 0.957–0.968) in the Mayo Cohort. For the outcome of need for invasive mechanical ventilation (IMV), the model demonstrated moderate discrimination (AUC 0.724 in MIMIC versus 0.736 in Mayo). Sensitivity was relatively low (0.347 vs. 0.321), indicating that a substantial proportion of patients requiring IMV were not identified at the chosen threshold. However, the NPV remained high (0.911 vs. 0.924), suggesting that patients classified as low risk were unlikely to require invasive ventilation. For the need for non-invasive mechanical ventilation (NIMV), discrimination was similarly moderate (AUC 0.708 in MIMIC vs. 0.732 in Mayo), with sensitivity of 0.636 vs. 0.619, and NPV of 0.880 vs. 0.892. Overall, model performance in the external validation cohort was consistent with that observed in the derivation cohort, with stable discrimination and preservation of high negative predictive value across outcomes, supporting its potential utility as a rule-out tool for identifying patients at low risk of early respiratory deterioration. Complete performance metrics are provided in Table 2. Decision curve analysis (Figure 2) suggested a range of threshold probabilities for which our model could yield higher net benefit than default treat-none or treat-all strategies. Compared to CURB-65 and NEWS2, some threshold probabilities (