Deep survival analysis from adult and pediatric electrocardiograms: a multi-center benchmark study Authors: Platon Lukyanenko, Joshua Mayourian, Mingxuan Liu, John K Triedman, Sunil J Ghelani, William G La Cava Published: 2025 ABSTRACT AbstractBackgroundArtificial intelligence applied to electrocardiography (AI-ECG) has recently shown potential for mortality prediction, but heterogeneous approaches and private datasets have limited generalizable insights into AI methodologies fit for this purpose. To address this, we systematically evaluated model design choices across three large medical center cohorts: Beth Isreal Deaconess (MIMIC-IV: n = 795,546 ECGs, United States), Telehealth Network of Minas Gerais (Code-15; n = 345,779, Brazil), and Boston Children’s Hospital (BCH; n = 255,379, United States).ResultsWe comprehensively evaluates models to predict all-cause mortality, comparing horizon-based classification and deep survival methods various neural architectures including convolutional neural networks and transformers. We also benchmarked against demographic-only and gradient boosting baselines. Top models yielded good performance (median concordance, Code-15: 0.83; MIMIC-IV: 0.78; BCH: 0.81). Incorporating age and sex improved performance across all datasets. Classifier-Cox models exhibited site-dependent sensitivity to horizon choice (median Pearson’s R, Code-15: 0.35; MIMIC-IV: −0.71; BCH: 0.37). External validation reduced concordance, and in some cases, demographic-only models outperformed externally trained AI-ECG models on Code-15. However, models trained on multi-site data outperformed site-specific models by margins ranging from 5% to 22%.ConclusionsThese findings highlight several key factors for robust AI-ECG deployment. Deep survival methods consistently provided advantages over horizon-based classifiers, while inclusion of demographic covariates such as age and sex improved predictive performance across sites. The sensitivity of classifier-based models to horizon selection underscores the need for site-specific calibration. The multi-site experiment reveals that cross-cohort training, even between adult and pediatric cohorts, can substantially improve performance on those cohorts compared to cohort-specific training. Together, these results emphasize the importance of model type, demographic features, and training data diversity in developing AI-ECG models that can be reliably applied across populations.Supplementary InformationThe online version contains supplementary material available at 10.1186/s13040-025-00510-4.Keywords: Electrocardiography, AI-ECG, Convolutional neural networks, Survival analysis Introduction Electrocardiography (ECG) measures the electric activity of the heart. Abnormal 12-channel ECGs often indicate cardiovascular pathology and are thus a marker for disease and mortality. AI-ECG refers to the application of AI or machine learning to ECG interpretation. A common AI-ECG task is risk prediction, which can be framed as predicting event occurrence (e.g. mortality or passing a diagnostic threshold). Several studies have recently applied AI-ECG to predict patient outcomes including mortality risk [1–5], ventricular hypertrophy [6], and ventricular dysfunction in both adults [7] and children [8]. While these studies demonstrate AI-ECG’s potential value, most use private data and only report a single, or a small set, of modeling approaches. Consequently, there is little consensus on best approaches to AI-ECG model development in general, including basic task formulation (e.g., classification or time-to-event prediction), the choice of model architecture, covariates, and other training procedures. Most studies develop AI-ECG models on single, large, private ECG datasets (e.g., ECGs: 2.4M [9], 2.3M [4]; 1.2M [10]); the majority report results for a single AI-ECG model configuration (e.g [1, 2, 9, 10, 15]), and still fewer make models publicly available (e.g [1, 15]). Another notable challenge in the mortality prediction setting is for site-specific models to generalize to new sites; Sau et al. [10] and Hughes et al. [1] report model performance drops (AUROC) of 6–20% in external testing. Verifiably evaluating different modeling approaches and creating reproducible public ECG models requires large, public ECG datasets linked to high-quality patient-level data. Fortunately, such data has recently become more available with the public releases of the MIMIC-IV-ECG dataset (~ 800k ECGs) in 2023 [13, 14] and the Code-15 dataset (~ 345k ECGs) in 2020 [15], vastly expanding on pre-2020 datasets [16]. Given the high interest in risk prediction from ECGs and the availability of these new datasets, the primary goal of this study is to benchmark state-of-the-art deep learning architectures and survival modeling approaches to inform AI-ECG best practices. To do so, we develop AI-ECG models of all-cause mortality using Code-15, MIMIC-IV-ECG, and a pediatric dataset from Boston Children’s Hospital (BCH). We quantify model performance as a function of model architecture, survival analysis approach, and covariate inclusion choice, across datasets, classifier time horizons, and patient subgroups. We create robust baseline models to contextualize AI-ECG performance relative to traditional machine learning approaches and simple baselines. Finally, we evaluate the impact of cross-site training on model generalization and site-specific performance. Background Most AI applications to ECG signals have considered simpler prediction tasks than mortality; for example, many studies focus on arrhythmia detection, where classical AI methods leveraging feature extraction regularly report accuracies exceeding 99% [31]. While such performance is, in part, made possible by multi-center ECG benchmarking challenges [32], these have yet to explore mortality and tend to focus on longer signals with lower patient counts. Mortality prediction from ECGs is a more complex task, given the myriad pathologies that influence risk, yet has considerable utility from a clinical point of view, as it provides a general way for providers to risk-stratify patients and manage treatment. Recent studies have demonstrated 5-year mortality prediction from ECGs, beginning with Raghunath et al. in 2020 [4] and, more recently Hughes et al.’s work in 2023 [1]. Many studies utilize the time-series-adapted residual network that was proposed by Ribeiro et al. in 2020 [15]. However, these works did not fully consider mortality as a time-to-event prediction (i.e., as a survival analysis problem), nor did they utilize state of the art approaches to deep learning for survival analysis [20]. Most recently, Sau et al. [10] demonstrated survival analysis from ECGs using a deep survival approached called DeepHit [24]. However, the study does not provide code or sufficient methods details to support future research, which is one of our motivations here. Our study covers these methods and benchmarks several additional approaches. For example, we benchmark a recent transformer-based method called ECGTransform [30], as well InceptionTime, which is considered a state-of-the-art time series classification architecture [26]. We benchmark three additional deep survival approaches. In contrast to prior studies, we evaluate the impact of cross-site training, benchmark on two large, publicly available databases, and make our code and model benchmarks publicly available to support downstream scientific research. Methods Patient populations Code-15 [15] is an ECG dataset from the Telehealth Network of Minas Gerais, a Brazilian public agency providing telehealth services to Minas Gerais and Amazonian and Northeast states. Patient ECGs were recorded in primary care facilities by technicians and examined remotely by a cardiologist. The publicly available dataset includes 345,779 ECGs collected from April to September of 2018. Code15 ECGs are 7–10 s signals sampled at 400 Hz, centered and padded with zeros to total a length of 4096. The Code-15 Dataset provides multiple ECGs per subject and indicates the patient’s age and the follow-up time after the patient’s final ECG. We only use the one entry per subject that provides a specific follow-up time. MIMIC-IV-ECG [13], referred to as simply MIMIC-IV for brevity, includes 795,546 ECGs from 159,608 patients collected between 2008 and 2019 at the Beth Israel Deaconess Medical Center in Boston, Massachusetts. Patient ECGs were recorded in various settings, including emergency settings, hospitals, and outpatient care centers. Each ECGs is a 10 s signal sampled at 500 Hz. MIMIC-IV tracks date-of-death with state and hospital records, and censors deaths one year after a final recorded hospital visit. These signals have channels 4 and 5 switched compared to Code-15 and are altered to match the Code-15 format. Machine measures are used “as is” when generating models, as neither normalization nor indicator variables improved results in preliminary analyses. BCH includes 225,379 ECGS from 79,568 patients collected 1990 to 2018 at Boston Children’s Hospital in Boston, Massachusetts from emergency, hospital, and outpatient care settings [17]. In contrast to the adult cohorts, this dataset predominantly represents a pediatric congenital heart disease cohort. BCH ECGs are 8-second signals sampled at 250 Hz. Death was tracked by an internal institutional database. Contrasting with adult data, pediatric ECGs have higher heart rates, a relatively larger right ventricle, and a strong age dependency; pediatric mortality is also more likely to have a congenital cause. Dataset preparation Dataset preparation is shown in Fig. 1. Data is limited to [Patient ID, Time-To-Event, Event, ECG, Age, Sex, Machine Measures (MIMIC-IV only)], and entries with missing ECG samples or Time-To-Event are excluded. Time-To-Event is set to a minimum of 1 day. All ECGs are limited to 7 s, resampled to 400 Hz, and symmetrically padded with zeros for an end shape of 12 × 4096. ECGs are split 64/16/20 into train/validation/test sets by patient ID (Figure S1). We do not limit the number of ECGs per patient. The random seed determines the Training/Validation split while the Test set remains fixed. Before model training or evaluation, all ECGs are z-score normalized per ECG channel based on the training set. Flow diagram describing the experimental design Survival analysis Survival analysis builds survival functions, S(t), denoting the probability of not having experienced an event by a time t. Survival analysis uses data in the form [Time-To-Event, Event], which denotes event state at a follow-up time. This leverages information even when final event time is unknown: a device that works for two years before being lost still provides two years of evidence of non-breakage. ‘Censoring’ refers to not knowing event outcomes for some subjects. The most common survival functions are Kaplan-Meier curves which display survival over time from a tracked population. If the population is clustered into groups, an individual’s trajectory can be estimated from their cluster’s survival function. Otherwise, survival functions usually fit an exponential decay or Weibull function to regressors (demographics, measurements, etc.). The Cox proportional hazards model [18] models hazard, h(t) = d/dt (1 – S(t)), as an unknown base function scaled by exponential decays: h(t) = h0(t)exp(β1 × 1 + β2 × 2+. + βmxm) where {x1, xm} are regressors and {β1, βm} are learned weights. The Breslow estimator is often used to fit h0(t) [19]. The Cox regression assumes that log-risk (β1 × 1 + β2 × 2+… + βmxm) is proportional (e.g. smoker risk / non-smoker risk = constant) and a linear function of regressors that is time-constant (there are no time-regressor interactions). Broadly, there are two survival modeling approaches with neural networks: Classifier-Cox (CC) The first approach converts inputs into a set of values through a neural network. These values are then treated as generic markers to fit a Cox regression [18]. The neural networks are typically trained to return a single value classifying inputs by whether an event occurs within a time horizon or not. This approach is easily implemented and can account for new covariates in the Cox regression stage but must handle data censoring when assigning classifier labels. We tune models to minimize binary cross-entropy loss at a fixed horizon time, sweeping several horizons. Deep survival (DS) The second approach trains neural networks with survival-specific loss functions to directly generate survival curves. We evaluate four different Deep Survival approaches from the PyCox [20] package. DeepSurv [21] models log-risk as a time-constant, proportional, non-linear function. LogisticHazard (LH) [22] models log-risk as a time-varying, non-proportional, non-linear function and attempts to eliminate batch size effects. MTLR [23] models risk with a time-varying, non-proportional, logistic regression on a non-linear function. DeepHit [24] models risk as a time-varying, non-proportional, non-linear function and allows for several competing risks – i.e., multiple possible outcome events – which we do not exploit in this study. These survival models were chosen for their variety and standardized implementation with PyCox, which minimizes loss functions customized to each approach. AI-ECG survival models Several recent studies predict mortality from ECGs. These are either built completely on large, private, ECG banks [3–5, 10, 16, 25] or are fine-tuned in the context of a particular setting or population [2, 11, 12]. Some studies use Deep Survival [3, 10, 11], and some use Classifier-Cox approaches [4, 5, 25]. Most studies use a convolutional network to interpret ECG, with a slightly-more-popular choice being the ResNet architecture from Ribeiro et al. [5, 10, 15, 17, 25]. Benchmark procedure Benchmark parameters are in Table 2. In total, more than 3,000 AI-ECG models are analyzed. We use two convolutional neural network (CNN) architectures, InceptionTime [26] and a modified ResNet architecture [15], and one transformer architecture, ECGTransform [30]. For each architecture, we train four deep survival models and four classifier-Cox models where neural net classifiers predict mortality up to one, two, five, and ten-year horizons. Models are trained with and without age and sex covariates. Additional modeling details are in the Supplement. We make this resource publicly available (see Data and Resource Availability Section). Table 1Settings for the mortality prediction benchmark experimentSettingValuesSurvival Analysis Deep-SurvivalDeepHit, DeepSurv, LogisticHazard, Multi-Task Logistic Regression Classifier-CoxClassification mortality time horizon (years): 1,2,5,10AI-ECG ArchitectureInceptionTime (CNN), Resnet (CNN), ECGTransformerCovariate optionsNone, Age + Sex, Age + Sex + ECG Machine Measure (MIMIC-IV only)Demographic-only modelsXGBoost, Feedforward NetworkNormalizationz-score per channel, based on model’s training data Settings for the mortality prediction benchmark experiment Neural network architectures Our neural networks had three modules: an ECG-processing module, a covariate-processing module (three feedforward-ReLU layers with output dimension 32), and a fusion module connecting the other two pieces (three feedforward-ReLU layers with output dimension 128) that heads into a final linear layer (dimension 1 for Classifier-Cox models and DeepSurv, dimension 100 for LH, MTLR, and DeepHit – one per time bin). When covariates are not included, the covariate-processing module is skipped. When ECG is excluded for covariate-only baselines, the ECG module returns a one-dimensional value of ‘0’, resulting in a feed-forward network. Along with the survival approaches, we benchmark three neural network architectures. “ResNet” is a multi-channel time series [15, 25] adaptation of the original ResNet [27]. Architecture parameters were kept at defaults tuned to the full Code dataset. This model has 6.9-7.5 M parameters. “InceptionTime” [26] adapts AlexNet [28] by widening convolutional kernel widths and including channel-wise bottlenecks to control model complexity. This architecture performs well on many small time-series classification benchmarks and has recently been used in fetal heart rate monitoring [29]. Architecture parameters were kept at publication defaults (kernel widths 11, 21, 41). This model has 510-530k parameters. “ECGTransForm” is a recent transformer architecture combining multi-scale convolutions with a bidirectional transformer. It reports state-of-the-art performance in arrhythmia classification and has 2.6 M parameters [30]. These architectures were chosen for their use in ECG or physiological time series analysis and adapted from existing repositories. Baselines We generate baseline models by training XGBoost and feedforward networks (‘FF’) on age and sex covariates. In addition, we leverage the automated machine measures available in MIMIC-IV (Axes – P, QRS, T; Durations – P, PQ, QRS, QT, RR.). We generate a baseline model using these measures, along with age and sex, as a baseline feature extraction modeling approach. We additionally test providing measures to AI-ECG models. Outcome measures and statistics Our primary outcome measure is the Concordance Index (concordance). Like area under the receiver operating characteristic curve (AUROC), concordance evaluates the ability of a model to rank patients correctly by risk, but also takes into account the timing of the event. At the time of an event, a subject should be at a higher risk than any other subject still under observation; concordance is the fraction of correctly ranked subject-subject comparisons. Additional measures (AUROC, AUPRC, time-censored concordance) are available in the Supplemental Material. Unless otherwise mentioned, paired comparisons use the Wilcoxon test and unpaired comparisons use the Mann-Whitney test. Multiple comparisons are adjusted for with Benjamini-Hochberg corrections at a 5% false discovery rate. Results Patient populations The patient cohorts resulting from our data preparation and inclusion criteria are summarized in Table 1. Overall, our experiment is conducted over 1,200,658 ECGs from 443,216 patients. Table 2Population characteristics for the three sites used for benchmarkingSiteMeasureAllEvent (Mortality)Non-Eventp valueCode-15 (2018)ECGs, N (%)233,6478,341 (3.6)225,306 (96.4)Patients, N (%)233,6478,341 (3.6)225,306 (96.4)Sex, N (%)Female138,911 (59.4)3,853 (2.8)135,058 (97.2)<< 1E-5Male947,36 (40.6%)4,488 (4.7)90,248 (95.3)Mean Age, Yr (Std)50.7 (19.8)70.4 (14.0)50.0 (19.6)<< 1E-5Mean Follow-up Time, Yrs (Std)3.7 (1.9)2.0 (1.6)3.7 (1.8)<< 1E-5MIMIC-IV (2008–2019)ECGs, N (%)785,035215,039 (27.4)569,996 (72.6)Patients, N (%)159,12225,107 (15.8)134,015 (84.2)Sex, N (%)Female384,923 (49.0)99,170 (25.8)285,753 (74.2)<< 1E-5Male400,112 (51.0)115,869 (29.0)284,243 (71.0)Mean Age, Yrs (Std)62.3 (17.1)70.4 (14.2)59.2 (17.1)<< 1E-5Mean Follow-up Time, Yrs (Std)2.6 (2.6)2.0 (2.4)2.9 (2.6)<< 1E-5BCH (1990–2018)ECGs, N (%)181,97612,638 (6.9)169,338 (93.1)Patients, N (%)50,44716,71 (3.3)48,776 (96.7)Sex, N (%)Female86,141 (47.3)6,048 (7.0)80,093 (93.0)0.23Male95,835 (52.7)6,590 (6.9)89,245 (93.1)Mean Age, Yrs (Std)12.8 (11.9)21.5 (16.7)12.2 (11.2)<< 1E-5Mean Follow-up Time, Yrs (Std)8.9 (6.7)7.2 (6.1)9.0 (6.7)<< 1E-5Categorical p-value comparisons use the Chi-square test; numerical comparisons use Student‘s t-test Population characteristics for the three sites used for benchmarking Categorical p-value comparisons use the Chi-square test; numerical comparisons use Student‘s t-test Local model concordance by architecture, survival approach The concordance of site-specific models, across model configurations, input and datasets, is summarized in Fig. 2. Because transformers showed markedly worse performance, their results, along with additional baselines, are detailed in Table S1. The following analysis, statistics, and discussion focus on the CNN and competitive baseline models. Model concordances across datasets (subplots), covariates (columns), survival modeling approaches (x-axis), and architectures (color). Hatched bars indicate deep survival approaches. “Cla-X” marks X-year horizon Classifier-Cox models. “Best XGB Model” is a Cla-2 model trained on demographics and automatic ECG machine measures. Error bars denote 95% confidence intervals of mean concordance over 5 random seeds. Error bars denote 95% confidence interval The model configurations with best test set performance on each dataset are shown in Table 3, including performance at the 1 and 5 year marks. For cross-reference, chance mortality rates per year are in Table S2. For MIMIC-IV-ECG, appending automatic machine measurements as covariates did not noticeably improve model performance. Table 3Concordance and AUPRC of top-performing, site-specific models, per dataset and input configurationDatasetModel InputsApproachArchitectureConcordance1 Year AUPRC5 Year AUPRCCode15y ~ ECGDeepSurvResNet0.80 (0.80–0.81)0.07 (0.07–0.07)0.14 (0.14–0.14)y ~ ECG + Age + SexMTLRResNet0.83 (0.82–0.83)0.07 (0.07–0.07)0.14 (0.14–0.15)MIMIC-IVy ~ ECGDeepHitResNet0.77 (0.77–0.77)0.42 (0.42–0.42)0.49 (0.49–0.49)y ~ ECG + Age + SexDeepHitInceptionTime0.78 (0.78–0.78)0.45 (0.44–0.45)0.52 (0.51–0.52)y ~ ECG + Age + Sex + Machine MeasuresDeepHitResNet0.77 (0.77–0.78)0.43 (0.43–0.44)0.46 (0.45–0.48)BCHy ~ ECGMTLRResNet0.79 (0.79–0.80)0.05 (0.04–0.05)0.13 (0.13–0.14)y ~ ECG + Age + SexDeepHitResNet0.79 (0.79–0.80)0.05 (0.05–0.06)0.14 (0.13–0.14)For cross-reference, mortality rates per year are in Table S2 Concordance and AUPRC of top-performing, site-specific models, per dataset and input configuration For cross-reference, mortality rates per year are in Table S2 Classifier-Cox performance is sensitive to time horizon We find that classifier-Cox horizon choice (the time at which binary mortality is determined) correlates with median model concordance in all cases (Table 4). This correlation can be positive (Code-15, BCH) or negative (MIMIC-IV), hindering the selection of a generally appropriate value. Table 4Classifier-Cox models are sensitive to the choice of time horizonECG + DemographicECG OnlyDatasetPearson Rp valuePearson Rp valueCode-150.352.66E-020.418.89E-03MIMIC-IV-0.712.66E-07-0.881.08E-13BCH0.371.95E-020.472.35E-03Pearson‘s R correlation between classification model concordances and the choice of time horizon used in training. The direction of the effect is positive in BCH and Code-15 and negative in MIMIC-IV Classifier-Cox models are sensitive to the choice of time horizon Pearson‘s R correlation between classification model concordances and the choice of time horizon used in training. The direction of the effect is positive in BCH and Code-15 and negative in MIMIC-IV Deep survival approaches outperform Classifier-Cox approaches Among local models trained with ECG and covariates, Deep Survival (DS) models, overall, show statistically higher concordance than Classifier-Cox (CC) models across sites (p < 1.98E-03, Table S3), although the performances differences are seldom large. On cross-evaluation, DS outperforms CC in 4/6 cases and CC outperforms DS in 1/6. CNN model architectures perform similarly, and outperform transformer architectures For models trained with ECG and covariates, InceptionTime (IT) outperforms ResNet (RN) on the BCH dataset; on cross-evaluation, RN outperforms IT in 3/6 cases and IT outperforms RN in 1/6 cases (Table S4). Transformers, even on top individual runs (concordance, Code-15: 0.793, MIMIC-IV: 0.762, BCH: 0.774) do not exceed median Deep Survival CNN model concordance (Table S1). Including demographic covariates improves model performance In paired evaluations, AI-ECG models that included age and sex covariates always outperform ECG-only models (p < 2.05E-4, Table S5; Fig. 2), improving concordance an average of 5.2%, and as much as 11.7%. Multi-site model training improves both global and local model performance Figure 3 shows the performances of ResNet AI-ECG models in local and external evaluation across sites. As expected, models perform best when trained, at least partially, on data generated from the same population as their respective test sets. Covariate-only models (age and sex models) provide surprisingly strong baselines and generalize well between sites. This is especially noteworthy for Code-15: covariate-only models generalize to Code-15 cohort better than CNNs trained on other single site data. Model concordance in cross-evaluation. Rows denote test performance on Code-15, MIMIC-IV, and BCH, respectively. Columns denote which site the models were trained on. The rightmost column shows concordance for models trained across the training sets of all three sites. Bar color denotes the survival analysis approach; hatches denote deep survival approaches. The performance of the best demographic only baseline model (trained on age + sex) is shown by a dotted line. Results here are for the ResNet model architectures trained with ECGs and demographics. Error bars denote 95% confidence interval The rightmost column of Fig. 3 also shows the performance of ResNet models trained on data from all sites. These provide maximum, or near-maximum, performance on all datasets, across modeling approaches. In Fig. 4, we more directly compare the performance of locally and globally trained ResNet models on each site, which demonstrates that globally trained models outperform site-specific models, even on test data from the sites on which the local models are trained. Multi-site training improves local performance. Median concordance of mortality models trained on site-specific data (light brown) versus training data from each site (red). Subplots denote the site-specific performance of these models, from left to right, on data from Code-15, MIMIC-IV-ECG, and BCH, respectively. X-axis denotes the modeling approach; all approaches use the ResNet architecture and include demographic covariates. Error bars denote 95% confidence interval We investigate site-specific model performances across test cohorts on age and sex subgroups in Figure S4. Although cohorts differ by age and sex which may make generalization challenging, we do not find clear evidence that performance differences are explained by these age or sex differences. Survival models capture expected survival rates We compare Kaplan-Meier curves for Code-15 and MIMIC-IV to mean predicted population survival from ECG-only models in Figure S2, confirming that model estimates closely match expected survival rates throughout the study period. Model explanations exhibit salient risk factors for mortality We investigate model explainability with a median-waveform analysis [8] in Fig. 5. The median high-risk waveforms show lower amplitude and QRS complexes, as well as flattened/inverted lateral precordial T waves indicating several pathological findings (delayed myocardial activation, possibly myocardial strain). The QRS complex is most salient, suggesting focus mostly on myocardial activation (heartbeat dynamics). High salience of the V2 lead indicates anteroseptal activation and aligns with analyses of left ventricular (LV) dysfunction. Model explainability across various heart lesions for a ResNet Classifier-Cox approach is also explored in prior work [17]. Median heartbeats across 12-lead ECG and SHAP analysis for a MIMIC-IV DeepHit ResNet. Green: median heartbeat for 100 ECGs with lowest-risk-estimate (mean prediction probability: 0%). Red: median heartbeat for 100 ECGs with highest-risk-estimate (mean prediction probability: 69%). Blue shading: SHAP salience. Model explanations highlight dampened QRS activity, indicative of LV dysfunction, in high-risk patients. For details of median waveform analysis, see [8] Discussion Contribution This work is the first to benchmark AI-ECG survival modeling approaches in all-cause mortality prediction on three broad datasets: Code-15, MIMIC-IV, and BCH. We provide verifiable results and baselines that inform ECG-processing and survival-analysis approaches for future AI-ECG modeling. We also release our code, which can be adapted to evaluate new modeling approaches or used to train AI-ECG survival-analysis models. This can also be applied beyond mortality modeling to model event occurrence (e.g. first diagnostic threshold crossing) from data in the [time-to-event, event] format. Our work may support future studies that develop, implement, and test AI-ECG models that prospectively inform patient care and case management. Deep survival approaches outperform Classifier-Cox approaches In this study we find Deep Survival approaches work better than Classifier-Cox approaches in two respects. First, the median Deep Survival models significantly increase model concordance over Classifier-Cox models. Second, deep survival approaches do not require a time horizon to be specified, and classifier-cox models exhibit performance sensitivity to this horizon choice. Our results indicate that horizon choice is both dataset-specific, with opposing trends for Code-15 and MIMIC-IV, and can substantially impact performance with the wrong choice. For example, short-horizon data includes few positive-event samples but assumptions about censored patients have little effect. Meanwhile, long-horizon data has more positive-event samples but the assumptions about censored patients have additional time to compound. Furthermore, when training classifiers it is necessary to combine (Time-to-Event, Event) data into a single variable to train a classifier (we discuss our approach in the Supplement). This choice can have substantial effects on what data models use (e.g. MIMIC-IV data is censored to 1 year unless linked to follow-up measures or a mortality). Deep Survival modeling does not require such a choice. Comparisons to past work and model generalizability Our results confirm the performance of ECG mortality risk prediction models from past studies at the one-year mark (0.81–0.85 concordance) [4, 5] and five-year mark (concordance: 0.78–0.83) [1, 5]. Additionally, our ResNet DeepHit, ECG-only MIMIC-IV results (concordance: 0.77) concur with a recent study that trained a ResNet DeepHit model, dubbed “AIRE” [10], on 1.2 M, 10-second ECGs from Beth-Israel Deaconess Medical Center in Boston, MA (concordance: 0.775). The results stand to reason as we expect this dataset to largely overlap with MIMIC-IV. Prior work developing AI-ECG mortality prediction models reports performance deterioration of different degrees across sites, from 0 to 6% AUROC differences (SEER [1]), to 1–18% concordance drops (AIRE [10]). Follow-up evaluations of SEER indicate larger performance deterioration on the UK Biobank cohort (UKB AUROC 0.57 [10], vs. 0.83 in Stanford cohort [1]). Our results confirm that AI-ECG models trained on site-specific data struggle to generalize to new sites, across AI-ECG models, and further suggest that these differences are not fully explained by demographic differences between cohorts (Figure S3) nor mortality rate differences (Table S2). However, by training on multiple sites, our study demonstrates that performance improves across cohorts and, importantly, outperforms site specific models. The importance of baseline models and demographic covariates Past work rarely compares AI-ECG to simpler models, such as demographic-only models, as we do here, yet AI-ECG’s value depends on its favorable performance to less complex baselines. Our results have important implications for studies reporting generalization performance on Code-15 [15], where demographic-only baseline models (median concordance: 0.79) outperform the AI-ECG models tested here, as well as those in prior work (e.g. AIRE concordance: 0.76 [10]). We conjecture that AI-ECG models may have more value in acute care settings (i.e., MIMIC-IV, BCH) than in outpatient settings (Code-15) due to the clinical context. In general, our results strongly support the include of age and sex covariates in all AI-ECG mortality prediction models. Towards clinical translation We hope that this benchmark can assist in AI-ECG modeling by characterizing higher-performing ECG mortality modeling approaches, giving expected ranges for in-dataset and cross-dataset evaluation, and enabling easier model training and evaluation through publicly released code. In turn, these models could help develop tools for automated patient risk estimation: just as current systems flag potential anomalies, patients could be flagged for high estimated risk, potentially informing intervention or follow-up timing (i.e., determining how closely to follow patients). Limitations While we conducted broad sweeps of model configurations in this study, they are not exhaustive. We cannot rule out that other model architectures, or fine-grained tuning of the architectures we benchmark, may improve performance on specific datasets. Furthermore, our results are specific to mortality risk prediction from AI-ECGs, which is one of several prediction targets of interest. Future work could expand the clinical interpretation of model behavior to understand specific pathologies leading to mortality, which we do not inspect deeply here. As a retrospective benchmark, this study may not fully capture deployment considerations; the ethical deployment of survival models in a clinical setting requires that model performance be evaluated in those contexts, with appropriate oversight and a focus on how model use impacts patient care in specific clinical workflows. Conclusions and summarized recommendations Based on this study, we recommend the following for AI-ECG model development: (1) including demographic covariates; (2) using deep survival, rather than classification, approaches; (3) the 1-D ResNet architecture; (4) supplementing local model training with data from additional sites. These model choices generally provide better performance, avoid classifier difficulties related to horizon choice and data censoring, and rely on a well-vetted architecture. Taken together, these design choices can be responsible for most of the improvements AI-ECG offers over simpler baseline models, which we measure to be 5–25% median concordance improvement, depending on cohort. We hope this study and its recommendations provide a valuable resource for future ECG and ECG-mortality studies. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 Acknowledgements We would like to thank Lorenzo Peracchio for help with data validation. Author contributions Conceptualization, P.L., J.M., W.G.L.; methodology, P.L., W.G.L.; investigation, P.L., W.G.L.; writing—original draft, P.L., W.G.L.; writing—review & editing, J.M., S.G., J.K.T., W.G.L.; funding acquisition, J.K.T.; resources, W.G.L.; supervision, J.M., J.K.T., S.G., W.G.L. Funding P.L., and W.G.L. were partially supported by the Kostin Innovation Fund at Boston Children’s Hospital. W.G.L. was partially supported by National Institutes of Health grant R01LM014300. Data availability This paper analyzes two existing, publicly available datasets, accessible at:- MIMIC-IV-ECG: [10.13026/4nqg-sb35] under the Open Data Commons Open Database License v1.0- Code-15: [10.5281/zenodo.4916206], under Creative Commons license CC-BY 4.0- Both datasets need preprocessing to align to the 7-second Code-15 format, available in the repository below.- The BCH data reported in this study cannot be deposited in a public repository because of Institutional Review Board policies to protect pediatric patient privacy. To request access, contact the corresponding author.- Benchmarking and data preprocessing scripts are available at https://github.com/cavalab/ecg-survival-benchmark and is publicly available under the Gnu Public License v3.- The top deep survival models for MIMIC-IV and Code-15 cohorts are available on Zenodo at 10.5281/zenodo.16877773. Declarations Ethics approval and consent to participate The BCH data study used here was approved for a waiver of informed consent by Boston Children’s Hospital Institutional Review Board (IRB-P00044967). Consent for publication N/A. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Supplementary Materials Supplementary Material 1 Data Availability Statement This paper analyzes two existing, publicly available datasets, accessible at:- MIMIC-IV-ECG: [10.13026/4nqg-sb35] under the Open Data Commons Open Database License v1.0- Code-15: [10.5281/zenodo.4916206], under Creative Commons license CC-BY 4.0- Both datasets need preprocessing to align to the 7-second Code-15 format, available in the repository below.- The BCH data reported in this study cannot be deposited in a public repository because of Institutional Review Board policies to protect pediatric patient privacy. To request access, contact the corresponding author.- Benchmarking and data preprocessing scripts are available at https://github.com/cavalab/ecg-survival-benchmark and is publicly available under the Gnu Public License v3.- The top deep survival models for MIMIC-IV and Code-15 cohorts are available on Zenodo at 10.5281/zenodo.16877773.