Enabling Granular Subgroup Level Model Evaluations by Generating Synthetic Medical Time Series
Mahmoud Ibrahim(✉), Bart Elen, Chang Sun, Gökhan Ertaylan, Michel Dumontier · 2025
Read Enabling Granular Subgroup Level Model Evaluations by Generating Synthetic Medical Time Series by Mahmoud Ibrahim(✉), Bart Elen, Chang Sun, Gökhan Ertaylan, Michel Dumontier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19728.txt.
Introduction
Mahmoud Ibrahim and four co-authors show that synthetic intensive-care data can be trusted not just to train medical prediction models, but to score them fairly across patient groups. Real test sets are small. So scores for fine-grained groups are hard to measure without exposing private records. Their Enhanced TimeAutoDiff maker adds alignment penalties to a latent diffusion model.
Tested on two large critical-care databases for death and length-of-stay prediction, the improved maker shrank the gap between synthetic-based and real-based scoring by more than seventy percent. For thirty-two cross-cutting subgroups, large synthetic cohorts cut scoring error by up to half, compared with small real test sets. The authors frame the method as a privacy-safe roadmap for trustworthy medical AI checks.
Catalog record
| CREATOR | Mahmoud Ibrahim(✉), Bart Elen, Chang Sun, Gökhan Ertaylan, Michel Dumontier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19728
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enabling Granular Subgroup Level Model Evaluations by Generating Synthetic Medical Time Series Mahmoud Ibrahim(✉), Bart Elen, Chang Sun, Gökhan Ertaylan, Michel Dumontier Abstract We present a novel framework for leveraging synthetic ICU time-series data not only to train but also to rigorously and trustworthily evaluate predictive models, both at the population level and within fine-grained demographic subgroups. Building on prior diffusion and VAE-based generators (TimeDiff, HealthGen, TimeAutoDiff), we introduce Enhanced TimeAutoDiff, which augments the latent diffusion objective with distribution-alignment penalties . We extensively benchmark all models on MIMIC-III and eICU, on 24-hour mortality and binary length-of-stay tasks. Our results show that Enhanced TimeAutoDiff reduces the gap between real-on-synthetic and real-on-real evaluation (“TRTS gap”) by over 70 %, achieving Δ T R T S ≤ \Delta_{TRTS}\leq 0.014 AUROC, while preserving training utility ( Δ T S T R ≈ \Delta_{TSTR}\approx 0.01). Crucially, for 32 intersectional subgroups, large synthetic cohorts cut subgroup-level AUROC estimation error by up to 50 % relative to small real test sets, and outperform them in 72–84 % of subgroups. This work provides a practical, privacy-preserving roadmap for trustworthy, granular model evaluation in critical care, enabling robust reliable performance analysis across diverse patient populations without exposing sensitive EHR data, contributing to the overall trustworthiness of Medical AI. 1 Introduction Machine learning in critical care increasingly depends on large-scale ICU time-series data (continuous and irregular measurements of vitals, labs, and interventions) to develop predictive tools such as early-warning systems, mortality risk estimators, and length-of-stay predictors. Despite extensive repositories like eICU [ 2 ] and MIMIC [ 1 ] , data sharing remains constrained by privacy regulations and limited access for underrepresented populations. Synthetic ICU time-series data offer a promising solution by providing realistic, privacy-preserving alternatives to real patient records. Recent advances in generative models, from GANs [ 6 ] to diffusion frameworks [ 10 ] , have shown strong fidelity in reproducing complex temporal patterns of physiological data. Beyond enabling model training without privacy exposure, synthetic data can also support evaluation when access to real test data is restricted. For instance, a model trained on institutional EHRs may need to be evaluated externally by regulators or collaborators, without sharing the real data. In such cases, a synthetic hold-out set should faithfully approximate the real test distribution so that performance metrics (e.g., AUROC, RMSE) on synthetic samples reflect true performance on real patients. Most prior work, however, emphasizes training utility: the “Train on Synthetic, Test on Real” (TSTR) setup while neglecting evaluation utility, or “Train on Real, Test on Synthetic” (TRTS). For synthetic data to be a reliable test substitute, models evaluated on synthetic hold-outs must achieve performance comparable to those tested on genuine real data. Beyond global model evaluation, a third, and often overlooked use case for synthetic ICU data is subgroup-level evaluation. ICU cohorts are inherently heterogeneous: patients differ in age, gender, ethnicity, comorbidities, socioeconomic status, and other factors. A high-impact clinical question is “How does my mortality prediction model perform on patients aged over 75, or on Black patients, or on females, or on black females aged over 75?” In this work, the focus is on proper model evaluation, specifically on addressing algorithmic bias through performance analysis on diverse population subgroups, which is recommended by [ 13 ] . Unfortunately, real EHR datasets frequently contain only a few hundred samples (or fewer) in any given fine-grained subgroup. Evaluating a downstream model on such a small test set yields wide confidence intervals and inaccurate evaluations that mask how a model truly performs on that subgroup. Synthetic data, when conditioned on subgroup attributes, can overcome this limitation by generating large, representative cohorts for fine-grained intersectional groups. This enables precise subgroup evaluation with narrow confidence intervals, improving transparency and fairness. Ultimately, such use of synthetic data contributes to more trustworthy and equitable AI systems , where performance metrics truly reflect clinical reality across all patient populations. Main Contributions This work focuses on using synthetic ICU time-series data for robust and trustworthy model evaluation, both globally and across demographic subgroups. Our goal is to ensure that evaluation metrics computed on synthetic data faithfully reflect true performance on real ICU patients. To this end, we develop an enhanced generative framework and systematically compare it to existing models: TimeDiff, TimeAutoDiff, and HealthGen regarding their ability to produce synthetic data that serve as reliable test proxies. 1. Enhanced TimeAutoDiff for Evaluation. We introduce an improved version of the latent diffusion model TimeAutoDiff (optimized for model evaluation purposes), augmented with two distribution-alignment losses (MMD and consistency regularization). These additions reduce the TRTS (Train on Real, Test on Synthetic) performance gap by over 70% while maintaining TSTR (Train on Synthetic, Test on Real) stability. 2. Benchmarking Synthetic Hold-Outs. We systematically compare Enhanced TimeAutoDiff, TimeDiff, TimeAutoDiff, and HealthGen to assess which generators produce synthetic hold-outs that most closely reproduce “real-on-real” (TRTR) evaluation metrics: which synthetic hold-outs can be most reliably trusted as proxies for real test data. 3. Subgroup-Level Evaluation. We evaluate whether synthetic data can support fine-grained, demographically stratified analyses by generating large, conditionally sampled subgroups, especially where real test data are insufficient for reliable statistics. 4. Open-Source Release. All code, model checkpoints, and subgroup-evaluation pipelines are publicly available at: github.com/mahmoudibrahim98/icu-auto-diff . The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 summarizes the used architectures. Section 4 details the methodology and the experimental setup .Section 5 presents the experimental results. Section 6 discusses implications and limitations. Finally, Section 7 concludes. 2 Related Work Very few studies explicitly use synthetic data for model evaluation or for assessing performance across intersectional subgroups. A structured regression framework was introduced in [ 14 ] . It pools information across demographic intersections to produce stable, confidence-aware subgroup accuracy estimates. SYNG4ME [ 15 ] similarly leverages synthetic cohorts to yield more reliable subgroup metrics than small held-out real subsets. However, this work does not address time-series data or medical datasets. Synthetic EHR generation must account for the inherent heterogeneity of real-world ICU records, which combine static demographics, high-cardinality categorical codes (e.g., ICD), and multivariate vital-sign time series. Broadly, researchers distinguish four EHR formats: snapshot, aggregated, longitudinal, and time-dependent, each presenting unique modeling challenges [ 12 ] ; our work focuses on the time-dependent setting. Early efforts like TimeGAN [ 5 ] adapted the GAN framework to capture temporal dynamics, while EHR-M-GAN [ 6 ] introduced separate encoders for each data modality to better synthesize mixed-type sequences. More recently, HALO [ 7 ] leveraged a hierarchical autoregressive language model to generate medical codes and visit sequences over long horizons, and HealthGen [ 8 ] adopted a VAE to produce continuous ICU trajectories. Diffusion-based approaches have also emerged: Diffusion-TS [ 9 ] uses a transformer-based latent diffusion to model complex time series, and TimeDiff [ 10 ] combines multinomial and Gaussian diffusion processes specifically for EHR data. Finally, FLEXGEN-EHR [ 11 ] and TimeAutoDiff [ 4 ] employs latent diffusion to synthesize heterogeneous longitudinal records end-to-end. It is noted that there is a noticeable lack of emphasis on conditional generation and incorporating patient and demographic context in the generation process. 3 Generative Models 3.1 Base Architecture The base architecture used in