Bridging the Generalisation Gap: Synthetic Data Generation for Multi-Site Clinical Model Validation
Bradley Segal, Joshua Fieggen, David A. Clifton, Lei Clifton · 2025
Read Bridging the Generalisation Gap: Synthetic Data Generation for Multi-Site Clinical Model Validation by Bradley Segal, Joshua Fieggen, David A. Clifton, Lei Clifton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20635.txt.
Introduction
Clinical machine learning models often fail when moved to a new hospital, because patient mix, disease rates, and local habits differ. Real data sets are scarce, carry hidden biases, and cannot be reshaped at will for testing. Generative models that mimic real data add little control over what drives the shift. The four authors propose a structured synthetic data framework that gives explicit control over the data-making process, including site-specific disease rates, subgroup effects, and feature links.
With that control, teams can test how a model reacts to one shift at a time, audit fairness, and spot failures in generalization. Their experiments show the framework can isolate site effects and reveal how model complexity interacts with site-specific quirks. The paper offers the tool as a reproducible, interpretable way to check robustness and fairness before a clinical model is trusted in a new setting.
Catalog record
| CREATOR | Bradley Segal, Joshua Fieggen, David A. Clifton, Lei Clifton |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20635
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bridging the Generalisation Gap: Synthetic Data Generation for Multi-Site Clinical Model Validation Bradley Segal, Joshua Fieggen, David A. Clifton, Lei Clifton Abstract Ensuring the generalisability of clinical machine learning (ML) models across diverse healthcare settings remains a significant challenge due to variability in patient demographics, disease prevalence, and institutional practices. Existing model evaluation approaches often rely on real-world datasets, which are limited in availability, embed confounding biases, and lack the flexibility needed for systematic experimentation. Furthermore, while generative models aim for statistical realism, they often lack transparency and explicit control over factors driving distributional shifts. In this work, we propose a novel structured synthetic data framework designed for the controlled benchmarking of model robustness, fairness, and generalisability. Unlike approaches focused solely on mimicking observed data, our framework provides explicit control over the data generating process, including site-specific prevalence variations, hierarchical subgroup effects, and structured feature interactions. This enables targeted investigation into how models respond to specific distributional shifts and potential biases. Through controlled experiments, we demonstrate the framework’s ability to isolate the impact of site variations, support fairness-aware audits, and reveal generalisation failures, particularly highlighting how model complexity interacts with site-specific effects. This work contributes a reproducible, interpretable, and configurable tool designed to advance the reliable deployment of ML in clinical settings. I INTRODUCTION Machine learning (ML) models for healthcare must generalise across diverse clinical settings, yet real-world deployment often reveals substantial degradation in performance due to shifts in patient demographics, disease prevalence, and site-specific practice patterns [ 1 , 2 , 3 ] . Such challenges, vividly illustrated by failures in transporting models for tasks like COVID-19 screening between hospitals [ 1 ] or predicting emergency room dispositions across different facilities [ 2 ] , are exacerbated in multi-site clinical studies. Addressing these issues requires robust validation strategies that systematically account for distributional shifts and site heterogeneity. Traditional model validation relies on real-world multi-site datasets. However, real data access is often limited by privacy restrictions, data-sharing limitations, cost, inherent biases [ 4 ] , and crucially, they hinder systematic isolation of failure modes under varying conditions. Existing synthetic data approaches, such as generative adversarial networks (GANs) and variational autoencoders (VAEs), aim to replicate the statistical properties of real-world data [ 5 ] . However, these methods often lack explicit control over distributional factors like site-specific prevalence or structured feature interactions crucial for rigorously evaluating model generalisability and fairness mitigation strategies [ 6 ] . There is a need for tools that bridge this gap, offering control alongside plausible data generation. To address these challenges, we propose a structured synthetic data framework designed to enable explicit control over key factors influencing model generalisability, including site-specific variations, hierarchical subgroup effects, and feature interactions. Unlike generative approaches that prioritise statistical realism often at the cost of transparency, our method allows for targeted benchmarking of model generalisability, fairness, and robustness in simulated clinical settings. Our framework provides configurability over: multi-site data generation with user-defined prevalence variations (e.g., simulating differing disease burden across hospitals); hierarchical subgroup effects based on demographic factors (e.g., modelling age-sex specific risk profiles); complex feature interactions, including non-linear dependencies; temporal dependencies and structured missing data patterns; and explicit modelling of feature-outcome relationships, providing a known ground truth for validation. By enabling structured control over these elements, the framework allows researchers to systematically study model performance under specific, challenging conditions relevant to real-world deployment. While this structured approach prioritises interpretability and control over potentially replicating every unknown statistical nuance present in real data, this focus is crucial for systematically studying the mechanisms underlying generalisation failures. This emphasis on transparency makes the framework particularly useful for benchmarking models, evaluating domain adaptation techniques, and conducting fairness-aware audits. A detailed comparison with alternative approaches is provided in Section II . Our contributions in this work include: • A configurable framework for generating multi-site clinical data with explicit control over site-specific variations, subgroup effects, and feature interactions relevant to generalisability. • Systematic evaluation demonstrating how controlled variations in site-specific interactions impact model generalisability, enabling the assessment of external validation performance under specific distributional shifts. • Analysis illustrating the framework’s utility in examining the relationship between model complexity and generalisability across sites. • An open-source implementation facilitating reproducible benchmarking of clinical ML models under controlled, challenging conditions. We validate our framework through controlled experiments that assess its ability to preserve feature-outcome relationships, capture realistic site-level variations, and provide actionable insights into model robustness. The remainder of this paper is organised as follows: Section II discusses related work, Section III describes the framework’s technical architecture, Section IV presents validation experiments, and Section V discusses broader applications, limitations, and future extensions. II Related Work The challenge of generating realistic and useful synthetic clinical data has spurred research across several distinct approaches. Our work builds upon and differentiates itself from these existing methodologies, particularly concerning the objective of evaluating model generalisability across diverse clinical settings. II-A Generative Models Deep generative models (GANs [ 7 ] , VAEs [ 8 ] ) can achieve high statistical fidelity by learning complex, high-dimensional distributions found in real clinical data [ 9 , 10 ] . Their strength lies in automatically capturing intricate correlations without explicit feature engineering. However, they typically lack mechanisms for explicit, fine-grained control over specific factors critical for generalisability testing, such as site-specific outcome prevalence, interaction strengths, or the precise nature of subgroup effects [ 6 ] . Systematically varying one factor or simulating specific counterfactuals can be difficult, hindering targeted simulation of distributional shifts relevant to multi-site validation. Our framework contrasts this by prioritising explicit, configurable control over data-generating mechanisms. II-B EHR Simulation Tools Tools like Synthea [ 11 ] focus on simulating realistic Electronic Health Record (EHR) structures and longitudinal patient journeys based on health statistics and clinical guidelines. While valuable for tasks requiring realistic EHR workflows (e.g., testing data pipelines), they are generally not designed for benchmarking ML model generalisability under controlled statistical variations. Their primary focus is plausible patient pathways, not providing the fine-grained control over distributions, feature-outcome relationships, site effects, or interaction strengths needed to systematically probe ML model robustness to known challenges like prevalence shifts or demographic bias. Our framework complements these tools by focusing on datasets tailored for ML validation with known ground truth. II-C Real Multi-Site Data Validation Validating models on large, real-world multi-site datasets [ 4 ] is the gold standard but faces significant practical hurdles: patient privacy concerns (e.g., GDPR, HIPAA), data sharing agreements, curation/harmonisation costs, and ethical approvals [ 4 ] . Furthermore, real data inherently contains confounding factors and unknown biases, making it difficult to isolate specific reasons for performance degradation or perform controlled experiments varying only one factor (e.g., site prevalence). Our framework provides a complementary approach: a controlled, accessible, reproducible environment for systematic pre-validation and benchmarking. II-D Procedural and Rule-Based Generation Simpler procedural or rule-based methods are fully interpretable but often struggle to capture the complex correlations, non-linearities, and interaction effects present in real clinical data. Our framework builds on this transparency but incorporates more sophisticated effect modelling, including interactions, non-linearities, and hierarchical effects, offering greater flexibility and realism for simulating complex clinical scenarios relevant to ML validation. In summary, our framework occupies a unique