Robustifying and Selecting Cohort-Appropriate Prognostic Models under Distributional Shifts
Dimitris Bertsimas, Carol Gao, Angelos Koulouras, Georgios Antonios Margonis · 2026
Read Robustifying and Selecting Cohort-Appropriate Prognostic Models under Distributional Shifts by Dimitris Bertsimas, Carol Gao, Angelos Koulouras, Georgios Antonios Margonis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16537.txt.
Introduction
Bertsimas and colleagues challenge the idea that external calibration guarantees transfer. Using six surgery cohorts from top centers, they test whether fit depends on cohort similarity. Calibration worsened as mismatch grew, with KL gap and the Calibration Index. From the builder side, they tune a model toward a meta-analysis target spread. From the user side, they propose a cohort-outcome similarity measure to pick the model for a cohort. Weighting improved fit in most settings without hurting ranking, and similar-cohort models gave lower error and greater care value.
Catalog record
| CREATOR | Dimitris Bertsimas, Carol Gao, Angelos Koulouras, Georgios Antonios Margonis |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56344 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.16537
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robustifying and Selecting Cohort-Appropriate Prognostic Models under Distributional Shifts Dimitris Bertsimas, Carol Gao, Angelos Koulouras, Georgios Antonios Margonis Abstract Introduction: External validation is widely regarded as the gold standard for prognostic model evaluation. In this study, we challenge the assumption that successful external validation of calibration guarantees model generalizability and propose two complementary strategies to improve the transportability of prognostic models across cohorts. Methods: Using six real-world surgical cohorts from tertiary academic centers, we first tested the hypothesis that successful external calibration does not necessarily guarantee model generalizability, but instead depends largely on similarity in covariates and outcomes between the development and validation cohorts. Cohort similarity was quantified using Kullback–Leibler (KL) divergence, and calibration was assessed using the Integrated Calibration Index (ICI). Second, from the model-developer’s perspective, we aimed to train the “best-on-average” prognostic model by tuning models toward a meta-analysis-derived covariate and outcome distribution, treating this distribution as an approximation of the broader target population, and then evaluating whether this improved calibration. Third, from the end-user perspective, we proposed a simple measure of cohort outcome similarity to identify, among available published models, the prognostic model most suitable for a given target cohort with respect to both calibration and clinical utility, as assessed by Decision Curve Analysis (DCA). Results: External calibration worsened as distributional mismatch increased. Higher KL divergence was associated with higher ICI in both surgery-alone (Spearman ρ = 0.614 , p = 0.004 \rho=0.614,p=0.004 ) and surgery + adjuvant chemotherapy cohorts (Spearman ρ = 0.738 , p < 0.001 \rho=0.738,p<0.001 ). Meta-analysis-informed weighting improved calibration in most settings without materially affecting discrimination, with the clearest benefit observed when performance was evaluated against the aggregated external population (p=0.037), consistent with the goal of approximating a broader target population. From the end-user perspective, models developed in cohorts more similar to the target cohort achieved lower ICI in both surgery alone (Spearman ρ = 0.803 , p < 0.001 \rho=0.803,p<0.001 ) and surgery + adjuvant chemotherapy cohorts (Spearman ρ = 0.737 , p < 0.001 \rho=0.737,p<0.001 ), and also provided greater clinical utility on DCA. Conclusion: Successful external calibration depends strongly on the similarity between development and validation cohorts and therefore does not, by itself, guarantee model generalizability to additional settings. We propose one strategy to train the “best-on-average” prognostic model and another to help users select the “best-for-their-cohort” prognostic model. 1 Introduction Accurate prognostication is one of the pillars of clinical practice. Prognostic model outputs are commonly used to inform patients about what to expect after treatment, to guide physicians in deciding whether treatment escalation is warranted (for example, surgery plus chemotherapy vs surgery alone), and to support shared decision-making between physicians and patients. Accordingly, evaluating the performance of prognostic models is a central task in clinical outcome research. The current standard for prognostic model evaluation includes two complementary dimensions. The first is discrimination, that is, the ability of the model to correctly rank patients according to their risk. In survival analysis, this is typically assessed using metrics such as the concordance index (C-index) or time-dependent area under the curve (AUC). The second, and arguably even more clinically relevant, dimension is calibration: whether the absolute risk estimates assigned to individual patients, such as the predicted probability of 5-year survival, are accurate. 1 The most informative assessment of both discrimination and calibration occurs in an independent external cohort from another center, where good performance is generally interpreted as evidence that the model can generalize beyond the cohort used to develop it. Nonetheless, calibration is not purely an intrinsic property of the model. Rather, it depends in part on the cohort in which the model is evaluated. In clinical outcome research, external validation is often limited to one or, at most, two outside cohorts. However, successful validation in such a small number of datasets may be misleading: performance may appear favorable simply because the external cohort happens to share a similar underlying risk distribution with the training cohort, rather than because the model is truly generalizable. In practical terms, this challenges the credibility of many published prognostic tools, particularly those supported by only one or two external validations. This problem is especially important because, in most areas of medicine, the factors that determine patient outcomes are only incompletely understood. As a result, the risk distributions of different cohorts are not known a priori. Consequently, when a model performs well in an outside cohort, it often remains unclear whether this reflects genuine robustness or merely favorable alignment between the derivation and validation populations. This raises a fundamental question: is it truly feasible to develop a single prognostic model that generalizes well across all settings, or should model selection be tailored to the specific characteristics and needs of the target cohort? In this study, we address the problem both from the perspective of the model developer, who aims to build a robust and generalizable model across various cohorts, and from the perspective of the end user, who wishes to identify the most appropriate available model for their specific real-world cohort, even when that model is not the best on average. Our contributions are as follows: 1. We demonstrate empirically on 6 real-world cohorts that successful external validation does not necessarily imply generalization and provide a theoretical justification. 2. We develop a prognostic weighting approach that is more robust to distribution shifts. Specifically, we show across a range of external cohorts that the models weighted towards a meta-analysis distribution, on average, perform better in calibration than competing models. 3. We provide a structured framework for selecting, among the models available in the literature, the one that best fits the prognostic needs of a given external cohort. 2 Methods The Methods are organized into five parts. First, we describe the study cohorts used for empirical evaluation. Second, we describe and formulate the problem of distribution shifts we attempt to solve. Third, from the perspective of the model developer, we present a framework to train prognostic models that are more robust to distributional shifts across cohorts. Fourth, from the perspective of the end user, we describe a framework for selecting the prognostic model that best fits a specific target cohort. Lastly, we summarize the statistical analyses used to evaluate model performance and clinical utility. 2.1 Study cohorts As noted above, the aim of this study was not to address a specific clinical problem, such as prognostication in a single disease setting, but rather to propose: (1) a structured approach for training the best average prognostic model from the perspective of the model developer, and (2) a framework for selecting, from the perspective of the end user, the prognostic model from the literature that best fits the needs of a given target cohort. To demonstrate the proposed methodology, we performed empirical analyses using high-quality data from tertiary institutions across distant geographic regions, including centers in the United States, Europe, and Japan. All cohorts consisted of patients undergoing surgery for colorectal liver metastases (CRLM), thereby providing a clinically coherent setting in which to evaluate model transportability across different populations. Specifically, we included patients who underwent surgical treatment for colorectal liver metastases at six academic institutions: Johns Hopkins Hospital (JHH, USA), Cleveland Clinic Foundation (CCF, USA), Erasmus University of Rotterdam