Bayesian sample size calculations for external validation studies of risk prediction models
Mohsen Sadatsafavi, Paul Gustafson, Solmaz Setayeshgar, Laure Wynants, Richard D. Riley · 2025
Read Bayesian sample size calculations for external validation studies of risk prediction models by Mohsen Sadatsafavi, Paul Gustafson, Solmaz Setayeshgar, Laure Wynants, Richard D. Riley (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15923.txt.
Introduction
Five authors, Mohsen Sadatsafavi, Paul Gustafson, Solmaz Setayeshgar, Laure Wynants, and Richard Riley, wrote this 2025 statistics paper on sizing validation studies for risk models. A risk model must be tested on new data before clinics trust it. Standard recipes for the test size ask users to fix the model's expected performance in advance.
That step hides real doubt, since past studies give only a rough sense of true performance. The authors propose a Bayesian frame that carries that doubt into the size choice. For measures such as discrimination and calibration, their rules target either an expected level of precision or a set chance of reaching it. They also treat net benefit, a measure of clinical use, where standard precision logic is shaky. The work aims at validation studies that are neither too small to trust nor larger than needed.
Catalog record
| CREATOR | Mohsen Sadatsafavi, Paul Gustafson, Solmaz Setayeshgar, Laure Wynants, Richard D. Riley |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15923
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bayesian sample size calculations for external validation studies of risk prediction models Mohsen Sadatsafavi, Paul Gustafson, Solmaz Setayeshgar, Laure Wynants, Richard D. Riley Abstract Contemporary sample size calculations for external validation of risk prediction models require users to specify fixed values of assumed model performance metrics alongside target precision levels (e.g., 95% CI widths). However, due to the finite samples of previous studies, our knowledge of true model performance in the target population is uncertain, and so choosing fixed values represents an incomplete picture. As well, for net benefit (NB) as a measure of clinical utility, the relevance of conventional precision-based inference is doubtful. In this work, we propose a general Bayesian framework for multi-criteria sample size considerations for prediction models for binary outcomes. For statistical metrics of performance (e.g., discrimination and calibration), we propose sample size rules that target desired expected precision or desired assurance probability that the precision criteria will be satisfied. For NB, we propose rules based on Optimality Assurance (the probability that the planned study correctly identifies the optimal strategy) and Value of Information (VoI) analysis. We showcase these developments in a case study on the validation of a risk prediction model for deterioration of hospitalized COVID-19 patients. Compared to the conventional sample size calculation methods, a Bayesian approach requires explicit quantification of uncertainty around model performance, and thereby enables flexible sample size rules based on expected precision, assurance probabilities, and VoI. In our case study, calculations based on VoI for NB suggest considerably lower sample sizes are needed than when focusing on precision of calibration metrics. † † footnotetext: From Faculty of Medicine and Faculty of Pharmaceutical Sciences, The University of British Columbia (MS); Department of Statistics, The University of British Columbia (PG); British Columbia Centre For Disease Control (SS); Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, and Department of Development and Regeneration, KU Leuven (LW); Department of Applied Health Sciences, School of Health Sciences, College of Medicine and Health, University of Birmingham, and National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre (RDR) † † footnotetext: \dagger Co-senior authors with equal contribution † † footnotetext: * Correspondence to Mohsen Sadatsafavi, Room 4110, Faculty of Pharmaceutical Sciences, 2405 Wesbrook Mall, Vancouver, BC, V6T1Z3, Canada; email: [email protected] Introduction Developing risk prediction models or validating existing ones in a new population represents significant investment in time, resources, and expertise. Like other empirical experiments, the design of such studies should be based on objective, transparent, and defendable principles. A particular aspect of study design is the sample size of such studies. The field has witnessed significant recent developments on this front [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] . An example is the multi-criteria approach by Riley et al for the sample size required for external validation studies, targeting pre-specified widths of the 95% confidence intervals around metrics of model performance [ 2 , 3 , 9 ] . For binary outcomes, such metrics are related to discrimination, calibration, and net benefit (NB). The sample size required for each component is computed separately, with the largest one advertised as the final requirement [ 3 ] . In addition to target precision criteria, each component of this approach requires as input an assumed true value of the metric of interest in the target population. In reality, we do not know the true value of model performance metrics with certainty (otherwise there would be no need to conduct a validation study in the first place). Further, due to sampling variability, the precision obtained in one particular dataset may differ substantially from the expected precision for that sample size. Thus, there is more uncertainty to be accounted for than the Riley criteria allow. A Bayesian approach towards sample size determination allows this uncertainty to be accounted for, which is advantageous on multiple fronts. First, it enables the full use of existing information on model performance at the time the study is designed, rather than forcing the investigator to express their knowledge as the fixed truth. Second, it enables users to choose from different classes of sample size rules, namely those that target the expected values of precision targets (as in Riley criteria), as well as ‘assurance’-type rules for the probability of meeting (or exceeding) precision targets. The latter can be insightful as focusing on expected values alone does not guarantee that the desired precision will be achieved in one particular dataset, and the investigator may want a stronger assurance against this. For example, an investigator that desires an interval width of 0.1 for the c-statistic might not perceive much benefit from the future interval being narrower, but might have a strong preference against not meeting this target. As such, targeting a sample size that will result in, say, a 90% probability that the CI width will be \leq 0.1 might be more appealing than a sample size that targets an expected CI width of 0.1. Finally, when assessing clinical utility, the relevance of precision-based criteria is challenged [ 10 , 11 ] . A Bayesian approach enables the use of novel, decision-theoretic approaches based on Value of Information (VoI) analysis, which focuses on the expected gain in clinical utility from increasing sample size [ 12 ] . Bayesian approaches for power and sample size calculations are mostly investigated in the context of experimental studies that are aimed at interrogating a null hypothesis [ 13 , 14 , 15 , 16 ] . However, risk model validation studies are not generally hypothesis-driven. Rather, the focus is on the precise estimation of a variety of metrics of model performance. As such, Bayesian sample size considerations in this context are worth exploring. Hence, in this article, we propose a Bayesian version of the sample size formula by Riley et al, which enables the investigator to 1) incorporate their uncertainty around assumed model performance in calculations, 2) use sample size rules that target assurance probabilities for meeting chosen criteria, and 3) use rules based on VoI analysis for clinical utility. The proposed framework can be used in two general ways: to quantify the anticipated precision or VoI outcomes from a planned study if the sample size is fixed (as is the case with validation studies that are based on already collected data), or to determine the minimum sample size that achieves pre-specified precision or VoI criteria. This framework can also be used in a hybrid form: determine the sample size based on certain criteria (e.g., targeting CI width for c-statistic and calibration slope), and investigate the consequence of the chosen sample size using other criteria (e.g, assurance probability around NB for the final sample size). Calculations can be done for the entire sample, as well as within subgroups, for example, imposing fairness criteria around the target precision of estimates among minority sub-groups. The rest of this manuscript is structured as follows. First, we briefly review common metrics of model performance and multi-criteria sample size formula by Riley et al [ 3 ] . We will then introduce our Bayesian extension of this framework based on various sample size rules. We propose a general approach for characterizing uncertainty