Adaptive Gaussian Process Search for Simulation-Based Sample Size Estimation in Clinical Prediction Models: Validation of the pmsims R Package

Oyebayo Ridwan Olaniran, Diana Shamsutdinova, Sarah Markham, Felix Zimmer, Daniel Stahl, Gordon Forbes, Ewan Carr · 2026

Read Adaptive Gaussian Process Search for Simulation-Based Sample Size Estimation in Clinical Prediction Models: Validation of the pmsims R Package by Oyebayo Ridwan Olaniran, Diana Shamsutdinova, Sarah Markham, Felix Zimmer, Daniel Stahl, Gordon Forbes, Ewan Carr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23688.txt.

Introduction

Olaniran and six co-authors validate pmsims, an R package that sizes clinical prediction studies by simulation. Picking a sample size guards a model's trust, yet sound guidance stays thin. The package fits a Gaussian process stand-in to steer the search for the right size. Tests pit that engine against bisection and a hybrid across three outcome types. The process engine gives the steadiest sizes, most in weak-signal, wide-data cases. Against rival tools, pmsims hits target performance with fewer model runs.

Catalog record

CREATOROyebayo Ridwan Olaniran, Diana Shamsutdinova, Sarah Markham, Felix Zimmer, Daniel Stahl, Gordon Forbes, Ewan Carr
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23680
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23688

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Gaussian Process Search for Simulation-Based Sample Size Estimation in Clinical Prediction Models: Validation of the pmsims R Package Oyebayo Ridwan Olaniran, Diana Shamsutdinova, Sarah Markham, Felix Zimmer, Daniel Stahl, Gordon Forbes, Ewan Carr Abstract Background Determining an adequate sample size is essential for developing reliable and generalisable clinical prediction models, yet practical guidance on selecting appropriate sample size calculation methods remains limited. Existing analytical and simulation-based tools impose restrictive assumptions and focus on mean-based criteria. We present and validate pmsims , an R package that uses Gaussian process (GP) surrogate modelling to provide a flexible and efficient simulation-based framework for sample size determination across a broad range of prediction modelling contexts. Methods We conducted a comprehensive simulation study with two aims. Aim 1 compared three search engines implemented in pmsims , a GP surrogate-based adaptive procedure ( gp ), a deterministic bisection method ( bisection ), and a hybrid GP-bisection approach ( gp-bs ), across binary, continuous, and survival outcomes. Scenarios varied outcome prevalence or event rate, predictor dimensionality ( p = { 10,100 } p=\{10,100\} ), target performance metric (discrimination and calibration slope), aggregation criterion (mean vs 80% assurance), and total simulation budget ( B = 200 − 2000 B=200-2000 ). Each scenario was replicated 100 times; estimator stability was assessed via the coefficient of variation (CV). Aim 2 benchmarked the best-performing pmsims engine against pmsampsize (analytical) and samplesizedev (simulation-based) across a wider range of realistic scenarios, evaluating recommended sample sizes, computational time, and achieved model performance on independent validation datasets of 30,000 observations. Results The GP-based search engine consistently yielded the most stable sample size estimates (lowest CV) across all outcome types, ranking highest in 9/12 outcome-aggregation metric configurations. Its advantage was most pronounced in low-signal, high-dimensional settings, and was accentuated with κ = 20 \kappa=20 replications per evaluation and a budget of B ≈ 1000 B\approx 1000 , after which gains were minimal. In benchmark comparisons, pmsims (mean) achieved performance deviations within ± 1 % \pm 1\% of the prespecified target across binary, continuous, and survival outcomes, comparable to samplesizedev and substantially outperforming pmsampsize in high-discrimination settings (deviations up to − 9.84 % -9.84\% ). Conclusions The pmsims package, using the GP-based search engine with κ ≥ 20 \kappa\geq 20 replications per evaluation and a budget of B ≈ 1000 B\approx 1000 , provides a computationally efficient and flexible framework for principled sample size planning in clinical prediction modelling. It reliably achieves performance targets across a wide range of scenarios while requiring fewer model evaluations than non-adaptive simulation approaches, offering a compelling alternative to both analytical formulae and exhaustive simulation-based search. keywords Sample size planning, prediction modelling, Gaussian process, adaptive simulation, discrimination, calibration, assurance, R package † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction Determining an adequate sample size is an essential prerequisite for developing stable, reliable, and generalisable clinical prediction models. Development samples that are too small can produce overfitted models with poor calibration and reduced discrimination when applied to new data, undermining clinical utility and potentially causing harm in decision-making contexts VanCalster2016 . In prediction modelling, sample size estimation seeks the minimum development sample size needed for model performance to meet or exceed a prespecified threshold on a chosen metric Diana2026pmsims . Targets can be defined using either a ‘mean’ criterion, where expected performance exceeds the threshold, or an ‘assurance’ criterion, where performance exceeds the threshold with high probability (e.g., 80%). The assurance approach explicitly accounts for training-sample variability and is therefore more robust to uncertainty in model development sadatsafavi2025bayesian ; pavlou2025sample . Analytic tools have been developed to estimate minimum sample sizes for traditional regression-based prediction models. For example, the pmsampsize R library and Stata modules target an acceptable degree of overfitting (via shrinkage) or ensure sufficiently precise estimation of the average outcome in the population Riley2020 . However, these approaches are limited to linear or generalised linear models estimated using maximum likelihood and are not generalisable to a wider class of machine learning models or alternative estimation methods such as penalised maximum likelihood. Moreover, they rely on assumptions that are often unmet in practice. In particular, they typically presume a correctly specified linear predictor, normally distributed covariates, and properties of maximum likelihood estimation—assumptions that can be violated in real-world clinical data characterised by non-linearities, interactions, and complex correlation structures VanderPloeg2014 . A further limitation of existing analytical methods is that they focus on the ‘mean’ criterion, which does not capture variability in performance arising from finite training samples. Consequently, a sample size that meets a mean target may still imply a substantial probability of underperformance in the target population—risk not captured by closed-form formulae. Simulation studies have also shown analytic approaches may underestimate the required sample size when model strength is high (C-statistic \geq 0.8) pavlou2024evaluation . To address the limitations of analytic approaches, simulation-based sample size tools have been developed. samplesizedev is a simulation-based R package that estimates sample sizes for binary and survival outcomes, quantifying variability in model performance metrics and supporting probability-based assessment of achieving target performance across training samples (the ‘assurance’ criterion) pavlou2025sample . More recently, Bayesian approaches have also been proposed to support probability-based (assurance-type) sample size criteria for prediction modelling sadatsafavi2025bayesian ; riley2025general ; pavlou2025sample . samplesizedev only supports logistic and Cox regression models, simulates predictors independently from standard normal distributions, and is not readily extendable to other models or data generators. Other approaches to sample size determination include learning curve methods , which extrapolate model performance from pilot data; and hybrid approaches , in which estimated learning curves are summarised as approximate analytical formulae parametrised by the characteristics of the datasets used to derive them. These approaches are reviewed in Diana2026pmsims . Despite these advances, there is a lack of practical, accessible tools for assurance-type sample size determination accommodating diverse model specifications and realistic data-generating mechanisms. This gap presents researchers with an unhelpful choice: rely on simplified approaches that may not reflect the complexities of the intended application or omit sample size justifications altogether. Consistent with this, systematic reviews show that most prediction modelling studies do not report an explicit sample size justification Dhiman2023 . In this paper, we present pmsims , an R package that provides a flexible simulation-based framework for estimating minimum sample sizes for developing clinical prediction models. The package addresses two core limitations of existing analytical approaches: limited flexibility in model and data-generating specifications, and the computational burden of naïve simulation-based searches over large sample size spaces Diana2026pmsims . The pmsims package incorporates two key innovations. First, it uses Gaussian process (GP) surrogate modelling of the sample size–performance relationship zimmer2024sample ; wilson2021efficient . Rather than exhaustively evaluating every candidate sample size via computationally expensive model fitting, the GP-based engine uses a Gaussian process surrogate model to approximate performance as a function of sample size. It iteratively selects the next sample sizes to evaluate by choosing points where the current Gaussian process shows the highest posterior uncertainty (or where an acquisition function that balances predicted performance and uncertainty is maximised), runs a limited number of Monte Carlo simulations at those points, updates the GP posterior with the new

Read the complete text