Sample size considerations for the external validation of a multivariable prognostic model: a resampling study.
Collins GS, Ogundimu EO, Altman DG. · 2016
Read Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. by Collins GS, Ogundimu EO, Altman DG. (2016) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc4738418.txt.
Introduction
Collins, Ogundimu, and Altman tackle a practical gap in medical prediction research: nobody knows how large an external validation study must be. Prognostic models are often published but rarely tested on fresh patients, and small validations give shaky verdicts. Using a large real dataset, they resample patients and watch how six published models' measured performance wobbles with sample size.
Precise, unbiased estimates of discrimination and calibration need a floor of about 100 outcome events, and 200 or more events are preferable. Below that, apparent model failures or successes may be noise. The authors offer the numbers as design guidance for anyone planning a validation study.
Catalog record
| CREATOR | Collins GS, Ogundimu EO, Altman DG. |
|---|---|
| YEAR | 2016 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sample size considerations for the external validation of a multivariable prognostic model: a resampling study By Gary S. Collins, Emmanuel O. Ogundimu, Douglas G. Altman Published in: Statistics in Medicine DOI: 10.1002/sim.6787 ABSTRACT After developing a prognostic model, it is essential to evaluate the performance of the model in samples independent from those used to develop the model, which is often referred to as external validation. However, despite its importance, very little is known about the sample size requirements for conducting an external validation. Using a large real data set and resampling methods, we investigate the impact of sample size on the performance of six published prognostic models. Focussing on unbiased and precise estimation of performance measures (e.g. the c‐index, D statistic and calibration), we provide guidance on sample size for investigators designing an external validation study. Our study suggests that externally validating a prognostic model requires a minimum of 100 events and ideally 200 (or more) events. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd. INTRODUCTION Prognostic models are developed to estimate an individual's probability of developing a disease or outcome in the future. A vital step toward accepting a model is to evaluate its performance on similar individuals separate from those used in its development, which is often referred to as external validation or transportability 1, 2. However, despite the widespread development of prognostic models in many areas of medicine 3, 4, 5, very few been externally validated 6, 7, 8. To externally validate a model is to evaluate its predictive performance (calibration and discrimination) using a separate data set from that used to develop the model 9. It is not repeating the entire modelling process on new data, refitting the model to new ‘validation’ data, or fitting the linear predictor (prognostic index) from the original model as a single predictor to new data 9. It is also not necessarily comparing the similarity in performance to that obtained during the development of the prognostic model. Whilst in some instances a difference in the performance can be suggestive of deficiencies in the development study, the performance in the new data may still be sufficiently good enough for the model to be potentially useful. The case‐mix (i.e., the distribution of predictors included in the model) will influence the performance of the model 10. It is generally unlikely that the external validation data set will have an identical case‐mix to the data used for development. Indeed, it is preferable to use a slightly different case‐mix in external validation to judge model transportability. Successful external validation studies in diverse settings (with different case‐mix) indicate that it is more likely that the model will be generalizable to plausibly related, but untested settings 11. Despite the clear importance of external validation, the design requirements for studies that attempt to evaluate the performance of multivariable prognostic models in new data have been little explored 7, 12, 13. Published studies evaluating prognostic models are often conducted using sample sizes that are clearly inadequate for this purpose, leading to exaggerated and misleading performance of the prognostic model 7. Finding such examples is not difficult 14, 15, 16. For example, a modified Thoracoscore, to predict in‐hospital mortality after general thoracic surgery, was evaluated using 155 patients, but included only eight events (deaths). A high c‐index value was reported, 0.95 (95% confidence interval 0.91 to 0.99) 16. In the most extreme case, a data set with only one outcome event was used to evaluate a prognostic model 14. In this particular study, an absurd value of the c‐index was reported, 1.00 (95% confidence interval 1.00 to 1.00)[sic]. Concluding predictive accuracy, and thus that the model is fit for purpose, on such limited data is nothing but misleading. The only guidance for sample size considerations that we are aware of is based on a hypothesis testing framework (i.e. to detect pre‐specified changes in the c‐statistic) and recommends that models developed using logistic regression are evaluated with a minimum of 100 events 12. However, a recent systematic review evaluating the methodological conduct of external validation studies found that just under half of the studies evaluated models on fewer than 100 events 7. It is therefore important to provide researchers with appropriate guidance on sample size considerations when evaluating the performance of prognostic models in an external validation study. When validating a prognostic model, investigators should clearly explain how they determined their study size, so that their findings can be placed in context 17, 18. Our view is that external validation primarily concerns the accurate (unbiased) estimation of performance measures (e.g., the c‐index). It does not necessarily include formal statistical hypothesis testing, although this may be useful in some situations. Therefore sample size considerations should be based on estimating performance measures that are sufficiently close to the true underlying population values (i.e., unbiased) along with measures of uncertainty that are sufficiently narrow (i.e., precise estimates) so that meaningful conclusions on the model's predictive accuracy in the target population can be drawn 9, 19. The aim of this article is to examine sample size considerations for studies that attempt to externally validate prognostic models and to illustrate that many events are required to provide reasonable estimates of model performance. Our study uses published prognostic models (QRISK2 20, QDScore 21 and the Cox Framingham risk score 22) to illustrate sample size considerations using a resampling design from a large data set (>2 million) of general practice patients in the UK. The structure of the paper is as follows. Section 2 describes the clinical data set and the prognostic models. Section 3 describes the design of the study, the assessment of predictive performance and the methods used to evaluate the resampling results. Section 4 presents the results from the resampling study, which are then discussed in Section 5. DATA SET AND PROGNOSTIC MODELS Study data: the health improvement network The Health Improvement Network (THIN) is a large database of anonymized primary care records collected at general practice surgeries around the UK. The THIN database currently contains medical records on approximately 4% of the UK population. Clinical information from over 2 million individuals (from 364 general practices) registered between June 1994 and June 2008 form the data set. The data have previously been used in the external validation of a number of prognostic models (including those considered in this study) 23, 24, 25, 26, 27, 28, 29, 30. There are missing data for various predictors needed to use the prognostic models. For simplicity, we have used one of the imputed data sets from the published external validation studies, where details on the imputation strategy can be found 23, 24. Prognostic models At the core of the study are six sex‐specific published models for predicting the 10‐year risk of developing cardiovascular disease (CVD) (QRISK2 20, and Cox Framingham 22) and the 10‐year risk of developing type 2 diabetes (QDScore 21). All six prognostic models are all predicting time‐to‐event outcomes using Cox regression. None of these models were developed using THIN, but THIN has previously been used to evaluate their performance in validation studies 23, 24. QRISK2 was developed using 1.5 million general practice patients aged between 35 and 74 years (10.9