External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges.

Riley RD, Ensor J, Snell KI, Debray TP, Altman DG, Moons KG, Collins GS. · 2016

Read External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. by Riley RD, Ensor J, Snell KI, Debray TP, Altman DG, Moons KG, Collins GS. (2016) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc4916924.txt.

Introduction

Richard D. Riley and co-authors wrote in The BMJ in 2016 on checking clinical prediction models in big datasets. Prediction models estimate a person's risk and guide diagnosis and prognosis calls. A model built in one setting must be tested in fresh data, called external validation. Health records and pooled trial data now offer huge validation sets.

Big sets let testers check models across places, times, and patient mixes at once. Snags include messy coding, missing values, and case mix shifts that can fake failure or success. Reporting rules also lag the new data scale. The paper maps both the chance and the traps, so big-data validation earns trust rather than borrowing it.

Catalog record

CREATORRiley RD, Ensor J, Snell KI, Debray TP, Altman DG, Moons KG, Collins GS.
YEAR2016
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC4916924

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges By Richard D Riley, Joie Ensor, Kym I E Snell, Thomas P A Debray, Doug G Altman, Karel G M Moons, Gary S Collins Published in: The BMJ DOI: 10.1136/bmj.i3140 ABSTRACT Abstract Access to big datasets from e-health records and individual participant data (IPD) meta-analysis is signalling a new advent of external validation studies for clinical prediction models. In this article, the authors illustrate novel opportunities for external validation in big, combined datasets, while drawing attention to methodological challenges and reporting issues. Summary points Clinical prediction models are used to predict the risk of disease presence and outcome occurrence in individuals, thereby informing clinical diagnosis and prognosisIncreasingly, researchers undertaking prediction model research have access to so-called “big” datasets from meta-analyses of individual participant data (IPD), or registry databases containing electronic health records for thousands or even millions of patientsSuch big datasets heralds an exciting opportunity to improve the uptake and scope of external validation research, to check whether a model’s predictions are reliable In particular, they allow researchers to externally validate a model’s predictive performance (eg, in terms of calibration and discrimination) across all clinical settings, populations, and subgroups of interest If a model has poor predictive performance, big datasets help identify if and how updating or tailoring strategies (such as recalibration) can improve performance for particular settings, clusters or subgroups (rather than simply discarding the model)However, big datasets may also bring additional methodological challenges and reporting criteria A popular type of clinical research is the development of statistical models that predict disease presence and outcome occurrence in individuals,1 2 3 thereby informing clinical diagnosis and prognosis. Such models are referred to here as diagnostic and prognostic prediction models, but they have many other names including risk models, risk scores, and clinical prediction rules. They are typically developed by use of a multivariable regression framework, which provides an equation to estimate an individual’s risk based on values of multiple predictors (such as age and smoking, or biomarkers and genetic information). Figure 1 gives the format of equations based on logistic or Cox regression, which involve an intercept or baseline hazard term combined with multiple predictor effects (corresponding to odds or hazard ratios). Well known examples are the Framingham risk score and QRISK2,4 5 which estimate the 10 year risk of developing cardiovascular disease; the Nottingham prognostic index, which predicts the five year survival probability of a woman with newly diagnosed breast cancer6 7; and the Wells score for predicting the presence of a pulmonary embolism.8 9 [Figure: Format of typical prediction models seen in the medical literature] In 2009, The BMJ published a series of four articles to guide those undertaking prediction model research,2 10 11 12 and further recommendations were made in the 2013 PROGRESS series.3 13 14 15 These articles all emphasised three fundamental components of prediction model research: model development, external validation, and impact evaluation. Model development is the process that leads to the final prediction equation, and involves many aspects detailed elsewhere.2 16 17 18 Impact studies evaluate, ideally in a randomised trial, whether the implementation of a prediction model in clinical practice actually improves patient outcomes by informing treatment decisions according to the model’s predicted risk. However, impact studies should not be considered until the robustness and generalisability of a developed model is verified in one or more external validation studies.3 19 External validation uses new participant level data, external to those used for model development, to examine whether the model’s predictions are reliable (that is, accurate enough) in individuals from potential population(s) for clinical use.20 Unfortunately, most prediction research focuses on model development and there are relatively few external validation studies.3 21 22 23 This leads to a plethora of proposed models, with little evidence about which are reliable and under what circumstances. Confusion then ensues: promising models are often quickly forgotten,24 and—of more concern—many models may be used or advocated without appropriate examination of their performance.25 A shortage of external validation studies is often attributed to the lack of data available besides those data used for model development. Data from one study (eg, a cohort study) usually have a limited number of events. Hence all data are best retained for model development, rather than splitting the data so that a part is used for development and the remainder for validation.26 However, increasingly researchers have access to “big” data, as evident by meta-analyses using individual participant data (IPD) from multiple studies,27 28 29 30 and by analyses of registry databases containing electronic health (e-health) records for thousands or even millions of patients from multiple practices, hospitals, or countries.31 For example, QRISK2 was developed by use of e-health data from the QRESEARCH database. The database uses over 1.5 million patients (with over 95 000 new cardiovascular events) from 355 randomly selected general practices,5 with external validation carried out by independent investigators in an additional 1.6 million patients from another 365 practices.32 In the IPD meta-analysis setting, an example is the IMPACT consortium, which developed a prediction model for mortality and unfavourable outcome in traumatic brain injury. The consortium shared IPD from 11 studies (8509 patients), and performed external validation using IPD from another large study (6681 patients).33 Such big, combined datasets heralds an exciting opportunity to improve the uptake of external validation research. Here, we describe the additional opportunities, challenges, and reporting issues involved in prediction research in this situation. We begin by introducing two key performance measures (calibration and discrimination) and a review of current practice in external validation research. Then, using five empirical examples, we show how big datasets allow a model’s predictive performance to be more fully interrogated across different populations, subgroups, and settings. We conclude by signposting methodological challenges and reporting criteria, which build on the recent TRIPOD statement for the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis.34 35 PREDICTIVE PERFORMANCE OF A MODEL IN TERMS OF DISCRIMINATION AND CALIBRATION External validation of a prediction model typically involves quantifying a model’s discrimination and calibration performance in data that were not used to develop the model. To be useful, a model’s predicted risks must discriminate (separate) well between those participants who do and do not have the outcome (disease or event) of interest. Discrimination is usually measured by the C statistic,18 and for survival outcomes also the D statistic (box 1).36 Calibration examines the agreement between predicted and observed risks, and can be quantified by measures such as the calibration slope and the expected/observed (E/O) statistic (box 1). Calibration can also be visualised graphically, for example, by plotting observed versus predicted risks across tenths of predicted risk,10 using a flexible calibration plot with a smoothed non-linear curve generated using a loess smoother or splines,10 37 or displaying observed and predicted survival curves over time for different risk groups.38 Box 1: Key measures for calibration and discriminationCalibration slopeFor a perfectly calibrated model, we expect to see that, in 100 individuals with a predicted risk of r% from our model, r of the 100 truly have the disease (for diagnostic prediction) or outcome (for prognostic prediction) of interest. The calibration slope

Read the complete text