Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.

Moons KG, de Groot JA, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, Reitsma JB, Collins GS. · 2014

Read Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. by Moons KG, de Groot JA, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, Reitsma JB, Collins GS. (2014) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc4196729.txt.

Introduction

Karel G. M. Moons, Gary S. Collins, and six co-authors wrote a checklist for reviewing prediction model studies in PLoS Medicine in 2014. Doctors use risk models to forecast who will get a disease or how a patient will fare. Studies of such models multiply fast, and reviewers lacked a shared way to judge them.

Their checklist, called CHARMS, starts with seven items for framing a sharp review question: who the model is for, what it predicts, and when. It then lists eleven domains for pulling data from each study and probing its weak points, from how participants were chosen to how missing data were handled. The items were drawn from existing reporting guides and methods work. The authors intend the list to raise the quality of evidence summaries behind clinical risk tools.

Catalog record

CREATORMoons KG, de Groot JA, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, Reitsma JB, Collins GS.
YEAR2014
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC4196729

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist By Karel G. M. Moons, Joris A. H. de Groot, Walter Bouwmeester, Yvonne Vergouwe, Susan Mallett, Douglas G. Altman, Johannes B. Reitsma, Gary S. Collins Published in: PLoS Medicine DOI: 10.1371/journal.pmed.1001744 ABSTRACT Carl Moons and colleagues provide a checklist and background explanation for critically appraising and extracting data from systematic reviews of prognostic and diagnostic prediction modelling studies. Please see later in the article for the Editors' Summary Summary PointsPublications on clinical prediction models have become abundant for both prognostic and diagnostic purposes. Systematic reviews of these studies are increasingly required to identify and critically appraise existing evidence.No specific guidance exists to help frame a well-defined review question and determine which details to extract and critically appraise from primary prediction modelling studies.Existing reporting guidelines, quality assessment tools, and key methodological publications were examined to identify seven items important for framing the review question and 11 domains to extract and critically appraise the primary included studies.Together these items and domains form the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS). INTRODUCTION Prediction models, both diagnostic and prognostic, are becoming increasingly abundant in the medical literature [1]–[3]. Diagnostic models are aimed at calculating the probability that an individual has a certain disorder, such as deep vein thrombosis [4],[5], ankle fractures [6], or conjunctivitis [7]. Prognostic prediction models concern the prediction of the probability or risk of the future occurrence of a particular outcome or event in individuals at risk of such an event. Prognostic models may involve models for individuals with a particular health condition, such as prediction of recurrence or death after diagnosis of breast cancer [8] or mortality after cardiac surgery [9], but also includes models for predicting the occurrence of future outcomes in apparently healthy individuals such as the risk of developing a coronary event [10] or type 2 diabetes mellitus [11]. There are over 100 models for predicting outcome after brain trauma [12], over 60 models for breast cancer prognosis [13], 45 models for cardiovascular events after being diagnosed with diabetes [14], 43 models for predicting prevalent and incident type 2 diabetes [15], and 20 models for predicting prolonged intensive care stay after cardiac surgery [16]. Furthermore, prediction models are increasingly being appraised and recommended for formal risk assessment in clinical guidelines [17],[18]. To evaluate the proliferation of prediction models, systematic reviews are necessary and led to the formation of the Cochrane Collaboration Prognosis Reviews Methods Group [19],[20]. Since then, search strategies for identifying prognostic and diagnostic prediction model studies have been developed [21]–[23], validated, and further refined [24]. However, no published checklists support the design of systematic reviews of prediction modeling studies, or what to extract and how to appraise primary prediction modelling studies. Existing guidance for synthesizing studies of prognostic factors [25],[26] does not address studies of multivariable prediction models. Instead, reviews of prediction model studies have created their own checklist [2],[12],[14],[15],[27]–[30], with variable inclusion of key details. Our aim was to design a CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS). The checklist is designed to help form a review question for and appraisal of all types of primary prediction modelling studies, including, regressions, neural network, genetic programming, and vector machine learning models [1]–[3],[12],[14],[15],[27]–[30]. Some items, such as “selection of predictors during multivariable modelling” and “model presentation”, are somewhat more specific to regression approaches. The checklist is not intended for systematic reviews of primary studies of prognostic factors, for which we refer to the QUIPS tool [25],[26], nor is it intended for prediction model impact studies in which, in principle, a comparative (intervention) design is used [1],[31],[32]. Box 1 shows the types of prediction modelling studies for which the CHARMS checklist was developed. Box 1. Types of Prediction Modelling Studies Prediction model development studies without external validation aim to develop a prognostic or diagnostic prediction model from the dataset at hand: the development set. Such studies commonly aim to identify important predictors for the outcome under study, assign mutually adjusted weights per predictor in a multivariable analysis, develop a final prediction model, and quantify the predictive performance (e.g., discrimination, calibration, classification) of that model in the development set. As model overfitting may occur, particularly in small datasets, development studies ideally include internal validation using some form of data re-sampling techniques, such as bootstrapping, jack-knife, or cross-validation, to quantify any optimism in the predictive performance of the developed model. Prediction model development studies with external validation in independent data have the same aim as the previous type, but the development of the model is followed by quantifying the model's predictive performance in participant data external to the development dataset. This may be done in participant data collected by the same investigators, commonly using the same predictor and outcome definitions and measurements, but from a later time period (temporal or narrow validation), or by other investigators in another hospital or country (geographical or broad validation). External model validation studies with or without model updating aim to assess and compare the predictive performance of an existing prediction model using new participant data that were not used to develop the prediction model and possibly adjust or update the model in case of poor performance based on the validation data.Prediction studies exploring which predictors independently contribute to the prediction of a particular prognostic or diagnostic outcome as well as studies aimed at quantifying the impact of using a prediction model (on, e.g., clinical decision making, patient outcomes, or cost-effectiveness of care) relative to not using the model may also be considered in a systematic review of prognostic and diagnostic prediction models [2]. However, data extraction and critical appraisal of those types of prediction studies is very different as they have different aims, designs, and reporting issues compared to studies developing or validating prediction models. Therefore, here we explicitly focus on reviews of studies aimed at developing, validating, or updating a prediction model. DEVELOPMENT OF THE CHECKLIST We developed our checklist based on published risk of bias tools, existing critical appraisal checklists for systematic reviews of randomised therapeutic trials and diagnostic test accuracy research, methodological recommendations for conduct and reporting of prediction model research, and data extraction sheets used in published reviews of prediction modelling studies after contacting authors. First, we reviewed the existing reporting guidelines for other types of clinical research including CONSORT, REMARK, STARD, STROBE, GRIPS [33]–[37] and for the reporting of systematic reviews (PRISMA) [38]. Furthermore, we considered existing quality assessment tools including the Cochrane Risk of Bias tool [39] for randomised therapeutic studies, QUADAS (and QUADAS-2) for diagnostic accuracy studies [40],[41], and the QUIPS checklist for appraisal of prognostic factor studies [25],[26]. We then reviewed published systematic reviews of prediction models and prognostic factor studies, along with the checklists or quality appraisal criteria used in those reviews [12],[27]–[29],[42]–[46]. Finally, we identified key methodological literature discussing recommended approaches for the design, conduct, analysis, and reporting of prediction models, followed by a search of the corresponding reference lists [3],[19],[31],[32],[37],[47]–[59]. Initial pilot versions of this checklist were presented and discussed at the annual Cochrane Prognosis Methods Group meetings and workshops from

Read the complete text