Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers.
Vickers AJ, Cronin AM, Elkin EB, Gonen M. · 2008
Read Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. by Vickers AJ, Cronin AM, Elkin EB, Gonen M. (2008) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc2611975.txt.
Introduction
Vickers, Cronin, Elkin, and Gonen extended decision curve analysis in 2008, a method they had introduced for judging diagnostic tests and prediction models. Unlike classic decision analysis, it needs no outside data on costs or preferences. It applies directly to a study data set. This paper adds the tools routine use demands.
The extensions cover four things. They correct curves for overfitting. They add confidence intervals. They handle censored data, where follow-up ends early. And they compute curves straight from predicted probabilities. Simulations showed repeated ten-fold cross-validation corrected overfit best. For censored data the method showed little bias and excellent coverage. Competing risks shifted curves as they should. Calculating from probabilities smoothed the curve. The authors provide software to implement the extended method.
Catalog record
| CREATOR | Vickers AJ, Cronin AM, Elkin EB, Gonen M. |
|---|---|
| YEAR | 2008 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers By Andrew J Vickers, Angel M Cronin, Elena B Elkin, Mithat Gonen Published in: BMC Medical Informatics and Decision Making DOI: 10.1186/1472-6947-8-53 ABSTRACT Abstract Background Decision curve analysis is a novel method for evaluating diagnostic tests, prediction models and molecular markers. It combines the mathematical simplicity of accuracy measures, such as sensitivity and specificity, with the clinical applicability of decision analytic approaches. Most critically, decision curve analysis can be applied directly to a data set, and does not require the sort of external data on costs, benefits and preferences typically required by traditional decision analytic techniques. Methods In this paper we present several extensions to decision curve analysis including correction for overfit, confidence intervals, application to censored data (including competing risk) and calculation of decision curves directly from predicted probabilities. All of these extensions are based on straightforward methods that have previously been described in the literature for application to analogous statistical techniques. Results Simulation studies showed that repeated 10-fold crossvalidation provided the best method for correcting a decision curve for overfit. The method for applying decision curves to censored data had little bias and coverage was excellent; for competing risk, decision curves were appropriately affected by the incidence of the competing risk and the association between the competing risk and the predictor of interest. Calculation of decision curves directly from predicted probabilities led to a smoothing of the decision curve. Conclusion Decision curve analysis can be easily extended to many of the applications common to performance measures for prediction models. Software to implement decision curve analysis is provided. BACKGROUND Clinical medicine has traditionally been divided into diagnosis, treatment and prognosis. From a research perspective, diagnosis and prognosis constitute a similar challenge: the clinician has some information and wants to know how this relates to the true patient state, whether this can be known currently (diagnosis) or only at some point in the future (prognosis). This information can take the form of a test – such as ultrasound for a blocked vein in the legs – or a statistical prediction model including several different variables. An example of the latter is the "Framingham risk calculator" which predicts death from cardiovascular disease on the basis of age, gender, smoking status, blood pressure and blood lipids[1]. Recent advances in medical biology, especially genomics, have raised the possibility that molecular markers might aid diagnosis or prediction. For example, it has been postulated that analysis of genes in breast cancer tissue can help predict whether breast cancer is likely to recur after surgery, and therefore whether chemotherapy would be of benefit[2]. Decision analytic and biostatistical approaches to the evaluation of tests, models and markers Decision curve analysis is a novel method for evaluating diagnostic tests, prediction models and molecular markers[3]. It was developed to overcome the limitations of traditional biostatistical methods on the one hand, and decision-analytic alternatives on the other. The traditional biostatistical approach to evaluating tests, models and markers focuses on accuracy, evaluating calibration and discrimination using metrics such as sensitivity, specificity or area-under-the-curve (AUC). Such methods are mathematically simple, can be used irrespective of whether the predictor is binary or continuous and generally have an intuitive interpretation. However, they have little clinical relevance. For example, how high an AUC is high enough to justify clinical use of a prediction model? Or take the case where a new diagnostic test increased specificity by 10% but decreased sensitivity by 5% compared to a standard test: should the new or old test be used? Answering such questions depends on the consequences of the particular clinical decisions informed by the test, model or marker. In the case of the test that was more specific, but less sensitive, than the standard, its value depends on the harm of missing a case of disease relative to the harm of treating a patient unnecessarily. Decision-analytic methods can explicitly consider the clinical consequences of decisions. They therefore provide data about the clinical value of tests, models and markers, and can thus determine whether or not these should be used in patient care. Yet traditional decision-analytic methods have several important disadvantages that have limited their adoption in the clinical literature. First, the mathematical methods can be complex and difficult to explain to a clinical audience. Second, many predictors in medicine are continuous, such as a probability from a prognostic model or a serum level of a molecular marker, and such predictors can be difficult to incorporate into decision analysis. Third, and perhaps most critically, a comprehensive decision analysis usually requires information not found in the data set of a validation study, that is, the test outcomes, marker values or model predictions on a group of patients matched with their true outcome. In the principal example used in this paper, blood was taken immediately before a biopsy for prostate cancer and various molecular markers measured. The data set for the study consisted of the levels of the various markers and an indicator for whether the biopsy was positive or negative for cancer. A biostatistician could immediately analyze these data and provide an investigator with sensitivities, specificities and AUCs; a decision analyst would have to obtain additional data on the costs and harms of biopsy and the consequences of failing to undertake a biopsy in a patient with prostate cancer. Perhaps as a result, the number of papers that evaluate models and tests in terms of accuracy dwarfs those with a decision-analytic orientation. Decision curve analysis Decision curve analysis has been described in prior methodologic[3] and conceptual papers[4]. In brief, the method is based on the principle that the relative harms of false positives (e.g. unnecessary biopsy) and false negatives (e.g. missed cancer) can be expressed in terms of a probability threshold. For example, if a man would opt for biopsy if he was told that his risk of prostate cancer was 20% or more, but not if his risk was less than 20%, it can be shown that he considers that harms associated with a missed cancer to be four times greater than the harms associated with an unnecessary biopsy, that is, the ratio of harms is the odds at the probability threshold[5]. This threshold probability can therefore be used to determine both whether a patient is defined as test-positive or negative and to model the clinical consequences of true and false positives using a clinical net benefit function: [formula: Net benefit=TruePositivesn−Falsepositivesn(pt1−pt)] where n is the total number of patients in the study and pt is the threshold probability. The threshold probability can then be varied to create the "decision curve" for any particular model, test or marker. The model, test or marker under study should first be converted to a predicted probability of the undesirable outcome (e.g. cancer on biopsy) denoted by p^: for a binary test, these probabilities are set to 1 and 0 for positive and negative test results; for a molecular marker, marker levels should be converted to a probability using logistic regression. The method of decision curve analysis is then as follows: 1. Select a pt 2. Define a patient as positive if p^ ≥ pt 3. Calculate the