Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events.

Pavlou M, Ambler G, Seaman S, De Iorio M, Omar RZ. · 2016

Read Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events. by Pavlou M, Ambler G, Seaman S, De Iorio M, Omar RZ. (2016) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc4982098.txt.

Introduction

Menelaos Pavlou and four co-authors review penalised regression methods for clinical risk prediction when events are few. Their review appeared in Statistics in Medicine in 2016. Prediction models often rest on logistic regression. With few outcome events per predictor, standard fitting overfits: the model looks sharp at home and fails on new patients. Shrinkage methods pull coefficients toward zero to calm that excess. The team compared frequentist and Bayesian shrinkage methods on real and simulated data.

Standard maximum likelihood fitting overfit badly in low event scenarios. Penalised methods improved calibration and accuracy, though gains in discrimination were modest. Ridge regression did well except with many noise predictors, where lasso did better. The authors advise model builders to reach for penalised methods whenever events are scarce. They also explain why shrunken probabilities keep patient ranking much the same while fixing the risk levels.

Catalog record

CREATORPavlou M, Ambler G, Seaman S, De Iorio M, Omar RZ.
YEAR2016
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC4982098

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Review and evaluation of penalised regression methods for risk prediction in low‐dimensional data with few events By Georg Heinze, Stefan Michiels, Martin Posch, Menelaos Pavlou, Gareth Ambler, Shaun Seaman, Maria De Iorio, Rumana Z Omar Published in: Statistics in Medicine DOI: 10.1002/sim.6782 ABSTRACT Risk prediction models are used to predict a clinical outcome for patients using a set of predictors. We focus on predicting low‐dimensional binary outcomes typically arising in epidemiology, health services and public health research where logistic regression is commonly used. When the number of events is small compared with the number of regression coefficients, model overfitting can be a serious problem. An overfitted model tends to demonstrate poor predictive accuracy when applied to new data. We review frequentist and Bayesian shrinkage methods that may alleviate overfitting by shrinking the regression coefficients towards zero (some methods can also provide more parsimonious models by omitting some predictors). We evaluated their predictive performance in comparison with maximum likelihood estimation using real and simulated data. The simulation study showed that maximum likelihood estimation tends to produce overfitted models with poor predictive performance in scenarios with few events, and penalised methods can offer improvement. Ridge regression performed well, except in scenarios with many noise predictors. Lasso performed better than ridge in scenarios with many noise predictors and worse in the presence of correlated predictors. Elastic net, a hybrid of the two, performed well in all scenarios. Adaptive lasso and smoothly clipped absolute deviation performed best in scenarios with many noise predictors; in other scenarios, their performance was inferior to that of ridge and lasso. Bayesian approaches performed well when the hyperparameters for the priors were chosen carefully. Their use may aid variable selection, and they can be easily extended to clustered‐data settings and to incorporate external information. © 2015 The Authors. Statistics in Medicine Published by JohnWiley & Sons Ltd. INTRODUCTION The usefulness of risk prediction models for informing patients and practitioners about the future course of a disease, guiding therapeutic strategies, aiding selection of patients for inclusion in randomised trials and in surveillance has been well established 1, 2, 3. Often, a risk prediction model is developed using a regression model that associates the outcome to patient characteristics, the predictor variables. For binary outcomes, a logistic regression model is commonly used. In model fitting, the regression model is fitted to the data at hand (training or development data set) to estimate the regression coefficients. These estimated coefficients can then be used to predict the outcome in new patients. However, a risk model that performs well on the training data set may not perform equally well when it is applied to new data. Therefore, its predictive performance needs to be assessed, that is, the model needs to be validated before it can be used to make predictions. A good risk model should be able to demonstrate good calibration, discrimination and predictive accuracy in new data. Since last decade, risk models that are commonly used in practice such as the ‘QRISK‐2’ and the ‘Framingham’ calculator for the risk of cardiovascular disease 4, 5 and the ‘HCM‐SCD calculator’ for the risk of sudden cardiac death in patients with hypertrophic cardiomyopathy 6, all use the coefficients estimates from the model fitting directly into a risk calculator to calculate risks for patients. In this paper, we focus on this form of risk calculation. There are several reasons why the apparent performance of a model might not hold in the validation data. One of these is model overfitting, which is likely to arise when the number of events in the training set is small compared with the number of estimated regression coefficients. In health research, such scenarios may arise, for example, in studies of rare events or rare diseases. An overfitted model tends to capture not only the underlying processes that generated the data but also noise in the training data set, and typically overestimates or underestimates the risk of the event for high‐risk or low‐risk patients. For example, a cardiac risk model has been developed to predict the 5‐year risk of sudden death (rare event) in patients with hypertrophic cardiomyopathy and guide the implantation of implantable cardioverter defibrillators (a small device used to regulate arrhythmias) 6. Among 3672 patients with complete data, there were only 42 sudden deaths within the first year after first evaluation. Overestimating the risk of sudden death could lead to the unnecessary implantation of implantable cardioverter defibrillators, exposing patients to unnecessary risks and also wasting resources. An example of a rare disease is penile cancer, which has an annual incidence of 1.3–2.0 per 100 000 men. In a study to predict the risk of death from penile cancer 7, only 128 patients with penile cancer were available of which 25 died, making the development of a reliable prediction model challenging. Often, after employing expert knowledge to reduce the number of potential predictors, the ratio of the number of events to the number of regression coefficients to be estimated (‘events per variable’ or EPV) is low. As a rule of thumb, it has been suggested that prediction models are likely to be reliable when the EPV is at least 10 (e.g. 8). This cut‐off has been challenged as not being based on convincing scientific reasoning 3, and in fact, it is based on empirical evidence for obtaining valid estimates of standard errors for regression coefficients, rather than reliable risk prediction 9, 10. Nevertheless, the EPV is frequently mentioned in reports of clinical prediction studies, as revealed by the systematic review of Bouwmeester et al., 11, and is useful for quantifying the amount of information in the data relative to model complexity. In order to meet the EPV recommendations, researchers employ practical approaches such as stepwise selection and univariable screening to reduce the EPV. Note that EPV should be based on the initial model, before any variable selection is carried out. These naive approaches have long been known to have significant drawbacks, and notes of caution have been raised 8, 12, 13. Additionally, it is not unusual that even after reducing the EPV by univariable screening or stepwise selection, the resulting model still has low EPV and, therefore, is susceptible to overfitting. One simple way to reduce overfitting is to fit the model using maximum likelihood estimation (MLE) and then shrink the regression coefficients, post‐estimation, by a common factor (linear shrinkage factor (LSF)). The shrinkage factor is in essence an estimate of the amount of overfitting and can be obtained using bootstrapping 8. Harrell 8 noted that using an LSF is not as good as building shrinkage into the estimation process using penalised regression. The latter allows differential shrinkage of the regression coefficients. Two popular penalised regression methods are ridge and lasso. Both shrink the regression coefficients towards zero; lasso can also perform variable selection by shrinking some coefficients to exactly zero. For risk prediction in low‐dimensional settings, little attention has been devoted to these methods. Ambler et al., 13 explored the use of ridge and lasso for low‐dimensional survival data with few events and concluded that both methods improve calibration, discrimination and predictive accuracy compared with standard Cox regression. They also used backwards elimination, LSF and univariable screening and found that these methods

Read the complete text