Using simulation studies to evaluate statistical methods.

Morris TP, White IR, Crowther MJ. · 2019

Read Using simulation studies to evaluate statistical methods. by Morris TP, White IR, Crowther MJ. (2019) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc6492164.txt.

Introduction

Morris and co-authors write the manual simulation studies always needed. Simulation studies are the lab bench of statistics, spinning data from a known truth to measure a method's bias or error. Yet they are too often designed, analyzed, and reported badly. The tutorial's scaffold is ADEMP: state the Aims, the Data-generating mechanisms, the Estimands, the Methods, and the Performance measures before running anything.

Coding practice gets detailed treatment. So do how to estimate these measures and their error bars, plus honest tables and graphics. Worked warnings show, for one, how root mean square error comparisons flip with sample size. To ground the advice, the authors audited 100 simulation papers from one journal volume and named the recurring failures.

Catalog record

CREATORMorris TP, White IR, Crowther MJ.
YEAR2019
COLLECTIONPAPERS
FORMATTXT
FILE SIZE100 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC6492164

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Using simulation studies to evaluate statistical methods By Tim P Morris, Ian R White, Michael J Crowther Published in: Statistics in Medicine DOI: 10.1002/sim.8086 ABSTRACT Abstract Simulation studies are computer experiments that involve creating data by pseudo‐random sampling. A key strength of simulation studies is the ability to understand the behavior of statistical methods because some “truth” (usually some parameter/s of interest) is known from the process of generating the data. This allows us to consider properties of methods, such as bias. While widely used, simulation studies are often poorly designed, analyzed, and reported. This tutorial outlines the rationale for using simulation studies and offers guidance for design, execution, analysis, reporting, and presentation. In particular, this tutorial provides a structured approach for planning and reporting simulation studies, which involves defining aims, data‐generating mechanisms, estimands, methods, and performance measures (“ADEMP”); coherent terminology for simulation studies; guidance on coding simulation studies; a critical discussion of key performance measures and their estimation; guidance on structuring tabular and graphical presentation of results; and new graphical presentations. With a view to describing recent practice, we review 100 articles taken from Volume 34 of Statistics in Medicine, which included at least one simulation study and identify areas for improvement. Keywords: graphics for simulation, Monte Carlo, simulation design, simulation reporting, simulation studies INTRODUCTION Simulation studies are computer experiments that involve creating data by pseudo‐random sampling from known probability distributions. They are an invaluable tool for statistical research, particularly for the evaluation of new methods and for the comparison of alternative methods. Simulation studies are much used in the pages of Statistics in Medicine, but our experience is that some statisticians lack the necessary understanding to execute a simulation study with confidence, while others are overconfident and so fail to think carefully about design and report results poorly. Proper understanding of simulation studies would enable the former to both run and critically appraise published simulation studies themselves and the latter to conduct simulation studies with greater care and report with transparency. Simulation studies are empirical experiments, and statisticians should therefore use knowledge of experimental design and analysis in running them. As we shall see, inadequacies with design, analysis, and reporting lead to uncritical use and interpretation of simulation studies. In this context, better understanding of the rationale, design, execution, analysis, and reporting of simulation studies is necessary to improve understanding and interpretation of the findings. Simulation studies are used to obtain empirical results about the performance of statistical methods in certain scenarios, as opposed to more general analytic (algebraic) results, which may cover many scenarios. It is not always possible, or may be difficult, to obtain analytic results. Simulation studies come into their own when methods make wrong assumptions or data are messy because they can assess the resilience of methods in such situations. This is not always possible with analytic results, where results may apply only when data arise from a specific model. “Monte Carlo simulation” means statistical techniques that use pseudo‐random sampling, and has many uses that are not simulation studies. For example, it is required to implement multiple imputation and Markov Chain Monte Carlo methods. The remainder of this paper does not consider these uses, unless the properties of some such method are being evaluated by a simulation study. There are many ways to use simulation studies in medical statistics. Some examples are: - ° To check algebra (and code), or to provide reassurance that no large error has been made, where a new statistical method has been derived mathematically. - ° To assess the relevance of large‐sample theory approximations (eg, considering the sampling distribution of an estimator) in finite samples. - ° For the absolute evaluation of a new or existing statistical method. Often a new method is checked using simulation to ensure it works in the scenarios for which it was designed. - ° For comparative evaluation of two or more statistical methods. - ° For calculation of sample size or power when designing a study under certain assumptions.1 This article is focused primarily on using simulation studies for the evaluation of methods. Simulation studies for this purpose are typically motivated by frequentist theory and used to evaluate the frequentist properties of methods, even if the methods are Bayesian.2, 3 It seems that as a profession we fail to follow good practice regarding design, analysis, presentation and reporting in our simulation studies, as lamented previously by Hoaglin and Andrews,4 Hauck and Anderson,5 Ripley,6 Burton et al,7 and Koehler et al.8 For example, few reports of simulation studies acknowledge that Monte Carlo procedures will give different results when based on a different set of random numbers and hence are subject to uncertainty, yet failing to report measures of uncertainty would be unacceptable in medical research. There exist some wonderful books on simulation methods in general6, 9, 10 and several excellent articles encouraging rigor in specific aspects of simulation studies,1, 4, 5, 8, 11, 12, 13, 14, 15, 16 but until now, no unified practical guidance on simulation studies. This tutorial provides such guidance. More specifically, we: introduce a structured approach for planning and reporting simulation studies; provide coherent terminology for simulation studies; offer guidance on coding simulation studies; critically discuss key performance measures and their estimation; make suggestions for structuring tabular and graphical presentation of results; and introduce several new graphical presentations. This guidance should enable practitioners to execute a simulation study for the first time and contains much for more experience practitioners. For reference, the main steps involved, key decisions and recommendations are summarised in Table 1. [Table: Key steps and decisions in the planning, coding, analysis and reporting of simulation studies] The structure of this tutorial is as follows. We describe a review of a sample of the simulation studies reported in Statistics in Medicine Volume 34 (Section 2). In Section 3, we outline a systematic approach to planning simulation studies, using the new “ADEMP” structure (which we define there). Section 4 gives guidance on computational considerations for coding simulation studies. In Section 5, we discuss the purposes of various performance measures and their estimation, stressing the importance of estimating and reporting the Monte Carlo standard error (SE) as a measure of uncertainty due to using a finite number of simulation repetitions. Section 6 outlines how to report simulation studies, again using the ADEMP structure, and offers guidance on tabular and graphical presentation of results. Section 7 works through a simple simulation to illustrate in practice the approaches that we are advocating. Section 8 offers some concluding remarks, with a short Section 8.1 that considers some future directions. Examples are drawn from the review and from the authors' areas of interest (which relate mainly to modeling survival data, missing data, meta‐analysis, and randomised trial design). SIMULATION IN PRACTICE: A REVIEW OF STATISTICS IN MEDICINE, VOLUME 34 We undertook a review of practice based on articles published in Volume 34 of Statistics in Medicine (2015). This review recorded information relevant to the ideas in this article. In this section, we briefly outline the review but do not give results, which instead are provided at relevant points. The raw data on which results are based are provided as a Stata file in the supplementary

Read the complete text