Power analysis for random-effects meta-analysis.

Jackson D, Turner R. · 2017

Read Power analysis for random-effects meta-analysis. by Jackson D, Turner R. (2017) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc5590730.txt.

Introduction

Jackson and Turner examine statistical power in random-effects meta-analysis, in Research Synthesis Methods. Pooling studies is meant to boost the chance of detecting an effect. Under a fixed-effect model that always holds. The random-effects model adds between-study variance, which must itself be estimated.

The authors derive ways to compute the power of such a pooled analysis. They also compute the average power of the single studies feeding it. Then they apply the methods to 1,991 meta-analyses from the Cochrane Database. In practice, five or more studies were needed for pooling to beat the parts with any consistency. With very few studies, the pooled test may add nothing. The common faith that pooling always helps takes a hit. Readers get tools to check power before pooling.

Catalog record

CREATORJackson D, Turner R.
YEAR2017
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC5590730

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Power analysis for random‐effects meta‐analysis By Dan Jackson, Rebecca Turner Published in: Research Synthesis Methods DOI: 10.1002/jrsm.1240 ABSTRACT Abstract One of the reasons for the popularity of meta‐analysis is the notion that these analyses will possess more power to detect effects than individual studies. This is inevitably the case under a fixed‐effect model. However, the inclusion of the between‐study variance in the random‐effects model, and the need to estimate this parameter, can have unfortunate implications for this power. We develop methods for assessing the power of random‐effects meta‐analyses, and the average power of the individual studies that contribute to meta‐analyses, so that these powers can be compared. In addition to deriving new analytical results and methods, we apply our methods to 1991 meta‐analyses taken from the Cochrane Database of Systematic Reviews to retrospectively calculate their powers. We find that, in practice, 5 or more studies are needed to reasonably consistently achieve powers from random‐effects meta‐analyses that are greater than the studies that contribute to them. Not only is statistical inference under the random‐effects model challenging when there are very few studies but also less worthwhile in such cases. The assumption that meta‐analysis will result in an increase in power is challenged by our findings. Keywords: cochrane, empirical evaluation, random‐effects meta‐analysis, power calculations INTRODUCTION Meta‐analysis is now a very commonly used statistical tool. There are many motivations for including meta‐analyses in systematic reviews. The Cochrane Handbook1 (their section 9.1.3) gives 4 reasons for doing so: (1) to increase power, (2) to improve precision, (3) to answer questions not posed by the individual studies, and (4) to settle controversies or generate new hypotheses. The focus here is on the first of these reasons, namely, to increase power. To quote directly from the Cochrane Handbook, “Many individual studies are too small to detect small effects, but when several are combined there is a higher chance of detecting an effect.” This statement furthers our belief that there is a commonly held implicit assumption that meta‐analysis necessarily provides a way to increase statistical power and so detect effects of interest. In this paper, we will challenge this assumption. This issue should be of interest to applied analysts regardless of their preferences concerning whether to present confidence intervals or results from hypothesis tests. The former is often encouraged, and we would align ourselves with those who emphasise estimation over testing. Due to the connection between hypothesis testing and confidence intervals however, a significant hypothesis test result is equivalent to a confidence interval that excludes the null. All our results that follow are therefore immediately applicable to those who prefer to present confidence intervals because they can frame the findings of whether or not confidence intervals contain the null, and so whether they are able to detect an effect. The power of a hypothesis test is the probability that the null hypothesis is rejected when it is false. Bayesian methods would be needed to instead calculate the probability that the null hypothesis is false, but here we focus on classical methods. Power analysis for meta‐analysis is a sufficiently important topic to warrant an entire chapter devoted to it in the introductory text by Borenstein et al2 (their chapter 29). The methods for power analysis that we develop below are of this conventional type and do not attempt to allow for multiple testing within the same review3 or sequential testing in updated reviews.4 We will compare the power of the individual studies and the power of the meta‐analysis to which they contribute, because both types of power are important considerations, which impact on each other in practice.5, 6 The planning of future studies may be based on the power or precision of meta‐analysis results.5, 7, 8 Our other main focus is on the use of random‐effects meta‐analyses. The random‐effects model9, 10, 11, 12 relaxes the strong, and usually difficult to defend, assumption made by the fixed‐effect model that the studies estimate the same true effect. The random‐effects model introduces two reasons for doubting that the resulting meta‐analyses will possess more power than the individual studies, both of which are directly related to the between‐study variance. The first reason is that, compared to the fixed‐effect model, the need to estimate an additional parameter in random‐effects meta‐analyses could result in power loss even when there is no between‐study heterogeneity; in general, power is lost when introducing further parameters to a statistical model. However, a more obvious concern is that if the between‐study heterogeneity is very large then the variation in the study estimates could be very considerable; by (albeit, correctly) allowing for this variation, the standard error of the pooled estimate could then be very large. If this occurs, then the corresponding hypothesis test will possess very little power. Others have previously discussed power calculations under the random‐effects model.13, 14 Hedges and Pigott (2001)15 also discuss how the unknown between‐study variance complicates power analysis in the random‐effects setting. Although these methods are directly related to our new methods, and will also be used below, a key distinction between our new methods and most previous work is that we develop methods for evaluating the power whilst taking into account the uncertainty in the estimated between‐study variance. We should however be clear from the outset that meta‐analysis and study specific hypothesis tests involve testing different types of hypotheses: in a meta‐analysis, we test whether or not the average effect is a particular value (for example zero), and for individual studies we test whether or not the true study specific treatment effect is a particular value. The distinction between these 2 types of hypotheses is especially clear when using the random‐effects model, where we assume that the true treatment effects differ across studies. Whilst recognising that the meta‐analysis and study specific hypothesis tests differ in this way, we will still compare the power of these two types of tests. We should also recognise that the study‐specific and fixed‐effect model hypothesis tests possess, under the assumptions that these methods make, the correct significance level. However, the conventional random‐effects model hypothesis test only retains the nominal significance level approximately. To more fairly compare the power of different hypothesis tests, we should use the same actual (and not just the nominal) significance level throughout, and this is only approximately the case here. Some of the power of the conventional random‐effects model hypothesis test is therefore an artifact of the approximate nature of the methods used. One motivation for developing methods for power analysis under the random‐effects model is so that we can retrospectively determine the powers of the random‐effects meta‐analyses from a large sample of meta‐analyses from Cochrane. We therefore calculate what is sometimes referred to as the “observed power” in this empirical investigation. Then, by comparing the power of these meta‐analyses to the average power of the studies that contribute to them, we can empirically investigate the validity of the notion that random‐effects meta‐analyses result in an increase in power. This empirical investigation of a large number of random‐effects meta‐analyses is one important contribution of this paper. The new Monte Carlo method that we develop for this purpose is another important contribution. Retrospective power calculations have some value in practice because they have the potential

Read the complete text