Quantifying, displaying and accounting for heterogeneity in the meta-analysis of RCTs using standard and generalised Q statistics.

Bowden J, Tierney JF, Copas AJ, Burdett S. · 2011

Read Quantifying, displaying and accounting for heterogeneity in the meta-analysis of RCTs using standard and generalised Q statistics. by Bowden J, Tierney JF, Copas AJ, Burdett S. (2011) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc3102034.txt.

Introduction

Bowden, Tierney, Copas, and Burdett examined in 2011 how to measure disagreement between trials when their results are pooled, using eighteen cancer meta-analyses built from individual patient data. Pooling assumes trials estimate one effect, yet real trials differ, and that heterogeneity decides whether a pooled average means anything. The classic statistics for it can disagree with each other.

The two most discordant cases showed why blindly applying the standard random effects fix can mislead. A generalized version of the classic statistic estimated the amount of heterogeneity more accurately across the eighteen analyses. Exploring preplanned subgroups, diagnostic plots, and sensitivity checks explained more than any automatic model switch. The authors recommend building the generalized statistic into standard software, and they supply code to do it. Their larger point is that heterogeneity should be investigated, not just accommodated.

Catalog record

CREATORBowden J, Tierney JF, Copas AJ, Burdett S.
YEAR2011
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC3102034

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantifying, displaying and accounting for heterogeneity in the meta-analysis of RCTs using standard and generalised Q statistics By Jack Bowden, Jayne F Tierney, Andrew J Copas, Sarah Burdett Published in: BMC Medical Research Methodology DOI: 10.1186/1471-2288-11-41 ABSTRACT Abstract Background Clinical researchers have often preferred to use a fixed effects model for the primary interpretation of a meta-analysis. Heterogeneity is usually assessed via the well known Q and I2 statistics, along with the random effects estimate they imply. In recent years, alternative methods for quantifying heterogeneity have been proposed, that are based on a 'generalised' Q statistic. Methods We review 18 IPD meta-analyses of RCTs into treatments for cancer, in order to quantify the amount of heterogeneity present and also to discuss practical methods for explaining heterogeneity. Results Differing results were obtained when the standard Q and I2 statistics were used to test for the presence of heterogeneity. The two meta-analyses with the largest amount of heterogeneity were investigated further, and on inspection the straightforward application of a random effects model was not deemed appropriate. Compared to the standard Q statistic, the generalised Q statistic provided a more accurate platform for estimating the amount of heterogeneity in the 18 meta-analyses. Conclusions Explaining heterogeneity via the pre-specification of trial subgroups, graphical diagnostic tools and sensitivity analyses produced a more desirable outcome than an automatic application of the random effects model. Generalised Q statistic methods for quantifying and adjusting for heterogeneity should be incorporated as standard into statistical software. Software is provided to help achieve this aim. BACKGROUND Meta-analysis provides a way of quantitatively synthesising the results of medical studies or trials that target a particular research question. As shown in a 2005 review of the clinical research literature [1], it is still most common to meta-analyse results across clinical studies using the inverse variance approach, to yield a 'fixed' or 'common' effect estimate. By obtaining individual patient data (IPD) from all trials in a meta-analysis, some aspects of clinical heterogeneity can be minimised through data cleaning [2]. However, regardless of whether the meta-analysis is based on IPD or aggregate data, substantial statistical heterogeneity between studies may still remain. Cochran's Q statistic has long been used to assess statistical heterogeneity in meta-analysis. When Q is larger than its expected value E[Q] under the null hypothesis of no heterogeneity, the difference Q - E[Q] can be used to furnish the most popular estimate of the heterogeneity parameter, using the DerSimonian and Laird method [3]. Higgins and Thompson's I2 statistic [4,5] is also a simple function of Q and quantifies the proportion of total variation that is between trial heterogeneity. Unlike Q, I2 is designed to be independent of the number of trials constituting the meta-analysis and independent of the outcome's scale, so it can easily be compared across meta-analyses. It is now reported as standard, with or without Cochran's Q. The presence of significant and substantial heterogeneity demands some form of action. Ideally, after exploration of the data, heterogeneity can be explained by variation in the constituent trial's characteristics. If this is not possible then some may feel a meta-analysis inappropriate altogether, whereas some would opt for fitting a random effects model to the data instead. There is no accepted rule for deciding on when a move from a fixed to a random effects model is the right course of action [6]. Clearly, all other things being equal, the larger the magnitude of the heterogeneity the stronger the case for a shift. However, as the amount of heterogeneity increases, so too does the potential impact of moving from one model to the other. Thus, with increasingly diverging interpretations, it is sometimes very difficult to make a satisfactory decision on which model to choose, or indeed whether to pool the trials in a meta-analysis at all. In Methods we review the standard approach to meta-analysis and heterogeneity quantification based on the Q statistic. We then introduce a similar approach based on a 'generalised Q' statistic that has recently been proposed. In Results we analyse the summary data from 18 separate IPD meta-analyses to see whether the original conclusions could have been sensitive to the choice of fixed or random effects model. A more in-depth analysis is then conducted on the two meta-analyses with the largest observed heterogeneity. The 18 meta-analysis are then used to illustrate the relative performance of the standard and generalised Q statistics in measuring the extent of heterogeneity present. Finally, in Discussion and Conclusions we review the issues raised and offer recommendations for the future quantification and reporting of heterogeneity in meta-analysis. The data The MRC Clinical Trials Unit has carried out systematic reviews and IPD meta-analyses, predominantly in cancer since 1991. Their common primary aim has been to assess whether treatment interventions have improved patient survival. Specific areas of focus include cancers of the brain, lung, cervix, ovaries and bladder. Table 1 shows the summary statistics of 18 such IPD meta-analyses [7-17]. The usefulness of these meta-analyses is that they all pre-specified subgroup analyses by trial and patient characteristics in order to explain potential heterogeneity. For illustration these analyses are done ignoring any pre-specified groupings and are only with respect to the primary outcome of overall survival. A two-stage approach was taken for each meta-analysis treatment comparison. That is, fixed effect hazard ratio estimates were calculated for each trial using the log rank method, [18], these estimates were then combined using fixed and random effects models in the same manner as for aggregate data. The meta-analyses differed in terms of their size (from 5-19 studies), their fixed effect hazard ratio effect estimate (0.65-1.20) and their heterogeneity (Q statistic p-values from 1.97 × 10-5 to 0.99 and I2 from 0 to 75%). [Table: The summary statistics for 18 meta-analyses carried out by the MAG.] METHODS Consider a meta-analysis of M studies. When study i out of M's effect estimate - denoted by - is assumed to be normally distributed with a known variance , then one can think of the study estimates as centered around a common mean parameter θ as in formula (1): [formula: (1)] The ϵi term relates to the precision of study i's estimate, and is assumed to follow a N (0, ) distribution. The ui term is assumed to have zero mean and a variance of τ2; it is included to represent potential between trial heterogeneity. When τ2 equals 0 all studies provide an estimate of the same mean parameter θ . Under the assumption that τ2 is 0 the fixed effect (FE) estimate, associated variance and assumed asymptotic distribution can be obtained: [formula: (2)] where Wi = 1/ is study i's precision. Heterogeneity quantification using the standard Q-statistic If the fixed effects assumption is true then Cochran's statistic: [formula: (3)] should follow, asymptotically, a χ2 distribution, with expected value equal to M - 1. However, if τ2 is non zero so that there is a degree of heterogeneity among the trials, study i provides an estimate of θ + ui and the expected value of Q equals [formula: (4)] where , and is referred to as the 'typical' within study variance. The most commonly applied estimate of τ2 is due to DerSimonian and Laird [3]. This simply

Read the complete text