Undue reliance on I(2) in assessing heterogeneity may mislead.
Rücker G, Schwarzer G, Carpenter JR, Schumacher M. · 2008
Read Undue reliance on I(2) in assessing heterogeneity may mislead. by Rücker G, Schwarzer G, Carpenter JR, Schumacher M. (2008) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc2648991.txt.
Introduction
Rücker and colleagues warned in 2008 that a popular statistic can mislead people who combine study results. When researchers merge trials in a meta-analysis, they check how much the trials disagree, a problem called heterogeneity. Many report a measure called I-squared, read as the share of variation due to real differences between studies.
The team showed that this measure depends on study size in a hidden way. Using a real meta-analysis, they reran the numbers as if each trial had been much bigger. The actual spread of effects did not change, yet the I-squared value shot toward 100 percent. The same pattern showed in 157 published meta-analyses. Bigger studies inflate the score even when nothing else changes. The authors say choices about merging studies should rest on whether differences matter in practice, judged by a second measure, tau-squared.
Catalog record
| CREATOR | Rücker G, Schwarzer G, Carpenter JR, Schumacher M. |
|---|---|
| YEAR | 2008 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Undue reliance on I2 in assessing heterogeneity may mislead By Gerta Rücker, Guido Schwarzer, James R Carpenter, Martin Schumacher Published in: BMC Medical Research Methodology DOI: 10.1186/1471-2288-8-79 ABSTRACT Abstract Background The heterogeneity statistic I2, interpreted as the percentage of variability due to heterogeneity between studies rather than sampling error, depends on precision, that is, the size of the studies included. Methods Based on a real meta-analysis, we simulate artificially 'inflating' the sample size under the random effects model. For a given inflation factor M = 1, 2, 3,... and for each trial i, we create a M-inflated trial by drawing a treatment effect estimate from the random effects model, using si2/M as within-trial sampling variance. Results As precision increases, while estimates of the heterogeneity variance τ2 remain unchanged on average, estimates of I2 increase rapidly to nearly 100%. A similar phenomenon is apparent in a sample of 157 meta-analyses. Conclusion When deciding whether or not to pool treatment estimates in a meta-analysis, the yard-stick should be the clinical relevance of any heterogeneity present. τ2, rather than I2, is the appropriate measure for this purpose. BACKGROUND In meta-analysis, three principal sources of heterogeneity can be distinguished. These are (i) clinical baseline heterogeneity between patients from different studies, measured, e.g., in patient baseline characteristics and not necessarily reflected on the outcome measurement scale; (ii) statistical heterogeneity, quantified on the outcome measurement scale, that may or may not be clinically relevant and may or may not be statistically significant, and (iii) heterogeneity from other sources, e.g. design-related heterogeneity. In this article, we only deal with statistical heterogeneity. References [1-7] give an introduction to the large literature in this area. We do not discuss how to assess clinical baseline heterogeneity. In this paper, we show that I2 increases with the number of patients included in the studies in a meta-analysis. In the light of this, we argue that I2 is in general of limited use in assessing clinically relevant heterogeneity. The article is structured as follows. After introducing existing measures of heterogeneity in meta-analysis and discussing their properties, we illustrate the problem of interpreting the measure I2 using an example from the literature. We then present a simulation study which explores the effect of sample size inflation on I2, and finally conclude with a discussion. METHODS Let k be the number of studies in a meta-analysis. Further, let xi be the within-study treatment effect estimate (e.g., a log odds ratio), si2 the within-study variance of xi, and wi the weight of study i (i = 1,..., k). In this article, we always use inverse variance weights, that is, wi = 1/si2 if the fixed effect model is used, and wi = 1/(si2 + τ2) if the random effects model is used (see below for definition and estimation of the heterogeneity variance τ2). Several measures of statistical heterogeneity are widely used: 1. Cochran's Q statistic, which under the null hypothesis of no heterogeneity follows a χ2 distribution with k - 1 degrees of freedom [8]. Q is given by [formula: Q=∑i=1kwi(xi−∑wjxj∑wj)2;] 2. Higgins' and Thompson's I2, derived from Cochran's Q by defining [4] [formula: I2=max{0,Q−(k−1)Q};] 3. the between-study variance, τ2, as estimated in a random effects meta-analysis. There are several proposals for estimating τ2 in a meta-analysis, such as the REML estimator or the Hedges-Olkin estimator [5-7,9]. Nevertheless, most reviewers use the moment-based estimate of τ2 [10], implemented in RevMan [11] and calculated as [formula: τ^2=max{0,Q−(k−1)∑wi−∑wi2∑wi};] 4. H2, derived from Cochran's Q by defining [4] [formula: H2=Qk−1,] and 5. R2, similar to H2 and calculated from τ2 and a so-called 'typical' within-study variance σ2 (which must be estimated), and defined as: [formula: R2=τ2+σ2σ2.] As seen here, and described elsewhere [4], some measures are directly related, and others approximately related. Table 1 shows key properties of the various measures; more details are given in [4]. In summary: [Table: Properties of measures of heterogeneity.] 1. Q, which follows a χ2 distribution with k - 1 degrees of freedom under H0, is the weighted sum of squared differences between the study means and the fixed effect estimate. It always increases with the number of studies, k, in the meta-analysis. 2. In contrast to Q, the statistic I2 was introduced by Higgins and Thompson [4] as a measure independent of k, the number of studies in the meta-analysis. I2 is interpreted as the percentage of variability in the treatment estimates which is attributable to heterogeneity between studies rather than to sampling error. 3. τ2 describes the underlying between-study variability. Its square root, τ, is measured in the same units as the outcome. Its estimates do not systematically increase with either the number, or size, of studies in a meta-analysis. 4. H2 is a test statistic. It describes the relative difference between the observed Q and its expected value in the absence of heterogeneity. Thus it does not systematically increase with the number of studies [4]. H corresponds to the residual standard deviation in a radial (Galbraith) plot [12]. H = 1 indicates perfect homogeneity. 5. R2 is the square of a statistic R which describes the inflation of the random effects confidence interval compared to that from the fixed effect model. It does not increase with k. R2 = 1 indicates perfect homogeneity [4]. Notice that, in contrast to τ2, the measures Q, I2, H and R all depend on the precision, which is proportional to study size [13]. Thus, given an underlying model, if the study sizes are enlarged, the confidence intervals become smaller and the heterogeneity, measured (say) using I2, increases. This is reflected in the interpretation: As I2 is the percentage of variability that is due to between-study heterogeneity, 1 - I2 is the percentage of variability that is due to sampling error. When the studies become very large, the sampling error tends to 0 and I2 tends to 1. Such heterogeneity may not be clinically relevant. We now explore this further using simulation. Note first that simply looking at the effect of scaling up all sample sizes by a common factor (leaving their treatment effects unchanged) is not appropriate. This is because if study sizes were truly to increase, estimates would approach the true value for each study and not be fixed at the original observed value. Instead, we simulate under the random effects model. Under this model, μ and τ2 are assumed constant, and the total variance in study i is σi2 + τ2, which decreases with increasing study sample size, eventually tending to τ2. Study size inflation based on the random effects model Suppose in a meta-analysis trial i reports treatment effect estimate xi (e.g., on the log odds scale) with observed sampling variance si2. Let τ2 denote the heterogeneity variance. The model is [formula: xi=μ+σi2+τ2ϵi,ϵi~N(0,1),] where μ is the average treatment effect. For a given inflation factor M = 1, 2, 3,..., the model with inflated sample size (corresponding to an M-fold increase in precision) is [formula: xM,i=μ+σi2/M+τ2ϵ′i,ϵ′i~N(0,1).] We generate an illustrative meta-analysis for each inflation factor. For each trial in each meta-analysis, we generate a random M-inflated trial by drawing a treatment effect estimate xM,i from this model, using si2/M as the within-trial sampling variance and the DerSimonian-Laird estimate τ^2 for