Which is more generalizable, powerful and interpretable in meta-analyses, mean difference or standardized mean difference?

Takeshima N, Sozu T, Tajika A, Ogawa Y, Hayasaka Y, Furukawa TA. · 2014

Read Which is more generalizable, powerful and interpretable in meta-analyses, mean difference or standardized mean difference? by Takeshima N, Sozu T, Tajika A, Ogawa Y, Hayasaka Y, Furukawa TA. (2014) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc3936842.txt.

Introduction

Nozomi Takeshima and five colleagues settle a methods question for meta-analysis. Their paper appeared in BMC Medical Research Methodology in 2014. When trials share the same unit, which pooled measure is better? The raw mean difference is one choice. The standardized mean difference is the other. The choice shapes how well results travel and how clear they read.

The team mined the Cochrane Database for answers. They found 1,068 meta-analyses with three or more trials each. All used the same continuous outcome within each set. For each trial, they set its result beside the pooled result of the rest. They did this with both measures. The standardized measure showed less strife between studies. It also agreed more with the pooled values. This held under both pooling models. The authors say the standardized measure travels better, even when raw units match.

Catalog record

CREATORTakeshima N, Sozu T, Tajika A, Ogawa Y, Hayasaka Y, Furukawa TA.
YEAR2014
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC3936842

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Which is more generalizable, powerful and interpretable in meta-analyses, mean difference or standardized mean difference? By Nozomi Takeshima, Takashi Sozu, Aran Tajika, Yusuke Ogawa, Yu Hayasaka, Toshiaki A Furukawa Published in: BMC Medical Research Methodology DOI: 10.1186/1471-2288-14-30 ABSTRACT Abstract Background To examine empirically whether the mean difference (MD) or the standardised mean difference (SMD) is more generalizable and statistically powerful in meta-analyses of continuous outcomes when the same unit is used. Methods From all the Cochrane Database (March 2013), we identified systematic reviews that combined 3 or more randomised controlled trials (RCT) using the same continuous outcome. Generalizability was assessed using the I-squared (I2) and the percentage agreement. The percentage agreement was calculated by comparing the MD or SMD of each RCT with the corresponding MD or SMD from the meta-analysis of all the other RCTs. The statistical power was estimated using Z-scores. Meta-analyses were conducted using both random-effects and fixed-effect models. Results 1068 meta-analyses were included. The I2 index was significantly smaller for the SMD than for the MD (P < 0.0001, sign test). For continuous outcomes, the current Cochrane reviews pooled some extremely heterogeneous results. When all these or less heterogeneous subsets of the reviews were examined, the SMD always showed a greater percentage agreement than the MD. When the I2 index was less than 30%, the percentage agreement was 55.3% for MD and 59.8% for SMD in the random-effects model and 53.0% and 59.8%, respectively, in the fixed effect model (both P < 0.0001, sign test). Although the Z-scores were larger for MD than for SMD, there were no differences in the percentage of statistical significance between MD and SMD in either model. Conclusions The SMD was more generalizable than the MD. The MD had a greater statistical power than the SMD but did not result in material differences. BACKGROUND A meta-analysis aggregates indexes of effectiveness of individual trials into one pooled estimate. When the outcome of interest is a dichotomous variable, the commonly used effect sizes include the odds ratio (OR), the relative risk (RR), and the risk difference (RD). When the outcome is a continuous variable, then the effect size is commonly represented as either the mean difference (MD) or the standardised mean difference (SMD) [1]. The MD is the difference in the means of the treatment group and the control group, while the SMD is the MD divided by the standard deviation (SD), derived from either or both of the groups. Depending on how this SD is calculated, the SMD has several versions such, as Cohen's d [2], Glass's Δ [3], and Hedges' g [4]. When the outcome is measured in different units across trials, then we have no other choice but to use the SMD to combine the outcomes in the meta-analyses. On the other hand, when the outcome is measured in the same unit in every trial, theoretically, we can use either the MD or the SMD. In this latter case, there currently appears to be no unanimous agreement about which effect size is preferable, and different textbooks of meta-analyses provide differently nuanced recommendations about the selection of the appropriate effect size for continuous variables. According to the Cochrane Handbook for Systematic Reviews of Interventions [1], the “selection of summary statistics for continuous data is principally determined by whether studies all report the outcome using the same scale when the MD can be used.” The American Psychological Association (APA) Task Force on Statistical Inference maintains that “if the units of measurement are meaningful on a practical level (e.g. number of cigarettes smoked per day), then we usually prefer a MD to a SMD” [5]. Egger et al. writes that “the overall treatment effect [in terms of SMD] can also be difficult to interpret as it is reported in units of standard deviation rather than in units of any of the measurement scales used in review” [6]. On the other hand, there are also authors who recommend the SMD along with or over the MD. The APA Publication Manual suggests that it can often be valuable to report not only the MD but also the SMD [7]. Borenstein, in his “Introduction to Meta-Analysis” [8], wrote that if the unit is unfamiliar, the SMD serves as an easy way to judge the magnitude of the effect, thanks to the general rules of thumb described by Cohen that suggest that an SMD of 0.2 represents a “small” effect, an SMD of 0.5 represents a “medium” effect, and an SMD of 0.8 represents a “large” effect [2]. For example, when you read that a treatment group’s mean post-treatment score on scale X was 10 points higher than that of a control group, there is no way of appreciating how much a difference this actually represents unless you are very familiar with the scale that is being used. But if the difference is expressed in terms of SMD as corresponding to an effect size of 0.5, for example, you can understand that it represents a moderate effectiveness in comparison with the control. In fact, Tian et al. noted that the SMD does not depend on the unit of measurement, and therefore the SMD has been widely used as a measure of intervention effect in many applied fields [9]. The preferability of the MD or the SMD can be examined from three aspects. First, which of the MD or the SMD is clinically more interpretable? The above-summarised arguments made by different authors seem to concern mainly this aspect. Second, which of the MD or the SMD is more generalizable (as any summary index should have a good generalizability so that it can be applied to the next group of patients) [10]? Third, which of the MD or the SMD is statistically more powerful (as we expect meta-analyses to provide as precise an estimate of the treatment effect as possible and to be as sensitive as possible to differences among treatments)? To the best of our knowledge, no systematic assessment focusing on the second or third aspects of the MD and the SMD has been conducted. The objective of this research was, therefore, to examine empirically which index is more generalizable and statistically powerful in meta-analyses when the same unit is used: MD or SMD? METHODS Selection of meta-analyses We included the following intervention meta-analyses from the Cochrane Database of Systematic Reviews (March 31, 2013) [11,12]. 1. The outcome of the meta-analyses was a continuous outcome in the same unit. 2. There were at least three studies contributing to that continuous outcome in question, because we need at least three studies in order to calculate the percentage agreement that we defined as one of our outcome (see below). 3. If there were two or more outcomes that met the above criteria, we selected the outcome that contained the greatest number of studies in the review. If there were 2 or more such outcomes, we chose the outcome reported for the greatest number of patients. If the numbers of the patients were the same, we selected the outcome that appeared first in the review. We excluded the outcomes reported in sensitivity analyses or subgroup analyses. Search methods and data extraction We searched the Cochrane Database of Systematic Reviews (March 31,

Read the complete text