Systematic review of the empirical evidence of study publication bias and outcome reporting bias.

Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR. · 2008

Read Systematic review of the empirical evidence of study publication bias and outcome reporting bias. by Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR. (2008) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc2518111.txt.

Introduction

Dwan and a large group of collaborators reviewed the evidence in 2008 that clinical trials are published selectively. Two distortions were in view: whole studies going unpublished, and published reports quietly changing which outcomes they feature. Both can mislead the reviews that doctors rely on to judge treatments.

Sixteen cohort studies that tracked trials from approval onward met the criteria. Trials with positive, statistically significant results were more likely to reach print. Significant outcomes had 2.2 to 4.7 times the odds of being fully reported compared with non-significant ones. Between 40 and 62 percent of studies changed, added, or dropped at least one main outcome between protocol and publication. The authors call for better trial reporting to curb both biases.

Catalog record

CREATORDwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR.
YEAR2008
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC2518111

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias By Kerry Dwan, Douglas G. Altman, Juan A. Arnaiz, Jill Bloom, An-Wen Chan, Eugenia Cronin, Evelyne Decullier, Philippa J. Easterbrook, Erik Von Elm, Carrol Gamble, Davina Ghersi, John P. A. Ioannidis Published in: PLoS ONE DOI: 10.1371/journal.pone.0003081 ABSTRACT Background The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention. Methodology/Principal Findings We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. Conclusions Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials. INTRODUCTION Study publication bias arises when studies are published or not depending on their results; it has received much attention [1], [2]. Empirical research consistently suggests that published work is more likely to be positive or statistically significant (P<0.05) than unpublished research [3]. Study publication bias will lead to overestimation of treatment effects; it has been recognised as a threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. There is additional evidence that research without statistically significant results takes longer to achieve publication than research with significant results, further biasing evidence over time [4]–[6], [29]. This “time lag bias” (or “pipeline bias”) will tend to add to the bias since results from early available evidence tend to be inflated and exaggerated [7], [8]. Within-study selective reporting bias relates to studies that have been published. It has been defined as the selection on the basis of the results of a subset of the original variables recorded for inclusion in a publication [9]. Several different types of selective reporting within a study may occur. For example, selective reporting of analyses may include intention-to–treat analyses versus per–protocol analyses, endpoint score versus change from baseline, different time points or subgroups [10]. Here we focus on the selective reporting of outcomes from those that were originally measured within a study; outcome reporting bias (ORB). Randomised controlled trials (RCTs) are planned experiments, involving the random assignment of participants to interventions, and are seen as the gold standard of study designs to evaluate the effectiveness of a treatment in medical research in humans [11]. The likely bias from selective outcome reporting is to overestimate the effect of the experimental treatment. Researchers have considered selective outcome reporting to be a major problem, and deserving of substantially more attention than it currently receives [12]. Recent work [13]–[19] has provided direct empirical evidence for the existence of outcome reporting bias. Studies have found that statistically significant results had a higher odds of being fully reported compared to non-significant results for both efficacy and harm outcomes. Studies comparing trial publications to protocols are accumulating evidence on the proportion of studies in which at least one primary outcome was changed, introduced, or omitted. Thus, the bias from missing outcome data that may affect a meta-analysis is on two levels: non-publication due to lack of submission or rejection of study reports (a study level problem) and the selective non-reporting of outcomes within published studies on the basis of the results (an outcome level problem). While much effort has been invested in trying to identify the former [2], it is equally important to understand the nature and frequency of missing data from the latter level. The aim of this study was to review and summarise the evidence from empirical cohort studies that have assessed study publication bias and/or outcome reporting bias in RCTs approved by a specific ethics committee or other inception cohorts of RCTs. METHODS Study inclusion criteria We included research that assessed an inception cohort of RCTs for study publication bias and/or outcome reporting bias. We focussed on inception cohorts with study protocols being registered before start of the study as this type of prospective design were deemed more reliable. We excluded cohorts based on prevalence archives, in which a protocol is registered after a study is launched or completed, since such cohorts can already be affected by publication and selection bias. Both cohorts containing exclusively RCTs or containing a mix of RCTs and non-RCTs were eligible. For those studies where it was not possible to identify the study type (i.e. whether any included studies were RCTs), we attempted to contact the authors to try to resolve this. In cases where it could not be resolved, studies were excluded. Those studies containing exclusively non-RCTs were excluded. The assessment of RCTs in the included studies had to involve comparison of the protocol against all publications (for outcome reporting bias) or information from trialists (for study publication bias). Search strategy The first author (KD) alone conducted the search. No masking was used during the screening of abstracts. MEDLINE (1950 to 2007), SCOPUS (1960 to 2007) and the Cochrane Methodology Register (1898 to 2007) were searched without language restrictions (final search December 2007 - see Appendix S1 for all search strategies). SCOPUS is a much larger database than EMBASE, it offers more coverage of scientific, technical, medical and social science literature than any other database. Over 90% of the sources indexed by EMBASE are also indexed by SCOPUS plus many other indexed sources as well. Additional steps were taken to complement electronic database searches: First, the references given in the empirical evidence section of the HTA report of Song et al [1] were checked for relevance. Second, the lead reviewer of the protocol on the Cochrane library entitled ‘Publication bias in clinical trials’ [20] (Sally Hopewell) was contacted in November 2007 for references to studies included and excluded in their review. Their search strategy was compared to our own and differences in included studies were discussed between PRW, KD and Sally Hopewell. Finally, the lead or contact authors of all identified studies were asked to identify further studies. Quality assessment To assess

Read the complete text