Publication bias in reports of animal stroke studies leads to major overstatement of efficacy.

Sena ES, van der Worp HB, Bath PM, Howells DW, Macleod MR. · 2010

Read Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. by Sena ES, van der Worp HB, Bath PM, Howells DW, Macleod MR. (2010) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc2846857.txt.

Introduction

Sena, van der Worp, Bath, Howells, and Macleod pooled animal stroke research in 2010 to measure how much unpublished work distorts the picture. Journals favor positive results, so negative experiments often sit in file drawers while summaries built on published work overstate benefit. Using systematic review data gathered on interventions tested in animal models of acute stroke, they measured the gap.

Adjusting for the missing studies cut the apparent benefit sharply: publication bias alone accounted for roughly a thirty percent overstatement of efficacy across the stroke treatments reviewed. The authors warn that candidate drugs chosen from such inflated summaries enter human trials looking stronger than they are, and they press for fuller reporting of animal experiments.

Catalog record

CREATORSena ES, van der Worp HB, Bath PM, Howells DW, Macleod MR.
YEAR2010
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC2846857

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Publication Bias in Reports of Animal Stroke Studies Leads to Major Overstatement of Efficacy By Emily S. Sena, H. Bart van der Worp, Philip M. W. Bath, David W. Howells, Malcolm R. Macleod, Ian Roberts Published in: PLoS Biology DOI: 10.1371/journal.pbio.1000344 ABSTRACT Publication bias confounds attempts to use systematic reviews to assess the efficacy of various interventions tested in experiments modelling acute ischaemic stroke, leading to a 30% overstatement of efficacy of interventions tested in animals. INTRODUCTION Few publications describing natural phenomena are in themselves sufficient to change our understanding of the world, and knowledge advances through the summarising of data in conference presentations, review articles, and books. Traditionally this process has been rather haphazard, with sometimes partisan experts using narrative review articles to emphasise their own particular perspective. Attempts have been made to account for this bias using the technique of systematic review, in which there is prespecification of the biological question being addressed, the methods through which contributing data will be identified, and the criteria that will be used to select which data are included in the analysis [1]. While systematic reviewers often go to some lengths to identify unpublished data sources, both approaches are potentially confounded by the ability to include only available data. If experiments have been conducted but are not available to reviewers, and if the results of these experiments as a group are not the same as results from experiments that were published, then both narrative and systematic reviews, and the resulting expert opinion and public understanding, will be biased. This is the “file drawer problem” [2],[3]: at its most extreme, the 95% of studies that were truly neutral (that is, which reported no significant effects) remain in the files of the investigators, the 5% of experiments that were falsely positive are published, and reviewers conclude—falsely—that the literature represents biological truth. The consequences of the drawing of erroneous conclusions would be troubling if it involved, for instance, the interpretation of data from clinical trials; indeed, the recognition of a substantial publication bias in this literature has led to the introduction of clinical trial registration systems to ensure that those summarising research findings are at least aware of all relevant clinical trials that have been performed [4]. Publication bias has also been observed in reports of genetic association studies [5] and in ecology and evolution, in which 40% of meta-analyses were confounded by publication bias, and adjusting for publication bias might have altered the conclusions in around one-third of cases [6]. A related group of biases, the citation biases [7], can be addressed through rigorous systematic review, in that an attempt is made to include all relevant publications describing data meeting predefined inclusion or exclusion criteria. However, until recently there has been a paucity of systematic reviews of animal studies [8]. The extent and any impact of publication bias in the experimental sciences are not clear. Timmer and colleagues investigated the process of publication for abstracts submitted to a leading gastrointestinal conference, and suggested both that the most responsibility for nonpublication rested with the authors (76% of unpublished projects were never submitted as a manuscript) and that for basic science studies there was no relationship between the rate of publication and whether the study reported positive, neutral, or negative findings [9]. It has previously not been possible to ascertain the impact of publication bias in animal studies because of the paucity of systematic reviews and meta-analyses, the substantial heterogeneity in the research questions asked in experimental science and in the outcomes reported, and the qualitative rather than quantitative nature of many of those outcomes. Since 2004 the Collaborative Approach to Meta-Analysis and Review of Animal Data in Experimental Studies (CAMARADES) has curated data collected in the context of systematic reviews of reports of studies describing the efficacy in animals of candidate interventions for stroke [10]–[21]. Here we use that dataset, which includes quantitative data for reported outcomes from individual experiments, to estimate the prevalence and impact of publication bias in laboratory science. RESULTS Systematic Identification of Previous Reports of Publication Bias in Animal Models of Human Disease A search for publications that might have addressed this problem previously identified 71 publications (Text S1). Of these, 11 described meta-analyses of studies reporting the animal modelling of human disease (including three CAMARADES reviews); six reported testing for the presence of publication bias (using funnel plot asymmetry or Egger regression), and publication bias was reported in four (Figure 1). No study gave quantitative estimates of the impact on effect size of publication bias. [Figure: QUOROM chart of fate of 71 publications identified in systematic search for studies reporting the quantitative impact of publication bias in reports of animal experiments modelling human disease.] Source Data As of August 2008 the CAMARADES database contained details from systematic reviews of 16 interventions tested in animal models of stroke. This database comprises data from 525 unique data sources, comprising 514 unique publications (15 contributed data to more than one review) and 11 unpublished communications, describing 1,359 experiments involving 19,956 animals (Table 1). Of these, only ten publications (2%; one publication [22] was represented in reviews of both tirilazad and tPA) described no significant effect on infarct volume, although four reported other statistically significant findings [23]–[26]. [Table: Meta-analyses included in this analysis.] Prevalence of Publication Bias For individual interventions, visual inspection of funnel plots suggested that only hypothermia and tissue plasminogen activator (tPA) had obvious inverted funnel shapes (examples for hypothermia, tPA, stem cells, and growth factors are shown in Figure 2a–2d). These interventions represented the largest individual datasets (222 experiments involving 3,256 animals and 256 experiments involving 4,029 animals, respectively). However, Egger regression suggested significant asymmetry of all 16 datasets (examples in Figure 2e–2h). In every case the intercept of the regression line was positive, suggesting an excess of imprecise studies reporting large effect sizes over that which would be expected from the overall distribution of the data. Trim-and-fill analysis of individual datasets (examples in Figure 2i–2l) suggested asymmetry affecting ten of 16 interventions, with six interventions without significant asymmetry (minocycline, NXY-059, piracetam, stem cells, tirilazad, and thrombolytics other than tPA). Funnel plotting of the entire dataset showed asymmetry, confirmed by both Egger regression and trim-and-fill (Figure 3; Table 2). [Figure: Example funnel plots, Egger regressions, and trim-and-fill plots.Data from meta-analyses of hypothermia (a,e,i), tPA (b,f,j), stem cells (c,g,k), and growth factors (d,h,l). (a–d) Funnel plots showing precision plotted against effect size. In the absence of publication bias the points should resemble an inverted funnel. (e–h) Egger regression showing precision plotted against the standardised effect size. In the absence of publication bias the regression line should pass through the origin. (i–l) Funnel plots showing the data from (a) to (d) in black, and the additional missing studies imputed by trim-and-fill in red.] [Figure: Plots describing the complete dataset.Funnel plot (a), Egger regression (b), and trim-and-fill plots (c). See Figure 1 legend for details.] [Table: Prevalence and potential impact of publication bias.] Impact of Publication Bias Trim-and-fill imputes the number and most probable results of unpublished experiments to calculate an estimate of what the effect size would be in the absence of publication bias [27] (Figure 2i–2l) The proportion of missing experiments ranged

Read the complete text