Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study.

Savovic J, Turner RM, Mawdsley D, Jones HE, Beynon R, Higgins JPT, Sterne JAC. · 2018

Read Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study. by Savovic J, Turner RM, Mawdsley D, Jones HE, Beynon R, Higgins JPT, Sterne JAC. (2018) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc5928453.txt.

Introduction

Savović and co-authors test whether bias ratings predict trial results in Cochrane reviews. Their study appeared in the American Journal of Epidemiology. They drew bias judgements from 228 meta-analyses. Those covered 2,443 randomised trials in total. Four bias types were checked, from randomisation to missing data. Outcomes were classed as death, objective, or subjective.

Effects were exaggerated in higher-risk trials, they report. Poor sequence generation, concealment, and blinding all inflated results. The inflation was about 8 to 13 percent on average. Subjective outcomes did not show clearly worse bias. Lack of blinding did raise spread between trials there. Reviewer inconsistency in ratings is a noted limit.

Catalog record

CREATORSavovic J, Turner RM, Mawdsley D, Jones HE, Beynon R, Higgins JPT, Sterne JAC.
YEAR2018
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC5928453

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study By Jelena Savović, Rebecca M Turner, David Mawdsley, Hayley E Jones, Rebecca Beynon, Julian P T Higgins, Jonathan A C Sterne Published in: American Journal of Epidemiology DOI: 10.1093/aje/kwx344 ABSTRACT Abstract Flaws in the design of randomized trials may bias intervention effect estimates and increase between-trial heterogeneity. Empirical evidence suggests that these problems are greatest for subjectively assessed outcomes. For the Risk of Bias in Evidence Synthesis (ROBES) Study, we extracted risk-of-bias judgements (for sequence generation, allocation concealment, blinding, and incomplete data) from a large collection of meta-analyses published in the Cochrane Library (issue 4; April 2011). We categorized outcome measures as mortality, other objective outcome, or subjective outcome, and we estimated associations of bias judgements with intervention effect estimates using Bayesian hierarchical models. Among 2,443 randomized trials in 228 meta-analyses, intervention effect estimates were, on average, exaggerated in trials with high or unclear (versus low) risk-of-bias judgements for sequence generation (ratio of odds ratios (ROR) = 0.91, 95% credible interval (CrI): 0.86, 0.98), allocation concealment (ROR = 0.92, 95% CrI: 0.86, 0.98), and blinding (ROR = 0.87, 95% CrI: 0.80, 0.93). In contrast to previous work, we did not observe consistently different bias for subjective outcomes compared with mortality. However, we found an increase in between-trial heterogeneity associated with lack of blinding in meta-analyses with subjective outcomes. Inconsistency in criteria for risk-of-bias judgements applied by individual reviewers is a likely limitation of routinely collected bias assessments. Inadequate randomization and lack of blinding may lead to exaggeration of intervention effect estimates in randomized trials. Keywords: allocation concealment, bias, blinding, meta-analysis, missing data, randomization, randomized trials Meta-analyses of randomized trials are often more influential than single trials, and they increasingly inform health-care decisions made by clinicians and health authorities. For their results to be valid, randomized trials should employ rigorous methods that can achieve and preserve comparability of the intervention and control groups (1). For example, concealment of randomized allocation prevents an influence of patient characteristics on allocation to intervention and control groups; blinding of participants and trial personnel prevents differences in patient management between groups; and blinding of outcome assessors prevents knowledge of the assigned intervention group influencing outcome measurement. Randomized trials vary in methodological rigor, and flaws in trial conduct can lead to biased estimation of the intervention effect (2). Systematic reviewers should therefore assess the risk of bias in intervention effect estimates from each included trial. Meta-epidemiologic studies analyze collections of meta-analyses to provide empirical evidence about the influence of trial design characteristics on trial results (3). Such studies have, however, reached differing conclusions about which trial design characteristics most influence their results (4–8). For example, 4 studies found that lack of adequate allocation concealment was associated with overestimation of treatment effect (9–12), while several other studies did not find evidence for this (4, 5, 13–15). In a previous study, we explored reasons for these discrepancies by combining data from 7 meta-epidemiologic studies (16, 17). To our knowledge, this was the first study to explore the effects of bias on between– and within–meta-analysis heterogeneity using Bayesian hierarchical models. The results suggested that trial results based on subjectively assessed outcomes are more susceptible to bias and that the effect of bias is unpredictable, leading to increased heterogeneity in meta-analyses assessing subjective outcomes (16, 17). Further investigation of the effects of trial characteristics across different interventions, settings, and outcomes in larger collections of meta-analyses (not previously used) may provide more clarity and resolve inconsistencies between previous empirical studies. Since January 2008, authors of Cochrane reviews have used a “risk-of-bias” tool for assessing included trials (18). The assessors make judgements in relation to “sequence generation,” “allocation concealment,” “blinding of participants, personnel, and outcome assessors,” “incomplete outcome data,” “selective outcome reporting,” and a general category of “other potential threats to validity.” For each of these areas, review authors record whether there was a judgement of low, high, or unclear risk of bias for each trial, together with comments or quotes to justify each judgement. Accumulated standardized risk-of-bias assessments are a potentially useful resource for meta-epidemiologic research. In this paper, we describe and report the main results from a new, large empirical study investigating the associations of risk-of-bias judgements for sequence generation, allocation concealment, blinding, and incomplete outcome data with treatment effect estimates— the Risk of Bias in Evidence Synthesis (ROBES) Study. Our aims were to examine whether routinely collected risk-of-bias assessments relating to methodological characteristics are associated with effect estimates, to compare these associations with findings from our previous study (17), and to examine further the effect of outcome types in a new collection of meta-analyses. METHODS Data source The April 2011 issue of the Cochrane Database of Systematic Reviews (issue 4) included 4,371 intervention reviews (excluding protocols), of which 1,399 had at least 2 completed domains in the Risk of Bias tables. The complete 1,399 reviews were supplied by the Cochrane Informatics and Knowledge Management Department in electronic format, as Review Manager (version 5.0) files (19). We converted these to a customized Microsoft Access database (Microsoft Corporation, Redmond, Washington) using bespoke software which we commissioned from Riskaware Ltd. (Bristol, United Kingdom). Data selection and categorization We selected meta-analyses that fulfilled the following criteria: 1) address a binary outcome; 2) include at least 5 randomized trials, each with at least 1 event across the 2 trial arms; 3) accompanied by risk-of-bias assessments, with all 5 core domains of the tool having been assessed (sequence generation, allocation concealment, blinding, incomplete outcome data, and selective outcome reporting); 4) compare an active intervention with a control or “older” intervention; and 5) include no trials that overlap with another meta-analysis in the data set. Details of the process for selecting eligible meta-analyses are provided in Web Appendix 1 (available at https://academic.oup.com/aje). Meta-analyses can inform estimation of the bias associated with a particular domain only if they contain at least 1 trial at “low risk” of bias and 1 at “high or unclear” risk of bias. We refer to these as informative meta-analyses for that bias domain. We categorized each meta-analysis according to objectivity of the outcome measure (see below), direction of outcome (adverse or favorable) (16, 17), type of intervention (pharmacological, surgical, psychosocial and behavioral, care pathways, or other), clinical area (based on the World Health Organization’s International Classification of Diseases, Tenth Revision) (20), and whether the comparator was an active intervention (i.e., not a placebo, untreated, or standard care). Classification of outcome measure objectivity followed the method of Savović et al. (16, 17): We categorized outcome measures as 1) all-cause mortality; 2) other objectively assessed outcome (including live birth, noncephalic birth, low birth weight, miscarriage, pregnancy, and all automated laboratory outcomes); 3) semiobjective outcome (where the outcome event is considered to be measured accurately but the decision behind it is influenced by a clinician’s or patient’s judgement (e.g., hospital admission or readmission, study dropout/withdrawal for any reason, treatment completion, cesarean delivery, spontaneous vaginal birth, operative/assisted delivery, conversion to open surgery, additional treatments administered)); or 4) subjectively assessed outcome (e.g., clinician-assessed outcomes, symptoms and symptom scores, pain, mental health outcomes, cause-specific mortality). Too few meta-analyses had outcomes in the objective

Read the complete text