External validation of a measurement tool to assess systematic reviews (AMSTAR).

Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, Ramsay T, Bai A, Shukla VK, Grimshaw JM. · 2007

Read External validation of a measurement tool to assess systematic reviews (AMSTAR). by Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, Ramsay T, Bai A, Shukla VK, Grimshaw JM. (2007) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc2131785.txt.

Introduction

Shea and colleagues validated AMSTAR in 2007, a tool for judging the quality of systematic reviews. Thousands of reviews exist, but their methods vary widely in quality. Appraisal should be routine before results guide care. Two assessors applied AMSTAR to 42 reviews of acid-related diseases, including reflux and peptic ulcer treatments. Independent assessors also gave each review a global quality judgement for comparison.

AMSTAR scores in the sample ranged from 0 to 10 out of 11, and averaged 4.6 across the reviews assessed. That shows a wide quality spread. The study also checked how consistently independent raters scored the same reviews with this measurement tool. Agreement between raters was excellent for the total score. Nine items reached near-perfect agreement. AMSTAR results converged with the global assessments, supporting construct validity. The smallest detectable difference between reviews was 0.64 points.

Catalog record

CREATORShea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, Ramsay T, Bai A, Shukla VK, Grimshaw JM.
YEAR2007
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC2131785

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR) By Beverley J Shea, Lex M Bouter, Joan Peterson, Maarten Boers, Neil Andersson, Zulma Ortiz, Tim Ramsay, Annie Bai, Vijay K Shukla, Jeremy M Grimshaw, Joel Gagnier Published in: PLoS ONE DOI: 10.1371/journal.pone.0001350 ABSTRACT Abstract Background Thousands of systematic reviews have been conducted in all areas of health care. However, the methodological quality of these reviews is variable and should routinely be appraised. AMSTAR is a measurement tool to assess systematic reviews. Methodology AMSTAR was used to appraise 42 reviews focusing on therapies to treat gastro-esophageal reflux disease, peptic ulcer disease, and other acid-related diseases. Two assessors applied the AMSTAR to each review. Two other assessors, plus a clinician and/or methodologist applied a global assessment to each review independently. Conclusions The sample of 42 reviews covered a wide range of methodological quality. The overall scores on AMSTAR ranged from 0 to 10 (out of a maximum of 11) with a mean of 4.6 (95% CI: 3.7 to 5.6) and median 4.0 (range 2.0 to 6.0). The inter-observer agreement of the individual items ranged from moderate to almost perfect agreement. Nine items scored a kappa of >0.75 (95% CI: 0.55 to 0.96). The reliability of the total AMSTAR score was excellent: kappa 0.84 (95% CI: 0.67 to 1.00) and Pearson's R 0.96 (95% CI: 0.92 to 0.98). The overall scores for the global assessment ranged from 2 to 7 (out of a maximum score of 7) with a mean of 4.43 (95% CI: 3.6 to 5.3) and median 4.0 (range 2.25 to 5.75). The agreement was lower with a kappa of 0.63 (95% CI: 0.40 to 0.88). Construct validity was shown by AMSTAR convergence with the results of the global assessment: Pearson's R 0.72 (95% CI: 0.53 to 0.84). For the AMSTAR total score, the limits of agreement were −0.19±1.38. This translates to a minimum detectable difference between reviews of 0.64 ‘AMSTAR points’. Further validation of AMSTAR is needed to assess its validity, reliability and perceived utility by appraisers and end users of reviews across a broader range of systematic reviews. INTRODUCTION High quality systematic reviews are increasingly recognized as providing the best evidence to inform health care practice and policy [1]. The quality of a review, and so its worth, depends on the extent to which, scientific review methods were used to minimize the risk of error and bias. The quality of published reviews can vary considerably, even when they try to answer the same question [2]. As a result, it is necessary to appraise their quality (as is done for any research study) before the results are implemented into clinical or public health practice. Much has been written on how best to appraise systematic reviews, and while there is some variation on how this is achieved, most agree on key components of the critical appraisal [3]. Methodological quality can be defined as the extent to which the design of a systematic review will generate unbiased results [4]. Several instruments exist to assess the methodological quality of systematic reviews [5], but not all of them have been developed systematically or empirically validated and have achieved general acceptance. The authors of this paper acknowledge that the methodological quality and reporting quality for systematic reviews is very different. The first, methodological quality, considers how well the systematic review was conducted (literature searching, pooling of data, etc.). The second, reporting quality, considers how well systematic reviewers have reported their methodology and findings. Existing instruments often try to include both types of methods without being conceptually clear about the differences. In an attempt to achieve some consistency in the evaluation of systematic reviews we have developed a tool to assess their methodological quality. This builds on previous work [6], and is based on empirical evidence and expert consensus. A measurement tool to assess systematic reviews (AMSTAR) was highly rated in a recent review (personal communication) of quality assessment instruments performed by the Canadian Agency for Drugs and Technologies in Health (CADTH). In this study we present the results of an external validation of AMSTAR using data from a series of systematic reviews obtained from the gastroenterology literature. METHODS The characteristics and basic properties of the instrument have been described elsewhere [7]. Briefly, a 37-item initial assessment tool was formed by combining a) the enhanced Overview Quality Assessment Questionnaire (OQAQ) scale, b) a checklist created by Sacks, and c) three additional items recently judged by experts in the field to be of methodological importance. In its development phase the instrument was applied to 99 paper-based and 52 electronic systematic reviews [6] [7]. Exploratory factor analysis was used to identify underlying components. The results were considered by methodological experts using a nominal group process to reduce the number of items and design an assessment tool with face and content validity. This process lead to an 11-item instrument [7]. A description of the instrument is provided in Annex S1. External validity For our validation test set we chose to use systematic reviews or meta-analyses in the area of gastroenterology, specifically upper gastrointestinal. CADTH's informational specialist searched electronic bibliographic databases (i.e. Medline, Central and EMBASE) up to and including 2005. A total of 42 systematic reviews met the a priori criteria and were included [8]. This sample included seven electronic Cochrane systematic reviews and 35 paper-based non-Cochrane reviews. The topics of the reviews ranged across the spectrum of GI problems like dyspepsia, gastro-esophageal reflux disease (GERD), peptic ulcer disease (PUD), and also GI drug interventions such as H2 receptor antagonists and proton pump inhibitors [9]–[50]. Two CADTH assessors from two review groups (SS and FA, AL and CY) independently applied AMSTAR to each review and reached agreement on the assessment results. To assess construct validity, two reviewers (JP, ZO) plus a clinician and/or methodologist (MB, DF, DP, MO, and DH) applied a global assessment to each review [51] (Annex S2). Agreement and reliability We calculated an overall agreement score using the weighted Cohen's kappa, as well as one for each item [52] (Table 1). Bland and Altman's limits of agreement methods were used to display agreement graphically [53], [54] (Fig. 1). We calculated the percentage of the theoretical maximum score. Pearson's Rank correlation coefficients were used to assess reliability of this total score. For comparisons of rating the methodological quality we calculated chance-corrected agreement (using kappa) and chance-independent agreement (using Φ) [52], [55], [56]. We accepted a correlation of >0.66. We further scrutinized items and reviews with kappa scores below 0.66 [52]. Kappa values of less than 0 rate as less than chance agreement; 0.01–0.20 slight agreement; 0.21–0.40 fair agreement; 0.41–0.60 moderate agreement; 0.61–0.80 substantial agreement; and 0.81–0.99 almost perfect agreement [52], [57]. We calculated PHI Φ for each question [55], [58]. [Table: Assessment of the inter-rater agreement for AMSTAR] [Figure: Bland and Altman limits of agreement plot for AMSTAR scores.] Construct validity We assessed construct validity (i.e. evaluation of a hypothesis about the expected performance of an instrument) by converting the total mean score (mean of the two assessors) for each of the 42 reviews to a percentage of the maximum score for AMSTAR and of the maximum score of the global assessment instrument. We used

Read the complete text