Newcastle-Ottawa Scale: comparing reviewers' to authors' assessments.

Lo CK, Mertz D, Loeb M. · 2014

Read Newcastle-Ottawa Scale: comparing reviewers' to authors' assessments. by Lo CK, Mertz D, Loeb M. (2014) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc4021422.txt.

Introduction

Carson Ka-Lok Lo, Dominik Mertz, and Mark Loeb test whether a common bias scoring tool gives the same grade in unlike hands, in BMC Medical Research Methodology in 2014. The Newcastle-Ottawa Scale grades cohort studies for risk of bias and is backed by a major review body. Sparse reporting in papers can bend such grades.

The team took cohort studies from a flu review and asked their own authors to score them by survey. Reviewers of the flu project had scored the same studies. Just over a third of authors replied. Their grades ran lower than the reviewers' grades for the same work. Agreement item by item ranged from slight to worse than chance, and overall agreement was near zero. The gap suggests reviewers should ask authors for facts missing from print before scoring. Without that step, the scale's grades may say more about reporting than about study quality.

Catalog record

CREATORLo CK, Mertz D, Loeb M.
YEAR2014
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC4021422

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments By Carson Ka-Lok Lo, Dominik Mertz, Mark Loeb Published in: BMC Medical Research Methodology DOI: 10.1186/1471-2288-14-45 ABSTRACT Abstract Background Lack of appropriate reporting of methodological details has previously been shown to distort risk of bias assessments in randomized controlled trials. The same might be true for observational studies. The goal of this study was to compare the Newcastle-Ottawa Scale (NOS) assessment for risk of bias between reviewers and authors of cohort studies included in a published systematic review on risk factors for severe outcomes in patients infected with influenza. Methods Cohort studies included in the systematic review and published between 2008–2011 were included. The corresponding or first authors completed a survey covering all NOS items. Results were compared with the NOS assessment applied by reviewers of the systematic review. Inter-rater reliability was calculated using kappa (K) statistics. Results Authors of 65/182 (36%) studies completed the survey. The overall NOS score was significantly higher (p < 0.001) in the reviewers’ assessment (median = 6; interquartile range [IQR] 6–6) compared with those by authors (median = 5, IQR 4–6). Inter-rater reliability by item ranged from slight (K = 0.15, 95% confidence interval [CI] = −0.19, 0.48) to poor (K = −0.06, 95% CI = −0.22, 0.10). Reliability for the overall score was poor (K = −0.004, 95% CI = −0.11, 0.11). Conclusions Differences in assessment and low agreement between reviewers and authors suggest the need to contact authors for information not published in studies when applying the NOS in systematic reviews. Keywords: Newcastle Ottawa Scale, Inter-rater, Reliability, Validity, Risk of bias, Observational studies BACKGROUND One of the most important aspects of systematic reviews is the assessment of the risk of bias of included studies [1]. Evaluating the reliability and validity of risk of bias assessment scores can help generate better assessment tools. The Newcastle-Ottawa Scale (NOS) is a risk of bias assessment tool for observational studies that is recommended by the Cochrane Collaboration [1,2]. Some characteristics of this tool have been assessed, such as inter-rater reliability, which has been variable ranging from fair to high depending on the types of observational studies assessed and the raters [3,4]. In addition to the potentially limited inter-rater reliability, lack of reporting of methodological details in published articles may potentially distort the risk of bias assessment as it was shown for randomized controlled trials (RCT) [5,6]. The objectives of our study were therefore to compare the NOS assessment between reviewers with access to the publications only and the authors of published studies, using the list of studies included in a recently published systematic review of observational studies on risk factors to predict severe outcomes in patients with influenza [7]. We hypothesized that reviewers in the systematic review who had to rely on the published information would have assigned lower NOS scores (i.e., greater risk of bias) compared with its authors, and that the agreement between authors and reviewers would be marginal. METHODS Eligibility criteria Authors of studies included in a published systematic review on risk factors for severe outcomes in patients with influenza and published between 2008 and 2011 were included [7]. More remote publications were not considered, as the contact information was more likely no longer accurate and authors may not recall details of their studies. Quality assessment: NOS and NOS-derived survey The NOS assigns up to a maximum of nine points for the least risk of bias in three domains: 1) selection of study groups (four points); 2) comparability of groups (two points); and 3) ascertainment of exposure and outcomes (three points) for case–control and cohort studies, respectively (Additional file 1) [2]. Survey questions were developed based on the NOS’ questions covering all three domains so that authors could provide detailed information about their studies. Because the NOS questionnaires are slightly different between case–control and cohort studies, two surveys were created. Vaccination status for influenza and antiviral treatment were defined as the most important covariates that defined comparability, consistent with the NOS assessment conducted by the reviewers in the systematic review [7]. The corresponding author and the first author were contacted by email and were provided with the link to the actual electronic survey. An online survey was provided to all authors where their name, responses and timestamp were securely documented. Authors were required to respond to all questions provided prior to submission and answered follow-up questions only when prompted to. An example of a follow-up question: For ascertainment of exposure, authors who claimed using structured interviews had to also report whether the interviewers were blinded to exposure status. A feedback form was provided at the end of the survey for authors to comment on the survey. A total of two reminders were sent to authors not completing the survey. Analysis Questions in the survey were matched to the original NOS items, so that points were awarded to the respective answers. Differences in the NOS scores between the reviewers’ and authors’ assessment were compared with the Wilcoxon paired sign-rank test. Inter-rater agreement Cohen’s kappa was used to compare agreement between the reviewers from the systematic review and the authors’ assessments. The scores were compared between raters across each NOS domain or item. Weighted kappa was performed for variables with more than two classifications (i.e., comparability had scores of 0, 1, and 2). A weighted kappa was also performed for the total NOS scores. Agreement interpretation was based on established categorizations: poor (K < 0.00), slight (0.00-0.20), fair (0.21-0.40), moderate (0.41-0.60), substantial (0.61-0.80), and almost perfect (0.81-1.00) [8]. MedCalc® was used for all weighted/non-weighted kappa analyses [9,10] and for the Wilcoxon paired sign-rank test. Mean and standard deviation (SD), median and interquartile range (IQR) were reported. RESULTS Of the 239 studies included in the systematic review, 196 studies (82%; 6 case–control and 190 cohort studies) met the inclusion criteria for this study (i.e., published from 2008 to 2011). Six cohort studies were excluded, as there was no information on contacting authors (three articles published by the World Health Organization (WHO) and three articles by Centre for Disease Control and Prevention (CDC)). An additional two studies were excluded, as contacting authors could not be identified. Authors of 65/182 (36%) eligible cohort studies and 2/6 (33%) case–control studies completed the survey (Additional files 2 and 3). Given the sparse data on case–control studies, these studies were excluded from analysis. Reviewer assessment was similar for included (n = 65, mean NOS score = 5.85, median = 6, IQR 6–6) and excluded studies (n = 117, mean = 5.81, median = 6, IQR 5–6). The distribution of NOS scores by reviewers and authors for cohort studies included in this analysis is shown in Figure 1. The risk of bias by the study authors was felt to be higher, with a lower average NOS score (mean = 4.85, SD = 1.19, median = 5, IQR 4–6) compared with the NOS reviewers (mean = 5.85, SD = 0.94, median = 6, IQR 6–6). The difference was statistically significant (p < 0.001, Wilcoxon paired signed-rank test). Thirty (46%) out of the 65 cohort studies had overall NOS scores that were two or more points apart, of which 27 studies had reviewers scoring two or more

Read the complete text