Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level.

Hutchins BI, Yuan X, Anderson JM, Santangelo GM. · 2016

Read Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level. by Hutchins BI, Yuan X, Anderson JM, Santangelo GM. (2016) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc5012559.txt.

Introduction

B. Ian Hutchins, Xin Yuan, James M. Anderson, and George M. Santangelo, with a co-author, introduce a new citation measure in PLoS Biology in 2016. Judging a paper by its journal's impact factor is widely seen as flawed, yet hiring and grant panels still lean on rough numbers. Their Relative Citation Ratio instead rates each article against its own field.

The ratio divides a paper's citation rate by the rate expected for its field, where the field is drawn from the papers cited alongside it. Scores therefore compare fairly across unlike subjects. Tested on 88,835 articles from 2003 to 2010, the scores lined up well with expert judgments of influence, and grant holders held steady ranks across fields. A free web tool, iCite, lets anyone compute the ratio for papers listed in PubMed.

Catalog record

CREATORHutchins BI, Yuan X, Anderson JM, Santangelo GM.
YEAR2016
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
LICENSECC0 1.0 Universal
RIGHTSCC0
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC5012559

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level By B Ian Hutchins, Xin Yuan, James M Anderson, George M Santangelo, David L Vaux Published in: PLoS Biology DOI: 10.1371/journal.pbio.1002541 ABSTRACT Abstract Despite their recognized limitations, bibliometric assessments of scientific productivity have been widely adopted. We describe here an improved method to quantify the influence of a research article by making novel use of its co-citation network to field-normalize the number of citations it has received. Article citation rates are divided by an expected citation rate that is derived from performance of articles in the same field and benchmarked to a peer comparison group. The resulting Relative Citation Ratio is article level and field independent and provides an alternative to the invalid practice of using journal impact factors to identify influential papers. To illustrate one application of our method, we analyzed 88,835 articles published between 2003 and 2010 and found that the National Institutes of Health awardees who authored those papers occupy relatively stable positions of influence across all disciplines. We demonstrate that the values generated by this method strongly correlate with the opinions of subject matter experts in biomedical research and suggest that the same approach should be generally applicable to articles published in all areas of science. A beta version of iCite, our web tool for calculating Relative Citation Ratios of articles listed in PubMed, is available at https://icite.od.nih.gov. INTRODUCTION In the current highly competitive pursuit of research positions and funding support [1], faculty hiring committees and grant review panels must make difficult predictions about the likelihood of future scientific success. Traditionally, these judgments have largely depended on recommendations by peers, informal interactions, and other subjective criteria. In recent years, decision makers have increasingly turned to numerical approaches such as counting first or corresponding author publications, using the impact factor of the journals in which those publications appear, and computing Hirsch (i.e., h-index) values [2]. The widespread adoption of these metrics and the recognition that they are inadequate [3–6] highlight the ongoing need for alternative methods that can provide effectively normalized and reliable data-driven input to administrative decision making, both as a means of sorting through large pools of qualified candidates and as a way to help combat implicit bias. Though each of the above methods of quantitation has strengths, accompanying weaknesses limit their utility. Counting first or corresponding author publications does on some level reflect the extent of a scientist’s contribution to his or her field, but it has the unavoidable effect of privileging quantity over quality and may undervalue collaborative science [7]. Journal impact factor (JIF) was for a time seen as a valuable indicator of scientific quality because it serves as a convenient, and not wholly inaccurate, proxy for expert opinion [8]. However, its blanket use also camouflages large differences in the influence of individual papers. This is because impact factor is calculated as the average number of times articles published over a 2-y period in a given journal are cited; in reality, citations follow a log-normal rather than a Gaussian distribution [9]. Moreover, since practitioners in disparate fields have differential access to high-profile publication venues, impact factor is of limited use in multidisciplinary science-of-science analyses. Despite these serious flaws, JIF continues to have a large effect on funding and hiring decisions [4,10,11]. H-index, which attempts to assess the cumulative impact of the work done by an individual scientist, disadvantages early-career stage investigators; it also undervalues some fields of research by failing to normalize raw citation counts [6]. Many alternative methods for quantifying scientific accomplishment have been proposed, including citation normalization to journals or journal categories [12–19]; note that one of these is a previously described Relative Citation Rate [19], which should not be confused with the method we describe here. Other methods include citation percentiles [14,20], eigenvector normalization [21,22], and source normalization [13,23]; the latter includes both the mean normalized citation score (MNCS) [17] and source-normalized impact per paper metrics [15,17,22–26]. Although some of these methods have dramatically improved our theoretical understanding of citation dynamics [27–30], none have been widely adopted. To combine a further technical advance with a high likelihood of widespread adoption by varied stakeholders, including scientists, administrators, and funding agencies, a new citation metric must overcome several practical challenges. From a technical standpoint, a new metric must be article level, field-normalized in a way that is scalable from small to large portfolios without introducing significant bias at any level, and correlated with expert opinion. From an adoption standpoint, it should be freely accessible, calculated in a transparent fashion, and benchmarked to peer performance in a way that facilitates meaningful interpretation. Such an integrated benchmark, or comparison group, is not used by any currently available citation-based metric. Instead, all current measures aggregate articles from researchers across disparate geographical regions and institutional types, so that, for example, there is no easy way for primarily undergraduate institutions to directly compare the work they support against that of other teaching-focused institutions or for developing nations to compare their research output to that of other developing nations [31]. Enabling these and other apples-to-apples comparisons would greatly facilitate decision making by research administrators. We report here the development and validation of the Relative Citation Ratio (RCR) metric, which is based on the novel idea of using each article’s co-citation network to field- and time-normalize the number of citations it has received; this topically linked cohort is used to derive an expected citation rate (ECR), which serves as the ratio’s denominator. As is true of other bibliometrics, article citation rate (ACR) is used as the numerator. Unlike other bibliometrics, though, RCR incorporates a customizable benchmarking feature that relates field- and time-normalized citations to the performance of a peer comparison group. RCR also meets or exceeds the standards set by other current metrics with respect to the ambitious ideals set out above. We use the RCR metric here to determine the extent to which National Institutes of Health (NIH) awardees maintain high or low levels of influence on their respective fields of research. RESULTS Co-citation Networks Represent an Article’s Area of Influence Choosing to cite is the long-standing way in which one scholar acknowledges the relevance of another’s work. However, the utility of citations as a metric for quantifying influence has been limited, primarily because it is difficult to compare the value of one citation to another; different fields have different citation behaviors and are composed of widely varying numbers of potential citers [32,33]. An effective citation-based evaluative tool must also take into account the length of time a paper has been available to potential citers, since a recently published article has had less time to accumulate citations than an older one. Finally, fair comparison is complicated by the fact that an author’s choice of which work to cite is not random; a widely known paper is more likely to be referenced than an obscure one of equal relevance. This is because the accrual of citations follows a power law or log-normal pattern, in accordance with a process called preferential attachment [27,33,34]. Functionally this means that, each time a paper is cited, it is a priori more likely to be

Read the complete text