Inflated Impact Factors? The True Impact of Evolutionary Papers in Non-Evolutionary Journals
Erik Postma · 2007
Read Inflated Impact Factors? The True Impact of Evolutionary Papers in Non-Evolutionary Journals by Erik Postma (2007) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/plos-pone-0000999.txt.
Introduction
Erik Postma questioned whether evolution papers earn inflated impact scores outside their own field, in a solo PLOS ONE article from 2007. Impact factors are used to judge journal quality, yet fields differ sharply in how much they cite. A paper's score may say more about its neighborhood than its worth.
Postma tests one specific suspicion: that evolutionary biology papers placed in general journals gain an extra lift. Fair comparisons should stay inside a field, in theory. His analysis checks whether even that fix holds. The paper is a measurement of measurement itself, aimed at how hiring and funding panels read scores.
Catalog record
| CREATOR | Erik Postma |
|---|---|
| YEAR | 2007 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| LICENSE | Creative Commons Attribution License |
| RIGHTS | CC BY |
| SOURCE | PLOS ONE |
Source: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0000999
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inflated Impact Factors? The True Impact of Evolutionary Papers in Non-Evolutionary Journals Authors: Erik Postma Published: 2007, PLOS ONE DOI: 10.1371/journal.pone.0000999 ABSTRACT Amongst the numerous problems associated with the use of impact factors as a measure of quality are the systematic differences in impact factors that exist among scientific fields. While in theory this can be circumvented by limiting comparisons to journals within the same field, for a diverse and multidisciplinary field like evolutionary biology, in which the majority of papers are published in journals that publish both evolutionary and non-evolutionary papers, this is impossible. However, a journal's overall impact factor may well be a poor predictor for the impact of its evolutionary papers. The extremely high impact factors of some multidisciplinary journals, for example, are by many believed to be driven mostly by publications from other fields. Despite plenty of speculation, however, we know as yet very little about the true impact of evolutionary papers in journals not specifically classified as evolutionary. Here I present, for a wide range of journals, an analysis of the number of evolutionary papers they publish and their average impact. I show that there are large differences in impact among evolutionary and non-evolutionary papers within journals; while the impact of evolutionary papers published in multidisciplinary journals is substantially overestimated by their overall impact factor, the impact of evolutionary papers in many of the more specialized, non-evolutionary journals is significantly underestimated. This suggests that, for evolutionary biologists, publishing in high-impact multidisciplinary journals should not receive as much weight as it does now, while evolutionary papers in more narrowly defined journals are currently undervalued. Importantly, however, their ranking remains largely unaffected. While journal impact factors may thus indeed provide a meaningful qualitative measure of impact, a fair quantitative comparison requires a more sophisticated journal classification system, together with multiple field-specific impact statistics per journal. INTRODUCTION Despite the fact that most scientists, funding organisations, promotion committees and journal editors are very much aware of the numerous problems associated with the use of journal impact factors as a measure of scientific quality or even impact [1] – [3] , impact factors continue to be amongst the most commonly used measures of journal quality, and thereby of individual papers and their authors. Consequently, researchers take the impact factor of a journal into account when deciding where to submit their work for publication, while journal editors may have the impact factor of their journal in mind when deciding which manuscripts to accept for publication [3] . One of the main problems associated with the use of impact factors as an objective measure of either quality or impact are the large and systematic differences that exist among different scientific disciplines, with impact factors of evolutionary and ecological journals being on the low end of the spectrum [4] , [5] . It is these systematic differences among scientific disciplines, which are unrelated to the quality or the size of the field, that make it impossible to directly compare impact factors of journals from different fields [4] – [6] . Indeed, for its annual Journal Citation Reports (JCR), ISI Scientific categorises journals by subject, and journals typically advertise not only their impact factor, but also their ranking within their subject category. Unfortunately however, in a diverse and multidisciplinary field like evolutionary biology, these categories, or any similarly non-hierarchical classification system in which journals can only be a member of a few categories at most for that matter, is far from satisfactory. For example, of the about 5000 articles published between 1996 and 2006 with the term “sexual selection” in either the title, keywords or abstract, only one-fifth was published in journals that are classified by ISI Scientific under ‘ Evolutionary Biology ’ (542 of which were published in J ournal of E volutionary B iology or E volution ), while 423 were published in P roceedings of the R oyal S ociety B (classified under Biology ), and 392 in A nimal B ehaviour (classified under Zoology and Behavioural Sciences ). Consequently, we simply cannot avoid making comparisons among journals from a wide range of subject categories, including Evolutionary Biology , but also, for example, Zoology, Ornithology and Genetics & Heredity. The problem is particularly obvious when comparing impact factors of biological journals on the one hand, and those journals that are classified under Multidisciplinary Sciences , which by their very nature publish articles from a wide range of scientific fields. Indeed, there is a general feeling that the extremely high impact factors of multidisciplinary journals like N ature and S cience are largely driven by the non-evolutionary and/or non-biological papers they publish [also see e.g. 7] . If this was indeed true, this would imply that, at least for an evolutionary biologist, publishing in these journals is currently overvalued. On the other hand, however, the impact factor of journals that are narrower in focus, and in particular of those limited to a particular taxon, may well underestimate the impact of the evolutionary papers they publish. In other words, what is the true impact of evolutionary papers in non-evolutionary journals? RESULTS Not surprisingly, there are highly significant differences among journals in the impact of the evolutionary papers they publish, as well as in their overall impact ( Figure 1A ). More interestingly, however, the size and/or direction of the difference between these two measures of impact varies among journals, as is indicated by the significant interaction between interaction between Evolutionary vs. Overall impact and Journal in Table 1 . In other words, the overall impact of a journal does not necessarily provide a good predictor of the impact of the evolutionary papers it publishes. A) The mean absolute impact of evolutionary papers for 39 journals from a range of different categories. The dotted line gives their overall impact. B) The impact of evolutionary articles relative to the impact of the average article for each of these journals. Relative impacts marked with an asterisk are significantly different from zero at the 5% level. As indicated by the significant Evolutionary vs. Overall impact × Journal interaction, the size and/or direction of the difference in impact between the evolutionary papers a journal publishes and its overall impact factor varies significantly among journals. Indeed, Figure 1B shows clearly that for many journals there are large and often significant differences between the impact of the evolutionary articles a journal publishes and a journal's overall impact. While the impact of evolutionary articles published in journals classified under Multidisciplinary Sciences are typically between 15 to 30% lower than the impact of the average paper that is published in these journals, evolutionary articles in many of the more specialized, but not explicitly evolutionary, journals are up to 40% percent higher. Interestingly, also the relative impact of ‘evolutionary’ articles (i.e. an article with one of the ten evolutionary keywords in title, abstract or keywords) published in E volution and J ournal of E volutionary B iology is significantly greater than zero (although less than 10%). This implies that the ten keywords used to classify an article as ‘evolutionary’ (see Materials and Methods for more details) may not be representative of the whole field of evolutionary biology. To further test for differences among