The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching.
Haddaway NR, Collins AM, Coughlin D, Kirk S. · 2015
Read The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. by Haddaway NR, Collins AM, Coughlin D, Kirk S. (2015) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc4574933.txt.
Introduction
Neal Robert Haddaway and four co-authors test Google Scholar as a tool for evidence reviews, in PLoS ONE in 2015. Systematic reviews must search grey literature, work outside commercial journals, and Google Scholar is free and indexes tens of millions of records, academic and grey alike. Whether it can carry a review search on its own was hotly debated and untested for grey material.
Using environmental science reviews as test cases, the team found grey items made up a moderate share of Scholar results, typically surfacing near page 80 of the results. Scholar also retrieved most items found through Web of Science when searched directly, but overlap between the two engines on matched search strings was only moderate to poor. The verdict: a useful supplement, not a standalone source.
Catalog record
| CREATOR | Haddaway NR, Collins AM, Coughlin D, Kirk S. |
|---|---|
| YEAR | 2015 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching By Neal Robert Haddaway, Alexandra Mary Collins, Deborah Coughlin, Stuart Kirk, K Brad Wray Published in: PLoS ONE DOI: 10.1371/journal.pone.0138237 ABSTRACT Abstract Google Scholar (GS), a commonly used web-based academic search engine, catalogues between 2 and 100 million records of both academic and grey literature (articles not formally published by commercial academic publishers). Google Scholar collates results from across the internet and is free to use. As a result it has received considerable attention as a method for searching for literature, particularly in searches for grey literature, as required by systematic reviews. The reliance on GS as a standalone resource has been greatly debated, however, and its efficacy in grey literature searching has not yet been investigated. Using systematic review case studies from environmental science, we investigated the utility of GS in systematic reviews and in searches for grey literature. Our findings show that GS results contain moderate amounts of grey literature, with the majority found on average at page 80. We also found that, when searched for specifically, the majority of literature identified using Web of Science was also found using GS. However, our findings showed moderate/poor overlap in results when similar search strings were used in Web of Science and GS (10–67%), and that GS missed some important literature in five of six case studies. Furthermore, a general GS search failed to find any grey literature from a case study that involved manual searching of organisations’ websites. If used in systematic reviews for grey literature, we recommend that searches of article titles focus on the first 200 to 300 results. We conclude that whilst Google Scholar can find much grey literature and specific, known studies, it should not be used alone for systematic review searches. Rather, it forms a powerful addition to other traditional search methods. In addition, we advocate the use of tools to transparently document and catalogue GS search results to maintain high levels of transparency and the ability to be updated, critical to systematic reviews. INTRODUCTION Searching for information is an integral part of research. Over 11,500 journals are catalogued by Journal Citation Reports (http://thomsonreuters.com/journal-citation-reports/), and the volume of published scientific research is growing at an ever-increasing rate [1,2]. Scientists must sift through this information to find relevant research, and do so today most commonly by using online citation databases (e.g. Web of Science) and search engines (e.g. Google Scholar). Just as the number of academic articles and journals is steadily increasing, so too are the number of citation databases. A citation database is a set of citations that can be searched using an online tool, for example Web of Science (https://webofknowledge.com/). These databases typically charge subscription fees for access to the database that do not cover the cost of access to the full text of the research articles themselves. Generally these databases selectively catalogue citations according to a predefined list of journals, publishers or subject areas. Several free-to-use services have recently appeared that search for citations on the internet, most notably Google Scholar and Microsoft Academic Search. These search engines do not store citations within a specific database, instead they regularly ‘crawl’ the internet for information that appears to be a citation. Some key characteristics of databases and search engines are compared in Table 1. [Table: Typical characteristics of academic citation databases and search engines.] According to Thomson Reuters, the Web of Science Core Collections citation database contains almost 50 million research records (http://wokinfo.com/citationconnection/realfacts/; February 2015), with Microsoft Academic Search reporting to catalogue in excess of 45 million records as of January 2013 (http://academic.research.microsoft.com/About/help.htm#9). Google Scholar does not report the volume of citations identifiable via their search facility, although attempts have been made to estimate this that suggest between 1.8 million [3] and 100 million records [4] are identifiable. “Grey literature” is the term given to describe documents not published by commercial publishers, and it may form a vital component of evidence reviews such as systematic reviews and systematic maps [5], rapid evidence assessments [6] and synopses [7]. Grey literature includes academic theses, organisation reports, government papers, etc. and may prove highly influential in syntheses, despite not being formally published in the same way as traditional academic literature e.g. [8]. Considerable efforts are typically required within systematic reviews to search for grey literature in an attempt to include practitioner-held data and also account for possible publication bias [5,9]. Publication bias is the tendency for significant, positive research to be more likely to be published than non-significant or negative research, leading to an increased likelihood of overestimating effect sizes in meta-analyses and other syntheses [10]. The inclusion of grey literature is a central tenet of systematic review methodology, which aims to include all available documented evidence and reduce susceptibility to bias. Academic citation databases are often the first port of call for researchers looking for information. However, access to databases is often expensive; some costing c. £100,000 per annum for organisations of up to 100 employees. Increasingly, researchers are using academic citation search engines to find information (Haddaway, unpublished data). Academic citation search engines appear to represent an attractive alternative to costly citation databases, cataloguing research almost immediately and not restricting results to certain journals, publishers or subject categories. Search engines are particularly attractive to systematic reviewers, since they have the potential to be used to search for grey literature quickly and simply using one search facility rather than a plethora of individual websites [5]. There is on-going debate regarding the utility of Google Scholar as an academic resource e.g. [11,12], but also as a replacement for traditional academic citation databases and in searches for grey literature in systematic reviews [13,14]. Google Scholar represents an attractive resource for researchers, since it is free-to-use, appears to catalogue vast numbers of academic articles, allows citations to be exported individually, and also provides citation tracking (although see criticism of citation tracking by Delgado Lopez-Cozar et al. [15]). Google Scholar is also potentially useful in systematic reviews, since reliance on just one such platform for searches would: i) offer resource efficiency, ii) offer cost efficiency, iii) allow rapid linking to full texts, iv) provide access to a substantial body of grey literature as well as academic literature, and v) be compatible with new methods for downloading citations in bulk that would allow for a very transparent approach to searching [16]. Previous research has shown that articles identified within systematic reviews are identifiable using Google Scholar [13]. However, other authors have suggested that this does not make Google Scholar an appropriate replacement for academic citation databases, as, in practice, there are considerable limitations in the search facility relative to those of academic databases [11], and there is on-going debate about Google Scholar’s place in research [12]. Shultz [17] listed many limitations that have been attributed to Google Scholar, including that the service permits use of only basic Boolean operators in search strings, which are limited to 256 characters, and that users cannot sort results (although some of the other cited disadvantages have been corrected in recent updates). Two further limitations to the use of Google Scholar in academic searches are the inability to directly export