TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences.
Litman L, Robinson J, Abberbock T. · 2017
Read TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. by Litman L, Robinson J, Abberbock T. (2017) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc5405057.txt.
Introduction
Leib Litman, Jonathan Robinson, and Tzvi Abberbock describe an online research platform called TurkPrime in a 2017 paper in Behavior Research Methods. Amazon's Mechanical Turk lets scientists gather behavior data fast from paid online workers, but it was built for business tasks, not research. Many study needs, such as barring past takers or running studies over time, are slow or hard to set up there.
TurkPrime sits on top of that worker pool and adds the missing research tools. It can screen out past takers, track dropouts, send group emails and bonuses, change a running study, and speed approvals. The authors say these tools save time, lift data quality, and allow study designs that were near impossible before. The platform is free for researchers to use.
Catalog record
| CREATOR | Litman L, Robinson J, Abberbock T. |
|---|---|
| YEAR | 2017 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences By Leib Litman, Jonathan Robinson, Tzvi Abberbock Published in: Behavior Research Methods DOI: 10.3758/s13428-016-0727-z ABSTRACT In recent years, Mechanical Turk (MTurk) has revolutionized social science by providing a way to collect behavioral data with unprecedented speed and efficiency. However, MTurk was not intended to be a research tool, and many common research tasks are difficult and time-consuming to implement as a result. TurkPrime was designed as a research platform that integrates with MTurk and supports tasks that are common to the social and behavioral sciences. Like MTurk, TurkPrime is an Internet-based platform that runs on any browser and does not require any downloads or installation. Tasks that can be implemented with TurkPrime include: excluding participants on the basis of previous participation, longitudinal studies, making changes to a study while it is running, automating the approval process, increasing the speed of data collection, sending bulk e-mails and bonuses, enhancing communication with participants, monitoring dropout and engagement rates, providing enhanced sampling options, and many others. This article describes how TurkPrime saves time and resources, improves data quality, and allows researchers to design and implement studies that were previously very difficult or impossible to carry out on MTurk. TurkPrime is designed as a research tool whose aim is to improve the quality of the crowdsourcing data collection process. Various features have been and continue to be implemented on the basis of feedback from the research community. TurkPrime is a free research platform. Access to participants is of central importance to researchers in the social and behavioral sciences. Until the late 1990s, most behavioral researchers had to rely on undergraduate subject pools as their primary source of research participants. University subject pools have a number of limitations, including lack of representativeness (Henrich, Heine, & Norenzayan, 2010), and can often be labor-intensive and time-consuming to use. Most importantly, such subject pools are not available to faculty in many small schools (see Kraut et al., 2004). Starting in the late 1990s, the Internet became more commonly used as a source of participant recruitment (Kraut et al., 2004). Recruiting participants over the Internet provided a number of advantages over the use of the traditional subject pool (Gosling & Johnson, 2010): Researchers were able to recruit vastly more participants in much shorter time periods (Nosek, Banaji, & Greenwald, 2002), and the data were typically more representative and less labor-intensive to acquire (Kraut et al., 2004). Accessing large numbers of participants from the Internet, commonly referred to as crowdsourcing, also has substantial limitations. For example, finding participants, incentivizing participation, and preventing multiple participation are just some of the common barriers encountered on most crowdsourcing platforms. One crowdsourcing platform that overcomes many of these limitations is Amazon Mechanical Turk (MTurk). MTurk has in recent years revolutionized behavioral research (Buhrmester, Kwang, & Gosling, 2011), and increasingly is being used as a participant recruitment tool across a wide spectrum of disciplines in the social sciences. MTurk is an online platform where researchers can post studies, instantaneously making them available to thousands of participants around the world. Hundreds of participants will typically have completed a study just hours after it is launched, drastically increasing the speed of the data acquisition process relative to traditional methods. Data acquired on MTurk have been found to be valid across numerous tasks and countries (Litman, Robinson, & Rosenzweig, 2015; Shapiro, Chandler, & Mueller, 2013; Sprouse, 2011), despite the relatively low price that participants are paid for completing tasks, making the MTurk platform a fast method of acquiring cheap and reliable data (Buhrmester et al., 2011). Despite the benefits that MTurk offers social science researchers, it also has its limitations. MTurk was not created with behavioral research in mind, and many features that are fundamental for the research process are not readily available through the MTurk graphical user interface (GUI). To overcome the current limitations of MTurk, multiple tools have been created to enable researches to accomplish specific tasks (Gureckis et al., 2015; Peer, Paolacci, Chandler, & Mueller, 2012; http://gideongoldin.github.io/TurkGate/). For example, Peer et al. (2012) created a Qualtrics tool through which it is possible to exclude participants from a study on the basis of participation in previous studies. However, currently no single, integrated platform provides researchers with a comprehensive set of research-specific flexible tools that are delivered through a simple GUI. For this reason, we developed the TurkPrime Data Acquisition Platform for the Social Sciences (TurkPrime). TurkPrime is a website on which users can create MTurk studies (HITs) and through which MTurk HITs can be controlled with substantially greater flexibility than the MTurk platform currently allows. TurkPrime (see Fig. 1) is a website that utilizes the MTurk application programming interface (API), as well as other programmatic tools, to provide MTurk requesters with a GUI environment with a high level of control over HITs. The TurkPrime GUI environment provides improved functionality over MTurk in six general areas: control over who participates in the study, flexible control over running HITs, more flexible communication and payment mechanisms, tools for longitudinal and panel studies, tools to increase sample representativeness, and enhanced study flow indicators.Fig. 1Screenshot of the TurkPrime home page In the present article, we show how the TurkPrime environment allows users to overcome the current limitations of the MTurk GUI. Because detailed tutorials including videos for using TurkPrime are available on the TurkPrime website, in this article we focus on discussing the methodological advantages that TurkPrime offers, and also provide a conceptual overview of the TurkPrime platform. MECHANICAL TURK BASICS Because of MTurk’s popularity, a number of MTurk guides are currently available (Berinsky, Huber, & Lenz, 2012; Chandler & Mueller, 2013; Paolacci, Chandler, & Ipeirotis, 2010; Mason & Suri, 2012). Here, we will not provide a comprehensive overview of MTurk, but only introduce those terms and concepts that are relevant for the TurkPrime features that are discussed throughout this article. Workers, requesters, and HITs The MTurk platform was created as a worldwide microtask marketplace. MTurk facilitates monetary transactions between people who need various jobs done (requesters) and people who are interested in performing those jobs in return for payment (workers). A task put up by a requester on MTurk is called a human intelligence task (HIT). Requesters can create HITs on the MTurk platform or use MTurk to announce an external HIT. For example, MTurk provides a platform for creating survey questions. A requester may use that built-in MTurk functionality, or instead can create a survey on an external platform such as Qualtrics and post a link to that survey as part of their HIT. We refer to HITs that are completed via an external platform as external HITs, and tasks that utilize the MTurk platform as internal HITs. MTurk makes HITs visible to workers by listing them in the All HITs window, where workers can select a HIT from the numerous ones available. Workers can organize HITs in various ways. By default, the All HITs window organizes HITs by their creation date, such that the newest HITs are presented at the top of the list. HITs can also be sorted by price, title, and HIT length. Worker ID The MTurk platform allows for two-way interactions between workers and