Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research.

Crump MJ, McDonnell JV, Gureckis TM. · 2013

Read Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. by Crump MJ, McDonnell JV, Gureckis TM. (2013) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc3596391.txt.

Introduction

Matthew Crump, John McDonnell, and Todd Gureckis tested Amazon's Mechanical Turk as a laboratory for psychology in PLoS ONE in 2013. The service pays anonymous online workers small sums for tasks, tempting researchers who struggle to recruit enough subjects on campus. Browser-based testing, though, gives up control over the room, the timing gear, and the subject's attention, all vital for reaction-time experiments that live on milliseconds.

Eight classic tasks, from the Stroop and flanker tests to attentional blink and category learning, were rerun online. Most effects reproduced well enough to validate the approach. A few diverged from laboratory results, exposing where home computers, distractions, or misunderstood instructions bite. The authors lay out lessons from each failure mode, covering timing checks, screening, and pay. Their verdict is measured: online crowds can serve real experiments when researchers design around the known traps.

Catalog record

CREATORCrump MJ, McDonnell JV, Gureckis TM.
YEAR2013
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC3596391

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research By Matthew J. C. Crump, John V. McDonnell, Todd M. Gureckis, Sam Gilbert Published in: PLoS ONE DOI: 10.1371/journal.pone.0057410 ABSTRACT Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers. INTRODUCTION One challenging aspect of experimental psychology research is the constant struggle for data. Typically, researchers depend on university undergraduates who participate in studies in exchange for experience, course credit, or money. Research progress depends on the ebb and flow of the semester. As a result, it can take weeks, months, or even years to conduct a large behavioral study. This issue is even more salient for researchers at smaller universities. One appealing solution is to collect behavioral data over the Internet. In theory, online experimentation would allow researchers to access to a large and diverse pool of potential subjects worldwide, using automated replicable techniques free of unintended experimenter effects. However, the main obstacle to conducting Internet-based research is finding people who are willing to participate and compensating them. Recently, a number of online crowdsourcing services have been developed which connect individuals willing to perform online tasks with other individuals willing to pay for work to be done. Perhaps the most popular system is Amazon's Mechanical Turk (AMT). AMT is useful for behavioral researchers because it handles recruitment and payment in a fairly automatic way. Most importantly, there are a large number of people who use AMT making it a great way to advertise and distribute studies (over 100,000 active users in 2007 [1]). There are a number of recent summaries about using AMT for research [2]. In addition, the service has been validated as a tool for conducting survey research [3], [4], one-shot decision-making research [5], [6], collective behavior experiments [7], [8], for norming stimuli, and conducting behavioral linguistics experiments [9], [10]. However, less is known about the viability of conducting behavioral experiments typical of those used in cognitive science and cognitive psychology. Such studies are unique in that they typically involve multi-trial designs, sustained attention on the part of participants, millisecond timing for response recording and stimulus presentation, and relatively complex instructions. These features present two key challenges for online data collection. First, there are technical challenges in programming web-based experiment protocols and then ensuring the browser systems of the participant and experimenter support the same features. Second, experiments where memory and timing are important are likely more sensitive to incidental aspects of the testing environment that are difficult to control online (e.g., presence of distractions, problems with display, pausing for long periods in the middle of the task, and misreading or misunderstanding of instructions). The aim of the present paper is to validate AMT as a tool for behavioral cognitive research, with a specific focus on complex multi-trial designs. We focus on AMT simply because it is the most popular system currently available and the one most researchers would likely consider. If the data obtained from AMT can replicate classic findings in the field with reasonable fidelity, it will validate the potential of the service for use in cognitive behavioral research. In this sense our study joins a number of recent articles exploring the relationship between data collected online and in the lab [11], [3]. However, unlike existing work, in the present study we focus on qualitative replication of theoretically significant findings rather than on comparisons of the performance variance or mean performance of lab and online participants (e.g., [11]). The standard of qualitative replication (i.e., the ability to detect reliable differences as the result of an established and widely accepted experimental manipulation) is likely the one of most interest to researchers and has the greatest importance for the field. In the present study we attempted to replicate influential findings from several classic cognitive phenomena from the attention, performance, and learning literatures. The replication studies were chosen to represent and validate a broad range of multi-trial designs that require millisecond control over response collection and stimulus presentation, as well as designs that require complex decision making and where task instructions are critically important. To foreshadow, our results suggest that data collected online using AMT closely resemble data collected in the lab under more controlled situations. However, for certain types of experiments the alignment between laboratory results and online results showed greater disparity. In the conclusion of the paper we suggest some general lessons we obtained which may help inform other researchers considering using online data in their work. AMT Basics There are a number of in-depth overviews of using AMT to conduct behavioral experiments [4], [2], [6]. In brief, the service allows individuals (known as requesters) to post human intelligence tasks (HITs) that other individuals (known as workers) can complete for small sums of money. Each HIT is a small unit of work that is typically completed in the worker's web browser. HITs can be composed of assignments which allow multiple workers to complete the same HIT. Common HITs include providing keywords for the objects in an image or giving feedback about a website. These tasks typically require some form of human intelligence, but can be completed by almost anyone in a few minutes. In the present case, we consider a HIT to be a request to participate in an entire cognitive experiment. Amazon provides web-based, point-and-click tools for creating several different kinds of HITs; however none of these tools are suitable for the control and flexibility necessary for conducting complex behavioral experiments. As an alternative, Amazon provides a way for tasks to be completed on an external webserver. As a result, requesters (in this case, psychologists) need only program and host an external website that is capable of running the desired experiment. Any task that can be programmed using standard web browser technology (e.g., HTML with JavaScript or Flash) can be used on AMT. Once the external website HIT has been designed, it must be set up to interface with Amazon's service so that workers who accept and complete the task can be paid (see [2] for details). Once a HIT is posted to the service it will be available for workers to complete. Restrictions can be set

Read the complete text