Towards Fair Pay and Equal Work: Imposing View Time Limits in Crowdsourced Image Classification

Gordon Lim, Stefan Larson, Yu Huang, Kevin Leach · 2024

Read Towards Fair Pay and Equal Work: Imposing View Time Limits in Crowdsourced Image Classification by Gordon Lim, Stefan Larson, Yu Huang, Kevin Leach (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00260.txt.

Introduction

Paying crowd workers a flat rate sounds fair until fast and slow workers turn the same rate into wildly different hourly wages. Gordon Lim and co-authors test a simple lever in a 2024 human study: a maximum view time for each image in a classification task. Time limits pin down how long a task can take, capping both overpayment and underpayment.

Performance loss from the cap shrank as the limit grew more generous. Even so, some images genuinely need longer looks. A consensus algorithm absorbed that problem, preserving data quality and flagging the hard images. Workers kept a steady pace throughout, and their survey answers showed they preferred shorter limits. The authors recommend time limits as a practical route to fairer, more equal crowd work.

Catalog record

CREATORGordon Lim, Stefan Larson, Yu Huang, Kevin Leach
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Fair Pay and Equal Work: Imposing View Time Limits in Crowdsourced Image Classification Gordon Lim, Stefan Larson, Yu Huang, Kevin Leach Abstract . Crowdsourcing is a common approach to rapidly annotate large volumes of data in machine learning applications. Typically, crowd workers are compensated with a flat rate based on an estimated completion time to meet a target hourly wage. Unfortunately, prior work has shown that variability in completion times among crowd workers led to overpayment by 168% in one case, and underpayment by 16% in another. However, by setting a time limit for task completion, it is possible to manage the risk of overpaying or underpaying while still facilitating flat rate payments. In this paper, we present an analysis of the impact of a time limit on crowd worker performance and satisfaction. We conducted a human study with a maximum view time for a crowdsourced image classification task. We find that the impact on overall crowd worker performance diminishes as view time increases. Despite some images being challenging under time limits, a consensus algorithm remains effective at preserving data quality and filters images needing more time. Additionally, crowd workers’ consistent performance throughout the time-limited task indicates sustained effort, and their psychometric questionnaire scores show they prefer shorter limits. Based on our findings, we recommend implementing task time limits as a practical approach to making compensation more equitable and predictable. Our code and data are available at https://github.com/gordon-lim/sdogs-10h . DOI: XXXXXXX.XXXXXXX Conference: ; ; CCS: Information systems Crowdsourcing CCS: Human-centered computing User studies 1. Introduction Supervised deep learning methodologies demand large amounts of annotated data ( Wang et al., 2024 ; Zhai et al., 2022 ) . Crowdsourcing has emerged as a popular method of data annotation due to its cost-efficiency ( Mathew et al., 2021 ; Anjum et al., 2021 ; Deng et al., 2009 ) . The flat rate compensation model is widely used for its convenience, being the default option on platforms like Amazon Mechanical Turk, and due to the tradition of compensating research participation based on output ( Salminen et al., 2023 ) . However, as crowdsourcing can be a full time job for some workers, they need a minimum wage to afford a decent standard of living ( Salminen et al., 2023 ; Posch et al., 2022 ) . As such, a recent movement towards giving crowd workers a fair pay has called on researchers to consider estimated hourly wages for completing their tasks ( Shmueli et al., 2021 ; Salminen et al., 2023 ) . For instance, venues like NeurIPS 1 1 1 https://nips.cc/public/EthicsGuideline and ACL 2 2 2 https://aclrollingreview.org/responsibleNLPresearch ask that researchers using crowdsourced labor disclose estimated hourly wages in their papers. A common way to estimate completion times is by conducting a pilot study and measuring an average completion time ( Gaikwad et al., 2015 ; Cheng et al., 2015 ) . However, the variability in individual completion times means that there are cases of overcompensation and undercompensation ( Salminen et al., 2023 ; Whiting et al., 2019 ; Schlagwein et al., 2019 ) . For example, Salminen et al. (2023) show that in one study, which paid a flat rate to honor a local minimum wage using average estimated time during a pilot study, the requesters paid 168% more than they would have if they had used the actual average time taken to complete the tasks. In another case, they paid 16% less than the intended minimum hourly wage. Consequently, there is a need to develop more equitable compensation schemes. Figure 1. Experimental setup of view time limit. In this paper, we investigate 100/1000/2500ms time limits to examine their impact on data quality and worker experience. Salminen et al. (2023) recommends that, instead of a flat rate, workers should be paid based on the time they spend on tasks. Whiting et al. (2019) developed an algorithm that uses self-reported estimates from crowd workers to automatically grant post-hoc bonuses to those falling below minimum wage. However, these time-adjusted models implicitly trust crowd workers, who are incentivised to intentionally inflate their completion times. Consequently, these methods have not been widely adopted, and instead the flat rate compenstation scheme remains dominant due to its convenience and ease of budgeting. The crowdsourcing platform Prolific 3 3 3 https://prolific.com uses the median completion time of all submissions 4 4 4 https://researcher-help.prolific.com/hc/en-gb/articles/360019777180-How-do-I-resolve-underpaying-studies to recommend fairer wages ( Vodrahalli et al., 2022 ) . However, even with median-based estimates, there remains the issue of overpayment and underpayment at the individual level ( Salminen et al., 2023 ) . In light of this discussion, we propose imposing time limits on crowdsourcing tasks for the following reasons: (1) to help workers save time and manage expectations while ensuring they receive fair pay, (2) to assist requesters in developing a fair and consistent payment strategy that is easy to implement, and (3) to prevent overcompensation and save costs. In this paper, we focus on image classification crowdsourcing tasks to compare our findings with previous work that did not use time limits ( Peterson et al., 2019 ) . One challenge with proposing a time limit is to address cognitive costs rather than psychomotor costs ( Cheng et al., 2015 ) , ensuring we do not discredit workers who have already begun cognitive processing (i.e., studying the image to identify it). To address this, we set a time limit on how long crowd workers can view the image but do not restrict the time needed to physically submit their response (Figure 1 ). For this approach to be widely adopted, it must also have the following two desirable properties: (1) it does not negatively impact data quality, and (2) it does not negatively impact worker sentiment, as a less desirable task might deter workers or lead to expectations of higher compensation. In this paper, we seek to validate these properties and guide the establishment of view time limits as a practical approach for addressing ethical concerns surrounding fair compensation and transparency over task completion times. Figure 2. Training procedure. Participants are shown examples of each class. They proceed after seeing all images in all classes. Figure 3. Qualification and Test trials. Participants are shown a dog image and must select the best breed category. In the qualification stage, there is no time limit. In the time-limited test stage, the image disappears after 100ms, 1000ms, or 2500ms. Participants can revisit training images by clicking each category. A grey bar shows overall progress. 2. Related Work Compensation Models Many crowdsourcing compensation models have been explored. Salminen et al. (2023) provides three categories of these approaches. First, the flat-rate model , where all participants are paid the same for completing a task ( Liu et al., 2022 ; Mallya et al., 2022 ; Gong and Zhang, 2022 ; Lu et al., 2022 ) . Second, the time-adjusted model , where pay is based on the time spent ( Whiting et al., 2019 ) . Third, the quality-adjusted model , which tries to incentivize quality work ( d’Eon et al., 2019 ; Radanovic et al., 2016 ; Huang and Fu, 2013 ) . Despite the benefits of time- and quality-adjusted models, the flat-rate model remains

Read the complete text