Ensuring Reliable Participation in Subjective Video Quality Tests Across Platforms

Babak Naderi, Ross Cutler · 2025

Read Ensuring Reliable Participation in Subjective Video Quality Tests Across Platforms by Babak Naderi, Ross Cutler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20001.txt.

Introduction

Video quality is judged by human panels, and crowdsourcing those panels invites cheaters. Babak Naderi and Ross Cutler tackle unreliable participation in subjective video tests run outside the lab. Recent exploits abuse video metadata, and remote-desktop connections let workers fake their setups. Both distort the scores that streaming and call platforms trust.

The authors build detectors, both objective and subjective, for remote-desktop users. Two mainstream crowdsourcing platforms are compared for susceptibility and for how well mitigations work. Test design itself is part of the defense. Subjective scores steer codec and product decisions worth large sums. Keeping the panel honest across platforms is unglamorous, essential work. The study treats fraud detection as part of measurement science.

Catalog record

CREATORBabak Naderi, Ross Cutler
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-20001

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ensuring Reliable Participation in Subjective Video Quality Tests Across Platforms Babak Naderi, Ross Cutler Abstract Subjective video quality assessment (VQA) is the gold standard for measuring end-user experience across communication, streaming, and UGC pipelines. Beyond high-validity lab studies, crowdsourcing offers accurate, reliable, faster, and cheaper evaluation-but suffers from unreliable submissions by workers who ignore instructions or game rewards. Recent tests reveal sophisticated exploits of video metadata and rising use of remote-desktop (RD) connections, both of which bias results. We propose objective and subjective detectors for RD users and compare two mainstream crowdsourcing platforms on their susceptibility and mitigation under realistic test conditions and task designs. † † address: Microsoft Corporation, Redmond, USA 1 Introduction Crowdsourcing enables rapid, scalable annotation, easing data bottlenecks for large models. In multimedia subjective quality assessment, progress began with ITU-T P.808 for speech. For video, ITU-T P.910 [ 1 ] defines laboratory procedures, and an open-source crowdsourcing adaptation was validated in [ 2 ] with mechanisms to ensure scores’ validity and reliability. The P.910 standard includes methods such as Absolute Category Rating (ACR), Degradation Category Rating (DCR), and Comparison Category Rating (CCR). In ACR, participants view a single video and rate quality on a five-point scale from Excellent (5) to Bad (1) ; scores are aggregated per clip/condition as Mean Opinion Scores (MOS). DCR and CCR are double-stimulus methods showing reference and processed videos. In CCR, participants see both in randomized order and rate the second relative to the first on a seven-point scale, ranging from Much worse (-3) through About the same (0) and to Much better (+3) . Ratings are normalized to reflect processed vs. reference quality and aggregated per clip/condition as Comparison MOS (CMOS). CCR typically offers higher sensitivity than ACR [ 1 ] . Low-effort or dishonest responses in crowdsourcing threaten validity. A layered defense is recommended in literature: (i) ex-ante controls (qualifications, screening), (ii) in-task validation (gold questions, attention checks, repeats, time/focus gates), (iii) behavioral analytics (interaction traces, timing), and (iv) ex-post filtering/aggregation (outlier removal, reliability modeling) [ 3 , 4 , 5 ] . For subjective VQA, best practices include environment/device checks (adequate resolution, brightness), qualification tests (color vision, acuity), embedded references and consistency checks (gold stimuli with known answers, trapping stimuli), and monitoring playback completion and viewing time to deter “rate-without-consuming” behavior [ 6 , 7 , 8 , 9 ] . With strict screening and post-hoc cleansing, these yield reliable scores consistent with lab studies [ 7 , 2 ] . A major threat is location misrepresentation: workers use Virtual Private Networks (VPNs), Virtual Private Servers (VPSs), or Remote Desktops (RDs) to bypass geo-eligibility. Empirical studies find this common, making IP-based blocking insufficient. Countermeasures combine network signals (ASN/VPN lists, IP–GPS mismatches), cultural/linguistic checks, latency and human reaction-time measures, duplicate-GPS heuristics, and survey-platform server-side blocks [ 10 , 11 , 12 ] . To the best of our knowledge, RD detection primarily uses reaction-time testing [ 11 ] , which is confounded by network, user responsiveness, and input-device effects. 2 Methods This paper focuses on VQA using Comparison Category Rating (CCR) in crowdsourcing and two suspicious-pattern scenarios: (i) limits of “gold” stimuli—some workers pass controls yet rate inconsistently; (ii) effects of Remote Desktop (RD) use. Our first observations on Amazon Mechanical Turk (AMT), 14% of raters appeared to connect from non-target countries, and 10% showed multiple IPs per worker, indicating Virtual Private Network (VPN) or RD usage. VPNs mainly affect demographics, but RD is worse for video quality: RD protocols decode and re-encode videos, adding/removing artifacts that corrupt subjective results. 2.1 Dataset A synthetic dataset was created using 10 talking-head videos from the VCD open-source dataset [ 13 ] , representing user-generated content recorded with diverse devices, environments, participants, and lighting conditions. All source videos have very good quality (MOS = 4.65). The source videos were processed with the degradations listed in Table 1 , each applied at multiple intensity levels, to generate Processed Video Sequences (PVS). To analyze suspicious rating patterns, a diagnostic dataset was also constructed with three variations of gold stimuli in addition to the original set (v0). In v1, source and PVS clips were swapped to invert rating polarity. In v2, clips were identical to v0 but renamed with random identifiers. In v3, PVSs were re-encoded with a low Constant Rate Factor (CRF) in H.264 to ensure higher file size and bitrate than the source videos, without changing the perceptual quality. Table 1 : Distortions used in Synthetic Dataset applied on source videos. Number of conditions shows number of intensity levels used. Distortions N conditions Blurring 7 Scaling (down and up) - Bilinear 5 JPEG Compression 10 H.264 Quantization 6 Frame freezing 9 JPEG Compression x Scaling 5x5 Random Combinations + noise and color distortions 10 Overall 72 2.2 Gold stimuli We invited 26 workers who had exhibited suspicious behavior in previous studies to participate in a new test using the diagnostic set. Three participants failed at least one sample in all four variations, while fifteen passed versions v0–v2 but failed in v3, where file sizes were manipulated. This pattern indicates systematic behavior among this subset of workers. Accordingly, we recommend designing gold-standard stimuli such that metadata from the file does not correspond to the expected answer. A follow-up study was conducted in which two sets of new gold stimuli were included in each session: one used for data cleaning and one as a hidden metric to measure rating accuracy after data cleaning. Additionally, 40 clips were processed with H.264 at three CRF levels (0, 17, 28). In a new subjective test on AMT, the proposed gold questions rejected 38.8% of submissions. Among the accepted submissions, participants passed the hidden gold questions in 97.5% of cases. CMOS results for the three CRF levels, obtained with different gold question types, are reported in Table 2 . Using the proposed gold set, the results indicate that H.264 encoding up to CRF 17 is perceptually lossless, in line with theoretical expectations. Table 2 : Effect of different gold stimuli strategy on CMOS values for three CRF categories. Gold stimuli CMOS (95%CI) CRF=0 CRF=17 CRF=28 No gold stimuli 0.738 (0.077) -1.601 (0.126) -1.618 (0.092) Original gold set -0.016 (0.047) -1.915 (0.122) -2.008 (0.080) Proposed gold set 0.023 (0.085) -0.091 (0.124) -0.188 (0.900) 2.3 RD Participants It is generally recommended that applications disable unnecessary graphical effects in RD sessions. Native applications can check registry keys to determine whether they are running in a RD environment and adapt accordingly. Modern web browsers such as Edge and Chrome employ similar mechanisms and communicate this through corresponding CSS settings to webpages. Since the crowdsourcing session are delivered as a webpage, this functionality was used to automatically label sessions as either RD or non–RD. Preliminary tests showed that this approach successfully detected all RD connections to Windows PCs with default settings. To evaluate the performance of the detection method and the impact of RD usage on perceptual quality in subjective video quality tests, a CCR crowdsourcing experiment was conducted using the synthetic dataset and the P.910-Crowd framework [ 2 ] on the AMT platform. Data cleaning was performed following the procedure in [ 2 ] , resulting in an average of 18.1 valid votes

Read the complete text