Crowdsourcing MUSHRA Tests in the Age of Generative Speech Technologies: A Comparative Analysis of Subjective and Objective Testing Methods
Laura, Chamran, Ivana · 2025
Read Crowdsourcing MUSHRA Tests in the Age of Generative Speech Technologies: A Comparative Analysis of Subjective and Objective Testing Methods by Laura, Chamran, Ivana (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00950.txt.
Introduction
Laura Lechler and co-authors ask whether crowd workers can grade speech quality as well as trained experts. The standard test, MUSHRA, catches subtle audio flaws but needs experts in quiet labs. That cost slows codec development. Objective metrics fill the gap badly for generative speech models, which they tend to underrate.
The team adapts MUSHRA for non-expert crowds on two platforms. They check repeat reliability against expert data. The platforms bias results in their own ways. Six objective metrics get scored too, and the traditional ones shortchange generative codecs. The authors offer guidance for cheap, scalable listening tests that still track human ears when new speech tech is judged.
Catalog record
| CREATOR | Laura, Chamran, Ivana |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00950
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Crowdsourcing MUSHRA Tests in the Age of Generative Speech Technologies: A Comparative Analysis of Subjective and Objective Testing Methods Laura, Chamran, Ivana Abstract The MUSHRA framework is widely used for detecting subtle audio quality differences but traditionally relies on expert listeners in controlled environments, making it costly and impractical for model development. As a result, objective metrics are often used during development, with expert evaluations conducted later. While effective for traditional DSP codecs, these metrics often fail to reliably evaluate generative models. This paper proposes adaptations for conducting MUSHRA tests with non-expert, crowdsourced listeners, focusing on generative speech codecs. We validate our approach by comparing results from MTurk and Prolific crowdsourcing platforms with expert listener data, assessing test–retest reliability and alignment. Additionally, we evaluate six objective metrics, showing that traditional metrics undervalue generative models. Our findings reveal platform-specific biases and emphasize codec-aware metrics, offering guidance for scalable perceptual testing of speech codecs. Lechler Moradi Balic Cisco SystemsEurope keywords evaluation, MUSHRA, crowdsourcing, codec † † email: [email protected], [email protected], [email protected] 1 Introduction Subjective perceptual studies are the gold standard for evaluating human perception of audio quality and related research tasks. Conducting these studies requires significant time and cost, often relying on specialized labs or resorting to small-sample internal listening tests with expert listeners. As a result, evaluation using objective metrics (e.g., PESQ [ 1 ] , POLQA [ 2 , 3 ] , and NiSQA [ 4 , 5 ] ) is widely used. However, recent advancements in generative models have compromised their reliability, as many rely on ground-truth references. Generated outputs may meaningfully differ without perceptual degradation. Similarly, non-intrusive objective metrics often fail to correlate well with subjective ratings. Many subjective test methodologies have successfully been adapted to a crowdsourcing environment, ensuring scalability and cost-effectiveness, compared to internal or professional laboratory tests. Examples are the application of MOS (mean opinion scores) for speech quality to the crowdsourcing environment [ 6 ] , leading to the implementation of ITU-T recommendation P.808 [ 7 ] , and the open-sourced adaptation of the Diagnostic Rhyme Test in multiple languages to assess speech intelligibility via crowdsourcing [ 8 ] . The MUSHRA (multiple stimuli with hidden reference and anchor) framework of evaluating speech quality [ 9 ] is a well-established test tool with high sensitivity to even small quality differences of high-quality speech codecs and other signal processing applications. Studies investigating the effects of conducting MUSHRA tests with non-expert listeners demonstrate that relative rankings are largely consistent between the two participant groups [ 10 , 11 ] , and some efforts have been made to facilitate online MUSHRA tests with the option to deploy these tests on crowdsourcing platforms [ 11 , 12 , 13 ] . However, such studies note that absolute scores may differ between expert and non-expert listeners, and non-expert ratings appeared to be significantly above the expert ratings, indicating a ceiling effect for very high-quality conditions under test. It is hence still a wide-spread assumption that the MUSHRA test framework is only suitable for an expert-listener audience (e.g., [ 14 ] ). This stance may be valid in certain application scenarios, such as evaluations for official accreditation and certification, where absolute scores are required that can be compared between test laboratories. However, for other use cases, such as continuous model evaluation throughout the development cycle and benchmarking against competitors, reliable relative rankings and scalability are priorities. Due to the recent popularity of generative methods for speech coding, we revisit the question of whether crowdsourced MUSHRA tests can reliably evaluate such models. We propose adaptations to the MUSHRA protocol for use with non-expert listeners in a crowdsourced setting and release open-source tools for replication. Our approach enables faster and more scalable evaluations across diverse populations. This work makes the following contributions: (i) it presents the first, to our knowledge, dedicated crowdsourced MUSHRA evaluation design, focusing on generative speech codecs, highlighting their unique perceptual challenges; (ii) it offers a direct comparison of two crowdsourcing platforms—Prolific and MTurk—revealing that while both replicate expert rankings with high test–retest reliability, only Prolific closely aligns with expert scores in absolute terms, whereas MTurk shows a ceiling effect; (iii) it provides a codec-specific analysis of subjective–objective metric correlation, showing that traditional metrics like PESQ and POLQA underestimate DNN-based codecs, while newer metrics such as SCOREQ show higher consistency across architectures. These insights inform evaluation design, highlight the value of subjective testing in generative codec research, and motivate further investigations into newer metrics. 2 Crowdsourced MUSHRA testing Traditional MUSHRA testing has always faced difficulties and high costs. Setting up controlled listening environments, recruiting trained participants, and managing equipment can be expensive and time-consuming. Moreover, ensuring consistent and reliable results can be challenging due to variations in listener focus and fatigue. By leveraging the power of the crowd, we can solve most of these difficulties. In addition, crowdsourced MUSHRA offers cost-efficiency, faster data collection, and greater reliability through a larger sample size. The larger and more diverse pool of participants accessible through crowdsourcing may be more representative of customer opinions and may avoid over-emphasizing the opinions of domain experts. Due to the low cost and fast collection pace, tests can be conducted more frequently. Given the significant variability in response reliability among crowdworkers—stemming from factors like fatigue, impairments, carelessness, equipment differences, or malicious intent—rigorous participant screening is crucial. To address this, our tool enhances existing solutions (e.g., webMUSHRA [ 12 ] ) with two key features: real-time score screening to immediately discard unreliable responses based on deviation thresholds, and automatic test partitioning. This partitioning divides stimuli into smaller, configurable sub-tests for individual listeners, enabling evaluation of more conditions while mitigating fatigue and supporting diverse audio files. Our procedure builds upon and adapts established guidelines from ITU-T P.808 [ 15 ] , which outlines best practices for crowdsourced MOS testing, and ITU-R BS.1534-3 [ 9 ] , the standard for MUSHRA testing. We extend these recommendations to address practical challenges encountered when conducting MUSHRA tests with crowdsourced non-expert listeners. 2.1 Pre-screening In line with subjective test recommendations for MOS tests in crowdsourced environments [ 7 ] , we applied quality filters on both MTurk and Prolific. Participants were required to have a success rate above 97% and at least 100 completed tasks. Based on empirical observations, no further filters were used on MTurk, while on Prolific, we applied filters for first language (’English’), hearing difficulties (’no’), and cochlear implant (’no’). Depending on the platform, additional filters may be available; researchers should consider those relevant to their use case and report all filters used. Whether native-speaker competence is required may depend on the research question. A study on cross-language evaluations of codec artefacts [ 16 ] found no significant rating differences, though longer listening times were observed for foreign-language conditions in high-quality audio. However, native speakers are likely more able to judge intelligibility. 2.2 Qualification phase Our survey flow included a questionnaire obtaining self-reported responses on several aspects relevant to participation eligibility, a digits-in-noise hearing test, and a training session similar to the actual rating task. The questionnaire is based on ITU recommendation P.800.1 [ 17 ] and assesses the listening device used, the participant’s tiredness level, the last participation in an