MUSHRA–1S: A scalable and sensitive test approach for evaluating top-tier speech processing systems
Laura Lechler, Ivana Balić · 2025
Read MUSHRA–1S: A scalable and sensitive test approach for evaluating top-tier speech processing systems by Laura Lechler, Ivana Balić (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19219.txt.
Introduction
Rating the quality of top speech systems is tricky: the most sensitive listening test does not scale, and the scalable one goes blind at high quality. Laura Lechler and Ivana Balic propose MUSHRA-1S, a variant of the sensitive MUSHRA test that judges each system on its own, scored against a fixed reference and anchor. Standard MUSHRA needs all systems side by side, which limits how many listeners and systems a study can handle.
In their experiments, the new variant tracked standard MUSHRA scores more closely than the scalable alternative, including in the high-quality range where that alternative saturates. Fixing the context also reduced a bias that comes from listeners stretching their scales to fit the set. The method still pinpoints specific artifacts in a system's output. The authors offer it as a robust way to benchmark the best speech systems at scale.
Catalog record
| CREATOR | Laura Lechler, Ivana Balić |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-19219
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MUSHRA–1S: A scalable and sensitive test approach for evaluating top-tier speech processing systems Laura Lechler, Ivana Balić Abstract Evaluating state-of-the-art speech systems necessitates scalable and sensitive evaluation methods to detect subtle but unacceptable artifacts. Standard MUSHRA is sensitive but lacks scalability, while ACR scales well but loses sensitivity and saturates at a high quality. To address this, we introduce MUSHRA–1S, a single-stimulus variant that rates one system at a time against a fixed anchor and reference. Across our experiments, MUSHRA–1S matches standard MUSHRA more closely than ACR, including in the high-quality regime, where ACR saturates. MUSHRA–1S also effectively identifies specific deviations and reduces range-equalizing biases by fixing context. Overall, MUSHRA–1S combines MUSHRA–level sensitivity with ACR–like scalability, making it a robust and scalable solution for benchmarking top-tier speech processing systems. † † address: Collaboration AI, Cisco Systems, United Kingdom 1 Introduction Recent advances in deep learning for speech processing have enabled high speech quality with fewer resources. For example, audio codecs now perform very well at low bitrates, e.g., [ 1 , 2 ] , and speech enhancement models can reconstruct speech from very low SNR cases [ 3 ] . Generative neural networks have recently succeeded in speech coding, enhancement, bandwidth expansion, and text-to-speech synthesis. However, these can generate high-quality human-like speech that may differ from the original input in unacceptable ways, such as changes in voice characteristics, phoneme errors, or inappropriate prosody. Hence, evaluation should be reference-based, scalable, and highly sensitive so it can quantify small quality differences as well as reference-relative errors at near-transparent quality. Standard MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA) [ 4 ] framework is effective and sensitive, including when conducted via crowdsourcing [ 5 ] , but it does not scale to many systems. ITU-R BS.1534-3 recommends at most 12 stimuli (ideally 7) for lab-based tests [ 4 ] , and in crowdsourcing we find 5–6 is optimal for avoiding listener fatigue [ 5 ] . As a result, MUSHRA is impractical for benchmarking many systems or tracking long-term performance. Absolute category ratings (ACR) [ 6 ] , which yield mean opinion scores (MOS), are often preferred for their scalability, cost-effectiveness, and simplicity. However, four key limitations must be considered: 1. Absolute category ratings are not truly absolute. Studies show range-equalizing biases [ 7 , 8 , 9 ] , making careful condition selection crucial. 2. The 5-point scale limits sensitivity. Its coarseness and lack of reference prevent raters from penalizing small degradations [ 10 , 11 , 12 ] . 3. Scale extremes saturate easily. The polar labels “bad” and “excellent” cover a wider perceptual range than intermediate levels [ 11 ] . 4. High-quality but incorrect (hallucinated) outputs go undetected without a reference. Reference files are needed to catch high-quality generative errors [ 5 ] . 5. Listener expectations shift over time. What is rated “excellent” changes with exposure to state-of-the-art systems [ 8 ] , particularly for synthetic speech. Recommendations to mitigate these issues include adding implicit anchors and references [ 12 ] , careful listener training, extended continuous scales [ 11 ] , or a staggered approach combining ACR and MUSHRA for top systems [ 12 ] . To balance scalability and reference-based evaluation, Müller et al. [ 13 ] recommend degradation category ratings (DCR) for neural audio codecs, noting lower sensitivity than MUSHRA and scale saturation at high quality. The DCR scale may also be less adaptable for specific tasks. Given these findings, we see a need for a scalable, stable, and sensitive test for high-performing speech models. We therefore propose MUSHRA–1S, a MUSHRA-like test evaluating one system at a time with a stable anchor and reference. This approach compromises between MUSHRA’s sensitivity and ACR’s scalability. MUSHRA scores may also map to the ACR-MOS scale [ 10 ] , complementing coarse ratings and aiding MOS predictor training. MUSHRA–1S could also reduce range-equalizing biases present in standard MUSHRA, as results become less dependent on the performance of other systems included in the test. While the direct comparison is lost, the range-equalizing bias may be limited to the constant reference and anchor context. 2 Experiment Setup 2.1 General test procedure The data set comprises 100 originally full-band (44.1--48 kHz) high-quality clean speech files and was designed to ensure gender balance, accent diversity, and inclusion of both adult and child speech. The set served as the clean blind test set in the Low-Resource Audio Codec (LRAC) Challenge 2025 1 1 1 https://crowdsourcing.cisco.com/lrac-challenge/2025/ and is publicly available 2 2 2 https://github.com/cisco/multilingual-speech-testing/tree/main/LRAC-2025-test-data/blind-test-set/track-1/clean/ . All tests were conducted on Prolific, with compensation based on an hourly rate of $8.50. Participants were recruited using the following screener criteria: first language English, no hearing difficulties, no cochlear implant, 98%+ approval rate, and 110+ studies approved. Each condition was first tested as a MUSHRA–1S, followed by a standard MUSHRA test with all conditions, and then as separate ACR tests. MUSHRA and MUSHRA–1S used the survey setup tools and procedure presented in [ 5 ] . In line with the Preliminary Study 2.2 , we collected a minimum of 6 responses per file. ACR studies followed P.808, with two gold questions (one low-quality, one high-quality) and two catch trials randomly added to 20 rating questions. Catch trials were instructional checks; gold questions included an Opus 6 kbps sample and a 24 kHz reference. Eight responses per file were collected, the minimum recommended in [ 14 ] . For normalization, mean scores for each condition in the standard MUSHRA test established anchor and reference targets for normalizing MUSHRA–1S per-file mean scores. ACR scores were not normalized directly; instead, ACR and MUSHRA-like MOS were plotted on separate y-axes and approximately aligned using adjusted y-axis limits. In long-term monitoring use cases, the scores can be adjusted to constant values for the anchor and reference, e.g., values obtained in the initial round of testing or 0–100 normalization. As MUSHRA tests are standardized and widely accepted as accurate and reliable, we consider the standard MUSHRA tests the ground truth test that MUSHRA–1S and ACR are compared to. 2.2 Preliminary Study: Number of responses per file In a preliminary study, we tested how many responses per file are required to produce a stable mean score with acceptable confidence intervals across the 100-file test set. The codec tested in this experiment is an experimental internal codec at 1 kbps and shows a wide range of output qualities. We assume that many other codecs will produce more homogeneous outputs and that the number established on this more variable test case will ensure sufficient accuracy on other codecs in general. Approximately 30 responses were collected for each of the 100 files in the test set. 2.3 Experiment 1: Medium-to-high quality codecs To evaluate the general performance and consistency of the MUSHRA–1S approach with respect to a standardized MUSHRA test and absolute category ratings (ACR), Exp. 1a compared three different flavors of the well-known codec Opus [ 15 ] (version 1.5.2) at 9 kbps: Opus (basic complexity) was compared with the enhancement options LACE [ 16 ] and NoLACE [ 17 ] . Opus at 6 kbps was chosen as the anchor condition. Clean wide-band (WB) files at 16 kHz sampling rate (Fs) served as reference files. We