Lessons Learned from the URGENT 2024 Speech Enhancement Challenge
Wangyou, Kohei, Samuele, Robin, Chenda, Zhaoheng, Anurag, Marvin, Wei, Yihui, Shinji, Tim, Yanmin · 2025
Read Lessons Learned from the URGENT 2024 Speech Enhancement Challenge by Wangyou, Kohei, Samuele, Robin, Chenda, Zhaoheng, Anurag, Marvin, Wei, Yihui, Shinji, Tim, Yanmin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01611.txt.
Introduction
Wangyou Zhang and co-authors draw in 2025 lessons from the URGENT 2024 speech enhancement challenge. The contest pushed for tools that work across many voices, noises, and rooms. Two understudied fronts get the deep look here: data cleaning and grading metrics.
Declared and true audio bandwidth often clash, and even prized corpora carry label noise. The hardest scenes, like speech overlap and strong echo, still defeat top systems. No single metric tracks human judgment well, so mixed metric sets are urged. The authors hope the lessons reshape how speech cleaning pipelines get built.
Catalog record
| CREATOR | Wangyou, Kohei, Samuele, Robin, Chenda, Zhaoheng, Anurag, Marvin, Wei, Yihui, Shinji, Tim, Yanmin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01611
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Lessons Learned from the URGENT 2024 Speech Enhancement Challenge Wangyou, Kohei, Samuele, Robin, Chenda, Zhaoheng, Anurag, Marvin, Wei, Yihui, Shinji, Tim, Yanmin Abstract The URGENT 2024 Challenge aims to foster speech enhancement (SE) techniques with great universality, robustness, and generalizability, featuring a broader task definition, large-scale multi-domain data, and comprehensive evaluation metrics. Nourished by the challenge outcomes, this paper presents an in-depth analysis of two key, yet understudied, issues in SE system development: data cleaning and evaluation metrics. We highlight several overlooked problems in traditional SE pipelines: (1) mismatches between declared and effective audio bandwidths, along with label noise even in various “high-quality” speech corpora; (2) lack of both effective SE systems to conquer the hardest conditions (e.g., speech overlap, strong noise / reverberation) and reliable measure of speech sample difficulty; (3) importance of combining multifaceted metrics for a comprehensive evaluation correlating well with human judgment. We hope that this endeavor can inspire improved SE pipeline designs in the future. Zhang Saijo Cornell Scheibler Li Ni Kumar Sach Wang Fu Watanabe Fingscheidt Qian Shanghai Jiao Tong University, China 2 Waseda University, Japan 3 Carnegie Mellon University, USA 4 Google DeepMind, Japan 5 Meta, USA 6 Technische Universität BraunschweigGermany keywords speech enhancement, universality, robustness, generalizability † † email: [email protected],[email protected] 1 Introduction The URGENT 2024 Challenge [ 1 ] is a newly launched competition that aims to assess and advance the universality, robustness, and generalizability of speech enhancement (SE) techniques. It features a broader SE task definition, large-scale and diverse source data, and extensive evaluation metrics, complementing existing SE challenges [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . Specifically, this challenge encourages research towards a single universal SE system that can handle speech signals with various sampling frequencies (SF) corrupted by diverse distortions, including additive noise, reverberation, clipping, and bandwidth limitation. To comprehensively assess SE performance, this challenge adopts as many as 13 evaluation metrics. The ranking of different submissions is then determined using a dedicated algorithm inspired by the Friedman test [ 10 ] that balances the metric performances from different perspectives [ 1 ] . The challenge consists of three phases, i.e., validation , non-blind test , and blind test phases. Each phase provides a corresponding dataset with 1000 samples for leaderboard evaluation. In the first two phases, all samples are simulated using unseen speech and noise data, and the corresponding ground-truth clean speech signals (labels) are provided after each phase ends. This allows for fast system development as all evaluations can be done automatically using objective metrics. In the final phase, only half of the samples are simulated, while the other half are collected from real-world scenarios, allowing for a more realistic and comprehensive evaluation. The final ranking is fully determined by the results obtained in the final phase 1 1 1 https://urgent-challenge.com/competitions/5 . . Through the above procedure, the challenge has received submissions from 21 participating teams in the final blind test phase. Different from [ 1 ] that only provides a basic introduction of the challenge, this paper aims to share the learnings from the challenge design choices as well as the final competition results. Specifically, the goal of this paper is to uncover overlooked issues and highlight potential improvements in the design of data and evaluation metrics, two critical components of system development. Below, we summarize our key findings. 1) A mismatch between declared and effective audio bandwidths commonly exist in both public audio corpora and in-the-wild data. For example, ~25% LibriTTS [ 11 ] , ~100% DNS5 LibriVox [ 3 ] , and ~100% CommonVoice 11.0 [ 12 ] English speech data have mismatched audio bandwidths. Furthermore, the label noisiness issue is also surprisingly common, even in supposedly clean corpora such as WSJ [ 13 , 14 ] , VCTK [ 15 ] and LibriTTS. Both issues can cause unexpected model behaviors and require extra care in data preparation. 2) Overlapped speech, strong wideband / instantaneous noise, and high reverberation remain the hardest challenges in SE tasks. Moreover, there is no good objective measure of the sample difficulty for the SE task, and conventional metrics such as signal-to-noise ratio (SNR) are often misleading. 3) Interestingly, some newly proposed non-intrusive metrics such as UTMOS [ 16 ] and VQScore [ 17 ] show better perceptual correlations than traditional ones (e.g., DNSMOS [ 18 ] ), despite that they may not be designed directly for the SE task. 4) It is beneficial to combine various metrics for a multifaceted evaluation (cf. Section 3.1 ) of the SE performance, which leads to high correlations with human judgments. On the other hand, using solely a single category of metrics, especially non-intrusive metrics, for SE performance assessment can be unreliable or even misleading. Figure 1: Sampling frequency distribution of speech and noise data in the training set. The original noise data are all in 48 kHz. Figure 2: Histogram of estimated SNRs of the “clean” speech labels in each dataset based on the WADA algorithm [ 19 ] . 2 Data Analysis 2.1 Training and validation data Both training and validation datasets of the URGENT 2024 Challenge are fully simulated based on 5 public speech corpora (DNS5 LibriVox speech [ 3 ] , LibriTTS, CommonVoice 11.0 English portion [ 12 ] , VCTK, and WSJ), 2 noise corpora (Audioset+FreeSound noise in DNS5 challenge [ 3 ] , and WHAM! noise recordings [ 20 ] ), and simulated room impulses (RIRs) from the DNS5 challenge data 2 2 2 Readers can refer to [ 1 ] for more details. . The curated dataset includes ~1300 hours of source speech data and ~250 hours of noise data. When revisiting the preprocessing of training and validation data (i.e., filtering and resampling), we found two major obstacles that may lead to compromised data quality. While Zhang et al. [ 1 ] managed to address the first one, the other remains an open question that requires more investigations in future work. Below we discuss each problem in detail. 1) Effective bandwidth vs. actual sampling frequency : We found that various public speech corpora contain “bad” samples with a lower effective bandwidth than the SF’s Nyquist frequency, suggesting that these samples have been resampled from a lower SF. For example, some LibriTTS samples only contain frequency components up to 4 kHz while their SF is always 24 kHz, leading to wasted storage and potential misuses. This is particularly important in our challenge as precise information about the effective bandwidth is pivotal for unifying bandwidth extension (BWE) and other SE subtasks. It is also crucial in speech generation tasks such as text-to-speech (TTS) and voice conversion (VC), where such “bad” samples may mislead the learning procedure. To address this issue, the challenge adopted an energy-based algorithm [ 21 ] to estimate the effective bandwidth of all audio samples. In Fig. 1 , we provide a breakdown of the SF distributions of different training corpora (speech and noise) before and after bandwidth re-estimation. Surprisingly, except for WSJ, VCTK, and WHAM!, other commonly-used corpora, i.e., DNS5, LibriTTS, and CommonVoice all suffer from the above issue, with ~100%, ~25%, and ~100% samples having mismatched audio bandwidths, respectively. This also calls