TS-URGENet: A Three-stage Universal Robust and Generalizable Speech Enhancement Network
Xiaobin, Dahan, Qinwen, Yushi, Yuxiang, Jing · 2025
Read TS-URGENet: A Three-stage Universal Robust and Generalizable Speech Enhancement Network by Xiaobin, Dahan, Qinwen, Yushi, Yuxiang, Jing (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18533.txt.
Introduction
Xiaobin Rong and co-authors present a three-stage network that cleans up damaged speech, in work from 2025. Calls and recordings suffer many flaws at once: dropouts, noise, echo, clipping, and codec damage. A universal cleaner must handle all formats and flaws together.
Stage one fills lost chunks so the signal has no gaps. Stage two strips noise, echo, and clipping to restore clarity. Stage three repairs bandwidth limits, codec marks, and leftover dropout damage. The staged split lets each part master one kind of harm. The system placed second in its track at a major 2025 speech challenge. It aims at real calls, not just lab clips.
Catalog record
| CREATOR | Xiaobin, Dahan, Qinwen, Yushi, Yuxiang, Jing |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18533
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TS-URGENet: A Three-stage Universal Robust and Generalizable Speech Enhancement Network Xiaobin, Dahan, Qinwen, Yushi, Yuxiang, Jing Abstract Universal speech enhancement aims to handle input speech with different distortions and input formats. To tackle this challenge, we present TS-URGENet, a T hree- S tage U niversal, R obust, and G eneralizable speech E nhancement Net work. To address various distortions, the proposed system employs a novel three-stage architecture consisting of a filling stage, a separation stage, and a restoration stage. The filling stage mitigates packet loss by preliminarily filling lost regions under noise interference, ensuring signal continuity. The separation stage suppresses noise, reverberation, and clipping distortion to improve speech clarity. Finally, the restoration stage compensates for bandwidth limitation, codec artifacts, and residual packet loss distortion, refining the overall speech quality. Our proposed TS-URGENet achieves outstanding performance in the Interspeech 2025 URGENT Challenge, ranking 2nd in Track 1. Rong Wang Hu Wang Hu Lu Nanjing UniversityChina Horizon RoboticsChina keywords universal speech enhancement, three-stage, universality, robustness, generalizability † † email: {xiaobin.rong, dahan.wang, qinwen.hu, yushi.wang}@smail.nju.edu.cn, [email protected], [email protected] 1 Introduction Conventional speech enhancement (SE) primarily focuses on eliminating undesired signals such as noise and reverberation. In contrast, a more general SE task proposed in the Speech Signal Improvement (SSI) Challenges aims to improve speech quality by considering a broader range of factors, including coloration, discontinuity, and loudness, in addition to noise and reverberation [ 1 , 2 ] . Recently, the concept of universal speech enhancement (USE) has been proposed. Compared to general SE, USE considers using a single model to handle diverse input formats, including variable channels, lengths, and sampling frequencies, in addition to various distortions [ 3 ] . In the Interspeech 2025 URGENT Challenge 1 1 1 https://urgent-challenge.github.io/urgent2025/ , a concrete task for USE is proposed: to build a single model capable of addressing seven types of distortions while accommodating sampling frequency flexibility. The distortions include additive noise, reverberation, clipping, bandwidth limitation, codec artifacts, packet loss, and wind noise. The supported sampling frequencies include 8, 16, 22.05, 24, 32, 44.1, and \SI 48kHz. To build a USE system flexible with different sampling frequencies, one straightforward solution is to always upsample the speech to \SI 48kHz as pre-processing, allowing the model to process data with a fixed sampling frequency. The output is then downsampled to the original sampling frequency as the final enhanced speech. Alternatively, sampling-frequency-independent (SFI) approaches can directly adapt to various sampling frequencies [ 4 ] . Extensive research has been conducted on addressing individual distortion types separately [ 5 , 6 , 7 , 8 ] . However, recent advances have revealed that various distortion types can be effectively handled using a unified framework, which can be classified into two categories: single-stage and multi-stage methods. In single-stage approaches, all distortions are addressed simultaneously. For instance, VoiceFixer [ 9 ] employs ResUNet [ 10 ] to address additive noise, reverberation, clipping, and bandwidth limitation in the Mel domain, and utilizes a neural vocoder to synthesize a waveform. In [ 11 ] , a score-based diffusion model is proposed to tackle 55 different distortions at the same time. In multi-stage approaches, distortions are grouped into different stages based on their characteristics. In [ 12 ] , a pipeline cascades a time-domain U-Net for clipping and packet loss with a time-frequency U-Net for codec distortions. In the winning solutions of the SSI Challenges, two-stage frameworks are widely employed, where the first stage addresses various distortions simultaneously, and the second stage eliminates residual noise and artifacts produced by the first stage [ 13 , 14 , 15 , 16 ] . The complex acoustic scenarios in the URGENT Challenge, where speech can be contaminated by strong noise, reverberation, and other distortions, make a single-stage framework highly challenging. This limitation can hinder the model’s generalization ability and ultimately degrade enhancement performance. On the other hand, the two-stage “restoration and enhancement” framework performs more effectively in high signal-to-noise ratio (SNR) scenarios. When applied to data with strong noise and reverberation, it may inadvertently generate noise instead of restoring speech, resulting in additional artifacts in the processed output. Considering the distinct characteristics of the seven targeted distortions, we propose a three-stage universal, robust, and generalizable speech enhancement network named TS-URGENet, which divides the complex enhancement into three stages: filling, separation, and restoration. Specifically, the filling stage fills the gaps caused by packet loss distortion; the separation stage simultaneously handles noise, reverberation, and clipping distortion; and the restoration stage addresses bandwidth limitation, codec artifacts, and residual packet loss distortion. Our contributions are as follows: • We propose a novel filling-separation-restoration framework for universal speech enhancement, to address various distortions with different input sampling frequencies. • We introduce an effective metric-aware fine-tuning loss function to directly and collaboratively optimize multiple target metrics. • Our proposed TS-URGENet achieves outstanding performance in the Interspeech 2025 URGENT Challenge, ranking 2nd in Track 1. 2 Method Figure 1: Overview of the TS-URGENet framework. As shown in Fig. 1 , our proposed framework is composed of three parts: filling, separation, and restoration stages. We first employ a small-scale TF-GridNet [ 17 ] (TF-GridNet-S) as the filling module to fill the gaps caused by packet loss distortion. A residual connection is added after the filling stage to minimize information loss. Subsequently, a larger-scale TF-GridNet (TF-GridNet-L) is applied as the separation module for denoising, dereverberation, and declipping. Finally, we sequentially integrate the channel-wise subband (CWS) [ 18 ] splitting operation, TF-GridNet, and the CWS merging operation to create CWS-TF-GridNet as the restoration module, which performs bandwidth extension (BWE), codec artifact reduction, and further packet loss concealment (PLC). As TF-GridNet inherently satisfies the sampling-frequency-independent (SFI) requirement, both the filling and separation stages are adaptive to various sampling frequencies. However, due to the constraints imposed by the CWS operation, CWS-TF-GridNet is restricted to processing signals at a fixed sampling frequency. To address this limitation, resampling modules are incorporated at both the input and output of the restoration stage. All the stages operate at the time-frequency (T-F) domain, the short-time Fourier transform (STFT) and inverse STFT (iSTFT) are integrated at the input and output of the network in each stage, while omitted in Fig. 1 for simplicity. 2.1 Filling stage We found that speech discontinuity caused by packet loss can degrade the performance of the separation module. Therefore, we first incorporate a filling stage, which is expected to restore the components of the lost regions while maintaining the integrity of the other regions. Although in practice, we found that the model often fills the gaps with noise rather than valid speech, this approach helps improve the continuity of signals, which in turn facilitates the subsequent separation stage. We use a GAN-based approach to perform the filling. TF-GridNet-S is employed as the generator. A residual connection is added afterward to minimize information loss. As for the discriminators, the multi-period discriminator (MPD) and the multi-resolution discriminator (MRD) used in BigVGAN [ 19 ] are adopted together. We employ the same training loss as in Gesper [ 13 ] , consisting of reconstruction, adversarial, and feature-matching terms, with respective weights of 20, 1, and 1. 2.2 Separation stage Inspired by [ 5 ] , we group clipping, noise, reverberation, and wind noise