ParaNoise-SV: Integrated Approach for Noise-Robust Speaker Verification with Parallel Joint Learning of Speech Enhancement and Noise Extraction

Minu, Kangwook, Hoirin · 2025

Read ParaNoise-SV: Integrated Approach for Noise-Robust Speaker Verification with Parallel Joint Learning of Speech Enhancement and Noise Extraction by Minu, Kangwook, Hoirin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07219.txt.

Introduction

Minu Kim, Kangwook Jang, and Hoirin Kim present ParaNoise-SV, a system that verifies speakers in noisy recordings. Joint training of speech cleanup and speaker checking helps, but most systems suppress noise only implicitly. They never learn what the noise itself looks like, so speech and noise stay tangled.

Their design runs two networks in parallel. One extracts the noise explicitly, while the other cleans the speech using guidance from the first. That split preserves traits that identify the speaker. In tests, the error rate fell by about eight percent relative to earlier joint models. The work shows that modeling noise head on strengthens voice checks.

Catalog record

CREATORMinu, Kangwook, Hoirin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07219

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ParaNoise-SV: Integrated Approach for Noise-Robust Speaker Verification with Parallel Joint Learning of Speech Enhancement and Noise Extraction Minu, Kangwook, Hoirin Abstract Noise-robust speaker verification leverages joint learning of speech enhancement (SE) and speaker verification (SV) to improve robustness. However, prevailing approaches rely on implicit noise suppression, which struggles to separate noise from speaker characteristics as they do not explicitly distinguish noise from speech during training. Although integrating SE and SV helps, it remains limited in handling noise effectively. Meanwhile, recent SE studies suggest that explicitly modeling noise, rather than merely suppressing it, enhances noise resilience. Reflecting this, we propose ParaNoise-SV, with dual U-Nets combining a noise extraction (NE) network and a speech enhancement (SE) network. The NE U-Net explicitly models noise, while the SE U-Net refines speech with guidance from NE through parallel connections, preserving speaker-relevant features. Experimental results show that ParaNoise-SV achieves a relatively 8.4% lower equal error rate (EER) than previous joint SE-SV models. Kim Jang Kim KAISTRepublic of Korea keywords speaker verification, speech enhancement, joint learning, noisy environments, noise disentanglement † † email: [email protected], [email protected], [email protected] 1 Introduction Rapid advancement of modern communication devices and voice-based technologies has highlighted the growing importance of speaker verification (SV) systems. These systems, whi-ch verify whether a given speech matches a target speaker, are critical for applications such as secure authentication and forensic analysis [ 1 ] . However, real-world environments are rarely free of noise, posing a major challenge for speaker verification systems [ 2 ] . Conventional solutions integrate separately trained speech enhancement (SE) modules to mitigate noise [ 3 , 4 ] , however, they can degrade speaker-specific information, leading to suboptimal embeddings and reduced verification accuracy [ 5 ] . Furthermore, independently trained SE and SV modules lack coherence with each other, as SE outputs may not align well with the learned feature distributions of the SV system [ 6 ] . To address these challenges, several studies have explored joint learning strategies for SE and SV. A previous study [ 7 ] integrates SE with an attention-based model, while another approach [ 6 ] introduces a multi-objective network for simultaneous feature enhancement and speaker embedding extraction. A U-Net-based approach [ 8 ] jointly optimizes SE and SV, and diffusion models [ 9 ] disentangle speaker and noise representations for improved robustness. Likewise, a noise-disentanglement adversarial framework [ 10 ] separates speaker-relevant and irrelevant information for robust embeddings in noise. Furthermore, a recent study [ 11 ] incorporates noise type and signal-to-noise ratio (SNR) level into SE, demonstrating the benefits of noise estimation. Meanwhile, self-supervised learning (SSL)-based verification [ 12 ] offers an alternative approach to improving noise robustness by fine-tuning large-scale pre-trained models. Despite its effective performance, its larger parameter size increases latency and computational demands. Beyond its role in joint learning with SV, SE has also been extensively studied as a standalone approach to improve speech robustness. Rather than relying solely on predefined noise attributes (e.g., amplitude and SNR) to guide enhancement [ 13 , 14 ] , recent SE methods focus on dynamically synthesizing noise representations within the SE network itself. For instance, a neural noise embedding approach [ 15 ] generates noise representations and applies conditional encoding with layer normalization, improving speech quality in end-to-end systems. Similarly, dual-stream SE models [ 16 , 17 ] process noise and speech separately, resulting in better enhancement. Figure 1: Overview of ParaNoise-SV architecture. Noise extraction (NE) network and speech enhancement (SE) network are trained in parallel and connected via parallel connections. The enhanced speech is then processed by speaker verification (SV) module for robust speaker embedding extraction. Building on these approaches, we propose ParaNoise-SV , a unified framework for noise-robust speaker verification. Instead of merely predicting coarse noise attributes, our model dynamically extracts and synthesizes background noise from speech to effectively remove speaker-irrelevant components. To achieve this, we design a noise extraction (NE) network and a speech enhancement (SE) network, both based on U-Net [ 18 ] , and train them simultaneously. These two networks are interconnected through parallel connections, allowing noise-related information to refine speech while preserving speaker-relevant features, as in Figure 1 . By jointly learning NE, SE, and speaker verification (SV), our system improves robustness against noise while maintaining high speaker discrimination. ParaNoise-SV achieves relatively 8.4% lower EER than previous joint SE-SV models in seen noise conditions and reduces EER by 8.2% in unseen noise conditions. 2 Methods 2.1 Overview of ParaNoise-SV ParaNoise-SV is a unified framework for noise-robust speaker verification, integrating NE, SE, and SV using dual encoder-decoder structures. It employs dual U-Nets with SE-ResNet [ 19 ] for simultaneous NE and SE, introducing parallel connections to ensure balanced separation while preserving speaker-relevant information. The NE network isolates noise, which the SE network leverages through parallel connections at each encoding stage, enabling dynamic noise suppression while maintaining speaker discriminability. Unlike conventional methods focusing solely on suppression or enhancement, ParaNoise-SV actively utilizes extracted noise at the feature level, preventing contamination in deeper representations. For speaker embedding extraction, ERes2NetV2 [ 20 ] is used, with channel adaptation blocks integrating the U-Net features via skip connections. The key frameworks are shown in Figures 2 and 3 . 2.2 Parallel Connections of Dual U-Nets The input spectrogram is first normalized using instance normalization [ 21 ] and processed through an initial convolutional layer, generating noise and speech feature maps N_{E,0} and S_{E,0} . Each encoder then extracts hierarchical representations using SE-ResNets [ 19 ] with depth L=4 . \displaystyle N_{E,i} \displaystyle=e_{N}^{i}(N_{E,i-1}),\quad i=1,\dots,L. (1) \displaystyle N_{D,0} \displaystyle=N_{E,L}. (2) \displaystyle N_{D,i} \displaystyle=d_{N}^{i}(N_{D,i-1},N_{E,L-i}),\quad i=1,\dots,L. (3) \displaystyle\hat{N} \displaystyle=\text{ConvTranspose}(N_{D,L},N_{E,0}). (4) The NE network encodes noise representations through encoder blocks e_{N} in Equation ( 1 ), refining noise features. The deepest encoded feature in Equation ( 2 ) initializes the decoding operations d_{N} , where skip connections aid noise extraction as in Equation ( 3 ). Finally, a transposed convolutional layer generates the estimated noise spectrogram \hat{N} in Equation ( 4 ). \displaystyle S_{E,i} \displaystyle=e_{S}^{i}(S_{E,i-1},N_{E,i-1}),\quad i=1,\dots,L (5) \displaystyle S_{D,0} \displaystyle=S_{E,L} (6) \displaystyle S_{D,i} \displaystyle=d_{S}^{i}(S_{D,i-1},S_{E,L-i}),\quad i=1,\dots,L (7) \displaystyle\hat{S} \displaystyle=\text{ConvTranspose}(S_{D,L},S_{E,0}) (8) Figure 2: Dual U-Nets for SE and NE networks are trained in parallel with parallel connections (dashed red line) for noise suppression. They use E and D layers based on SE-ResNet. The SV network processes enhanced speech using S layers from ERes2NetV2, with CA for channel alignment in skip connections and F for multi-channel fusion. Speaker embeddings are extracted in two stages via attentive statistics pooling (ASP). (C: convolution, T: transposed convolution, E: encoding, D: decoding, S: speaker extraction, F: bottom-up fusion, CA: channel adaptation) Figure 3: (a) SE-ResNet applies squeeze-and-excitation mechanisms to enhance important feature channels by adaptively reweighting them. (b) ERes2NetV2 incorporates channel-split hierarchical residual connections with expanded channel dimensions, improving multi-scale feature extraction. In Equation ( 5 ), the SE network incorporates parallel connections : information flows between two parallel networks, integrating noise features at each encoder block e_{S} . The encoded speech feature is decoded with skip connections as in Equation ( 7 ), and a final transposed convolutional layer outputs the enhanced speech spectrogram \hat{S} in Equation ( 8 ). This allows the SE network to utilize noise information from the NE network, improving noise suppression while preserving

Read the complete text