Test-Time Adaptation for Speech Enhancement via Domain Invariant Embedding Transformation

Tobias Raichle, Niels Edinger, Bin Yang · 2025

Read Test-Time Adaptation for Speech Enhancement via Domain Invariant Embedding Transformation by Tobias Raichle, Niels Edinger, Bin Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04280.txt.

Introduction

Tobias Raichle and two co-authors adapt speech-cleaning models to new rooms and voices at test time in a 2025 research paper. Models that remove noise from recordings shine when test audio resembles training audio, then stumble when accents, languages, or noise types change. Their method, LaDen, works inside the model's latent space, the compressed internal description of sound. A simple linear reshaping turns noisy latent patterns toward clean ones, guided by strong pre-trained speech representations.

That reshaping transfers across settings, producing stand-in clean labels for the new environment. The main model then tunes itself using those labels, with no hand-labeled target data needed. A new benchmark covers many shift types. LaDen beat rival approaches, especially across speakers and languages.

Catalog record

CREATORTobias Raichle, Niels Edinger, Bin Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-04280

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Test-Time Adaptation for Speech Enhancement via Domain Invariant Embedding Transformation Tobias Raichle, Niels Edinger, Bin Yang Abstract Deep learning-based speech enhancement models achieve remarkable performance when test distributions match training conditions, but often degrade when deployed in unpredictable real-world environments with domain shifts. To address this challenge, we present LaDen ( latent denoising ), the first test-time adaptation method specifically designed for speech enhancement. Our approach leverages powerful pre-trained speech representations to perform latent denoising, approximating clean speech representations through a linear transformation of noisy embeddings. We show that this transformation generalizes well across domains, enabling effective pseudo-labeling for target domains without labeled target data. The resulting pseudo-labels enable effective test-time adaptation of speech enhancement models across diverse acoustic environments. We propose a comprehensive benchmark spanning multiple datasets with various domain shifts, including changes in noise types, speaker characteristics, and languages. Our extensive experiments demonstrate that LaDen consistently outperforms baseline methods across perceptual metrics, particularly for speaker and language domain shifts. † † address: Institute of Signal Processing and System Theory University of Stuttgart 1 INTRODUCTION 1.1 Motivation Modern deep learning based speech enhancement (SE) models have achieved remarkable success and are able to deliver natural sounding denoised speech even for very noisy recordings. However, previous works focus mostly on the performance on a small number of benchmark datasets, e.g. the ubiquitous VoiceBank+DEMAND (VBD) [ 1 ] dataset, whereas generalization has not received as much attention. This has resulted in SE models performing exceptionally well, as long as the target data distribution closely matches the training distribution, but diminished performance under distribution shifts. Table 1 shows that the SE model CMGAN [ 2 ] trained on the source EARS-W [ 3 ] dataset performs significantly worse on the target datasets (EARS-D and VBD) than the same model trained directly on those target domains. Since SE models are commonly deployed in unpredictable environments, generalization is a key factor for successful, practical speech enhancement systems. Due to this unpredictability, training SE models on all possible target distributions is not feasible. To nonetheless achieve generalization, models must be able to adapt themselves to any test environment without relying on labeled data from the specific target domain. Methods that perform adaptation under these conditions can be summarized under unsupervised domain adaptation (UDA) . In practice, adaptation methods cannot rely on the availability of labeled source data at test time. Besides the privacy concerns of sharing labeled source data, it is not practical to store and re-process the large source domain dataset during adaptation [ 4 ] . This is especially true for speech enhancement, since these models are commonly deployed to edge devices with limited storage and compute resources. Additionally, real-world deployments demand adaptation to happen simultaneously to inference, as separate adaptation phases would introduce unacceptable latency. Online adaptation using only the source model and unlabeled target data is known as test-time adaptation (TTA) and presents the most general and practical adaptation paradigm [ 5 ] . While there are previous works covering UDA in the context of speech enhancement, to the best of our knowledge, this is the first work exploring TTA for speech enhancement. The main contributions of this work can be summarized as follows: • We explore the application of TTA to speech enhancement and propose a comprehensive benchmark for evaluation, spanning various domain shifts, including different noise types, speaker characteristics, and languages. • We propose and empirically verify DIET ( domain invariant embedding transformation ) as an effective method for translating between noisy and clean speech in embedding spaces. • Based on DIET we introduce LaDen (latent denoising), a novel approach that enables TTA for speech enhancement. • By conducting a large number of experiments, we identify the strengths and limitations of the proposed method across different domain shifts. Table 1 : Comparison between the source CMGAN trained on the EARS-W [ 3 ] dataset and models trained on the respective target domain (target CMGAN). EARS-D denotes the dataset introduced in Section 4 . SI-SDR in dB. EARS-D VBD PESQ \uparrow SI-SDR \uparrow PESQ \uparrow SI-SDR \uparrow No denoising 1.398 -1.109 1.970 8.444 Source CMGAN 2.701 4.091 3.234 10.094 Target CMGAN 2.763 11.823 3.399 20.071 1.2 Problem Setting Given a recording of corrupted speech \bm{y}\in\mathcal{A} , the goal of SE is to estimate the uncorrupted speech \bm{x}\in\mathcal{A} , where \mathcal{A} denotes the set of all audio signals [ 6 ] . Generally, corruptions can include additive noise, reverberation and echoes, limited bandwidth or compression artifacts. While the proposed method can be applied to all distortion types, this work only considers additive noise and leaves other corruptions for future research. Thus, the setting can be modeled by \displaystyle\bm{y}=\bm{x}+\bm{n}, where \bm{n}\in\mathcal{A} denotes the additive noise. The task of the SE model {f_{\theta}:\mathcal{A}\rightarrow\mathcal{A}} is to estimate the clean signal \hat{\bm{x}}\in\mathcal{A} \displaystyle\hat{\bm{x}}=f_{\theta}(\bm{y}). The source SE model f_{\theta} is trained using the labeled source dataset {\mathcal{D}_{\mathrm{S}}\subset\mathcal{A}\times\mathcal{A}} , consisting of pairs of clean and noisy speech segments. Given the trained source model, the task of TTA is to adapt the model with only the unlabeled target dataset {\mathcal{D}_{\mathrm{T}}\subset\mathcal{A}} , while simultaneously performing inference. We assume a distribution shift between the source ( \mathrm{S} ) and target ( \mathrm{T} ) datasets that can manifest itself in a shift in the speech distribution p_{X} and/or the noise distribution p_{N} , resulting in the mismatch p_{\mathrm{S};X,N}\neq p_{\mathrm{T};X,N} and therefore p_{\mathrm{S};Y}\neq p_{\mathrm{T};Y} . However, the predictive distribution p_{X|Y} , i.e., the SE task, stays the same p_{\mathrm{S};X|Y}(\bm{x}|\bm{y})=p_{\mathrm{T};X|Y}(\bm{x}|\bm{y}) [ 7 , 8 ] . TTA assumes the model f_{\theta} as given, i.e. no changes to the source training or model architecture can be made. This is in contrast to test-time training (TTT) , where an auxiliary self-supervised task is added during source training to be used later for adaptation [ 9 ] . In the field of TTA concerning classification, most methods rely on the probabilistic model output to perform adaptation [ 10 ] . The main approaches can be summarized under entropy minimization [ 4 ] , feature alignment [ 7 , 11 ] and pseudo-labeling [ 12 ] , each with various extensions to improve stability [ 13 , 5 ] , generalization [ 12 , 14 ] or efficiency [ 15 ] . Clearly, entropy minimization does not easily translate to SE because SE models output a direct estimate of the clean signal instead of a probability distribution. Feature alignment typically enforces consistency of model features under label-preserving input perturbations [ 16 ] . However, as SE models modify their input, they cannot be invariant to augmentations that affect the speech component. Designing an output preserving augmentation is therefore non-trivial, as the assumption of general invariance to small perturbations is not valid. Further, designing a consistency metric that is well aligned with the perceptual SE task poses an additional challenge. A subset of feature alignment methods update the model’s batch normalization statistics [ 11 ] , but this has limited applicability in SE , where many architectures (including those used in this study) do not rely on batch normalization layers. Similarly to entropy minimization, estimating pseudo-labels, i.e., computing a proxy for the clean signal, is not straightforward for SE , as the direct signal estimation in SE as

Read the complete text