MDDM: A Multi-view Discriminative Enhanced Diffusion-based Model for Speech Enhancement

Nan, Zhaolong, Xiaonan · 2025

Read MDDM: A Multi-view Discriminative Enhanced Diffusion-based Model for Speech Enhancement by Nan, Zhaolong, Xiaonan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13029.txt.

Introduction

Nan Xu, Zhaolong Huang, and Xiaonan Zhi clean noisy speech by marrying a discriminative first pass with a short diffusion polish. Their 2025 paper notes the split in the field: supervised models can distort speech, while pure generative ones cost heavy compute. Their model, MDDM, takes features from time, frequency, and noise domains.

A prediction network blends the three views into a rough spectrogram. Because that output already sits near clean speech, only a few diffusion sampling steps finish the job. Fewer steps mean competitive quality at lower cost than rival diffusion methods. Tests on a public dataset and a real-world set confirm gains on both listener and metric scores. The authors pitch the hybrid as a practical route to fast, clean enhancement.

Catalog record

CREATORNan, Zhaolong, Xiaonan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13029

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MDDM: A Multi-view Discriminative Enhanced Diffusion-based Model for Speech Enhancement Nan, Zhaolong, Xiaonan Abstract With the development of deep learning, speech enhancement has been greatly optimized in terms of speech quality. Previous methods typically focus on the discriminative supervised learning or generative modeling, which tends to introduce speech distortions or high computational cost. In this paper, we propose MDDM , a M ulti-view D iscriminative enhanced D iffusion-based M odel. Specifically, we take the features of three domains (time, frequency and noise) as inputs of a discriminative prediction network, generating the preliminary spectrogram. Then, the discriminative output can be converted to clean speech by several inference sampling steps. Due to the intersection of the distributions between discriminative output and clean target, the smaller sampling steps can achieve the competitive performance compared to other diffusion-based methods. Experiments conducted on a public dataset and a real-world dataset validate the effectiveness of MDDM, either on subjective or objective metric. Xu Huang Zhi BeijingChina keywords speech enhancement, multi-view, discriminative model, diffusion model, noise domain † † email: {xn430658,zhaolong.hzl,dexiao.zxn}@alibaba-inc.com 1 Introduction Figure 1: The overall architecture of MDDM, multi-view discriminative network and encoder layer. In subfigure (a), S_{n} and T_{n} are noisy signals in frequency and time domains, respectively. Two D-* are encoder and decoder of discriminative network and output spectrogram is S_{p} . S_{t} is the sampled spectrogram at time-step t . The intermediate multi-view feature C_{m} and output S_{p} are the conditions of diffusion model. In subfigure (b), S-* and T-* means frequency and time domains, respectively. NPM is the noise perception module and \oslash means the modulation mode. Note that the skip connections are not drawn. In subfigure (c), the encoder module is shown. GELU is Gaussian Error Linear Unit and GN denotes GroupNorm. The dilation rates are 1 and 2, respectively. In the real world, clean speech signals are always disturbed by various environmental noises and reverberations, which seriously affects speech perceptual quality and intelligibility. Therefore, speech enhancement is thus an underlying task, which aims to recover clean speech signal from noisy speech. Traditional speech enhancement methods can use the statistical properties of the noisy and clean target signals in time-frequency or spatial domain [ 1 ] . With the success of deep learning, the enhancement performance has achieved some breakthroughs in the last decade. Deep learning based methods can be divided into two different categories: discriminative approaches and generative approaches. Discriminative approaches are dominated by supervised learning algorithms that obtain clean target speech from noisy speech, always training with labeled samples. These methods typically take time domain, time-frequency (TF) domain or both as network input to learn a deterministic mapping. Specifically, the time or time-frequency domain methods take the single-view feature as input to predict waveform or magnitude-phase related training target [ 2 , 3 , 4 , 5 , 6 ] . Instead, some approaches integrate time domain and time-frequency domain in the speech enhancement framework, primely eliminating non-stationary (e.g., impulse-like noise) and stationary noises simultaneously [ 7 , 8 , 9 ] . In addition, several methods follow the ideas of generative models for speech enhancement, such as variational autoencoder (VAE) [ 10 , 11 , 12 ] or Generative adversarial networks (GANs) [ 13 , 14 ] . In contrast, diffusion-based speech enhancement models have recently gained attention due to the superior enhancement effects [ 15 , 16 , 17 , 18 ] . Diffusion-based enhancement framework usually adds noise from the Wiener process to make clean target speech into a tractable prior (e.g., standard normal distribution), which is called the forward process. For the reverse process, a trained neural network is used to generate clean speech from the aforementioned prior distribution, such as SGMSE+ [ 19 ] . Besides, the appropriate condition guided generative process can achieve more superior and stable enhancement performance [ 20 , 21 ] . Although the above approaches (discriminative and generative) have achieved significantly superior performance for speech enhancement, there are still some challenges: • For discriminative methods, a deterministic map is learned during the training process. However, training data is a finite set and cannot cover all possible noise conditions to guarantee the generalizability capacity in unseen situations. Various noise types and levels can also result in distortions, especially for complex data distributions. • Diffusion models are always trained to learn a standard normal distribution, which requires hundreds of inference steps leading to a heavy computational cost. Additionally, the defined standard white Gaussian is also not the case of environmental noise. Although Lemercier et al. [ 20 ] utilizes a predictive model as a guidance, the predictive results that deviate from target distributions to a certain extent are used to compute score function, resulting in a suboptimal performance. To address the aforementioned problems, we propose a multi-view discriminative enhanced diffusion-based model, named MDDM. In this work, a multi-view discriminative network is firstly used to predict an initial result that overlaps enough with a clean target distribution. Specifically in this discriminative network, the STFT-based U-Net framework is used as a backbone network. In addition, a parallel time U-Net and a noise modulation module are integrated into the backbone, which reduces speech distortions and improves noise perception abilities. To further improve the performance, we introduce a diffusion model using the discriminative intermediate feature with multi-view information as a conditioner. Furthermore, since there is an intersection distribution between target and discriminative result, they can achieve a nearly same noisy distribution by only several forward steps. Next, this noisy distribution can be converted to a more superior clean result by only several inference sampling steps, which accelerates inference speed. Assuming that discriminative results are reliable enough, the fewer sampling steps are required. Experiment results on two datasets show that our proposed method surpasses other baseline methods in terms of enhancement performance with the sampling step of only 30. 2 Proposed Method In this section, our proposed model will be introduced, as illustrated in Figure 1 a. First, the discriminative network takes multi-view features as inputs to get the initial output results. Next, the output results and intermediate features with multi-view fusion information guide the diffusion model during training and inference processes. Also, the discriminative and diffusion tasks are jointly optimized by a multi-task learning scheme to improve speech enhancement performance. 2.1 Multi-view discriminative network As illustrated in Figure 1 b, the multi-view discriminative network contains three modules: STFT-based backbone network, waveform-base U-Net and noise perception module. STFT-based backbone. Due to the superiority of U-Net in speech enhancement [ 22 , 9 ] , we select it as the backbone and take the noisy spectrogram S_{n} as input, the real and imaginary parts are considered as two channels. Specifically, the noisy spectrogram S_{n} is fed sequentially into four convolution down-sampling encoders, a TF-GRU module and four convolution up-sampling decoders. Particularly, the TF-GRU module learns to model time and frequency relations respectively, which is similar to [ 22 ] , except that GRU is adopted in our work. Note that a downsampling convolution layer after the TF-GRU module is used to transform feature dimension to 1 for integrating time branch. The decoder is built symmetrically with skip connections and outputs the predicted spectrogram S_{p} . Waveform-base

Read the complete text