AMDM-SE: Attention-based Multichannel Diffusion Model for Speech Enhancement

Renana Opochinsky, Sharon Gannot · 2026

Read AMDM-SE: Attention-based Multichannel Diffusion Model for Speech Enhancement by Renana Opochinsky, Sharon Gannot (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13140.txt.

Introduction

Renana Opochinsky and Sharon Gannot bring a fresh tool to speech clean up. Diffusion models can rebuild a clean image from a noisy one step by step. Like ideas have been used on speech, where a time and pitch map is treated as an image.

Most phones and aids now carry more than one mic. The team uses that fact. Their method takes in feeds from many mics at once. It aims to use the gaps between feeds to pull speech out of noise with more care. For users of aids and call tools, more mics plus a strong rebuild model may mean speech that is far more clear.

Catalog record

CREATORRenana Opochinsky, Sharon Gannot
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-13140

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AMDM-SE: Attention-based Multichannel Diffusion Model for Speech Enhancement Renana Opochinsky, Sharon Gannot Abstract Diffusion models have recently achieved impressive results in reconstructing images from noisy inputs, and similar ideas have been applied to speech enhancement by treating time–frequency representations as images. With the ubiquity of multi-microphone devices, we extend state-of-the-art diffusion-based methods to exploit multichannel inputs for improved performance. Multichannel diffusion-based enhancement remains in its infancy, with prior work making limited use of advanced mechanisms such as attention for spatial modeling—a gap addressed in this paper. We propose AMDM-SE, an Attention-based Multichannel Diffusion Model for Speech Enhancement, designed specifically for noise reduction. AMDM-SE leverages spatial inter-channel information through a novel cross-channel time–frequency attention block, enabling faithful reconstruction of fine-grained signal details within a generative diffusion framework. On the CHiME-3 benchmark, AMDM-SE outperforms both a single-channel diffusion baseline and a multichannel model without attention, as well as a strong DNN-based predictive method. Simulated-data experiments further underscore the importance of the proposed multichannel attention mechanism. Overall, our results show that incorporating targeted multichannel attention into diffusion models substantially improves noise reduction. While multichannel diffusion-based speech enhancement is still an emerging field, our work contributes a new and complementary approach to the growing body of research in this direction. † † address: Faculty of Engineering, Bar Ilan University, Ramat-Gan, 5290002, Israel {renana.klainman,sharon.gannot}@biu.ac.il 1 Introduction Multichannel speech enhancement has been the subject of rich and comprehensive research in model-based approaches, as reviewed in [ 4 ] , where methods such as MVDR and related beamformers explicitly exploit spatial coherence. Building on these foundations, recent works have integrated deep neural networks (DNNs) with classical frameworks, yielding hybrid strategies that combine statistical modeling with data-driven learning. Representative examples include FaSNet [ 14 ] , SpatialNet [ 19 ] , McNet [ 28 ] , and explainable multichannel architectures [ 3 ] , which illustrate how attention and learned beamforming mechanisms enhance separation, denoising, and dereverberation. In parallel, diffusion models, originally developed for image synthesis [ 6 ] , have shown remarkable success when adapted to speech. In the single-channel domain, they have been applied to enhancement [ 13 ] , separation [ 15 ] , and generation [ 18 ] , demonstrating strong potential thanks to their ability to capture complex data distributions and generate perceptually natural audio. Unlike predictive models that directly map noisy inputs to clean targets, diffusion models iteratively refine estimates through learned denoising steps. Recent advances such as SGMSE+ [ 21 ] , StoRM [ 10 ] , conditional diffusion [ 13 ] , and VAE-guided diffusion [ 27 ] have further improved robustness and quality in the single-channel setting. Nevertheless, the application of diffusion-based speech enhancement in the multichannel domain remains relatively underexplored. Recent studies have made substantial progress in this direction: the multi-stream framework [ 16 ] integrates model-based priors with data-driven diffusion, mSGMSE [ 8 ] generalizes score-based enhancement to a MIMO architecture, and DiffCBF [ 9 ] combines convolutional beamforming with diffusion refinement. These works highlight the promise of combining physical modeling with generative methods and demonstrate the feasibility of diffusion for preserving spatial cues in multi-microphone arrays. Building on these directions, we propose AMDM-SE, an Attention-based Multichannel Diffusion Model for Speech Enhancement. Unlike prior multichannel diffusion approaches, AMDM-SE explicitly incorporates unique cross-channel time–frequency attention, enabling the model to jointly leverage spectral, temporal, and spatial cues. Extending the SGMSE backbone [ 21 ] , our method bridges multi-microphone coherence with generative refinement. Experiments on simulated data and the CHiME-3 dataset demonstrate that AMDM-SE outperforms both single-channel SGMSE and a multichannel variant without attention in terms of perceptual quality and intelligibility. The project page with audio samples can be found at https://AMDMSE.github.io . Figure 1 : Network architecture, AMDM-SE in the green frame and Cross Channel TF-Attention in the blue frame. 2 Problem Formulation In this paper, we address the noise reduction problem using an array of M microphones. Let x_{m}(n),\,m=1,\ldots,M be a noisy speech mixture recorded by the m-th microphone: x_{m}(n)=h_{m}(n)\ast s(n)+v_{m}(n), (1) where s(n) is the clean signal, h_{m}(n) is the RIR ( RIR ) between the speaker and the m -th microphone, and v_{m}(n) is the additive noise component as recieved by the m -th microphone. The received microphone signal in the STFT ( STFT ) domain is given by x_{m}(\ell,k) , with \ell=0,1,\dots,L-1 , k=0,1,\ldots,K-1 , and L,K are the number of time-frames and frequency bins, respectively. Define \mathbf{x} as a tensor comprising all microphones, time frames, and frequency bins. Define one of the microphone channels as the reference channel \mathbf{x}_{\text{ref}} , and all other M-1 channels as \mathbf{\bar{x}} , such that \mathbf{x}=\{\mathbf{x}_{\text{ref}},\mathbf{\bar{x}}\} . We opt for a complex STFT representation as the input feature, rather than the more common magnitude-only representation. Adopting the “engineering trick” in [ 21 ] , we compress the dynamic range of the amplitudes as follows: x_{m}(\ell,k)\leftarrow\beta|x_{m}(\ell,k)|^{\alpha}e^{i\angle(x_{m}(\ell,k))} , with \alpha=0.5 and \beta=3 . The objective of the proposed method is to estimate the clean non-reverberant speech signal s(n) . 3 Method: Attention-based Multichannel Diffusion Model for Speech Enhancement (AMDM-SE) The overall architecture of our network is illustrated in Figure 1 . The input to the network is a multichannel speech signal comprising M channels. This signal is first transformed into the complex TF ( TF ) domain using the STFT . The resulting representation is then processed by the AMDM-SE module—the diffusion-based core module of our system—which produces a single-channel enhanced signal in the STFT domain. Finally, the enhanced signal is transformed back to the time domain using the inverse STFT (iSTFT). 3.1 AMDM-SE Algorithm 1 AMDM-SE (following SGMSE [ 21 ] ) 1: Input: Noisy spectrograms \mathbf{x}=\{\mathbf{x}_{\text{ref}},\bar{\mathbf{x}}\} , clean spectrogram \mathbf{s}_{0} 2: Hyperparameters (Diffusion Process): \gamma , \sigma_{\min} , \sigma_{\max} , T , t_{\epsilon} 3: Output: Enhanced spectrogram \hat{\mathbf{s}} 4: Forward Diffusion Process (Training) 5: for each training sample (\mathbf{s}_{0},\mathbf{x}) do 6: Sample t\sim\mathcal{U}(t_{\epsilon},T) 7: Compute \boldsymbol{\mu_{t}}=e^{-\gamma t}\mathbf{s}_{0}+(1-e^{-\gamma t})\mathbf{x} 8: Compute \sigma_{t}^{2}=\frac{\sigma_{\min}^{2}\left(\left(\frac{\sigma_{\max}}{\sigma_{\min}}\right)^{2t}-e^{-2\gamma t}\right)}{2\log\left(\frac{\sigma_{\max}}{\sigma_{\min}}\right)} 9: Sample \mathbf{z}\sim\mathcal{N}(0,\mathbf{I}) 10: Generate \mathbf{s}_{t}=\boldsymbol{\mu_{t}}+\sigma_{t}\cdot\mathbf{z} 11: Compute target score: \text{score}_{\theta}=\nabla_{\mathbf{s}_{t}}\log p(\mathbf{s}_{t}|\mathbf{s}_{0},\mathbf{x}_{\text{ref}},\bar{\mathbf{x}}) 12: Compute loss: \mathcal{L}=\mathbb{E}\left[\lambda(t)\cdot\left\|\text{score}_{\theta}+\frac{\mathbf{s}_{t}-\boldsymbol{\mu_{t}}}{\sigma_{t}^{2}}\right\|^{2}\right] 13: Update parameters \theta using gradient descent 14: end for 15: Reverse Sampling (Inference) 16: Initialize \mathbf{s}_{T}\sim\mathcal{N}(\boldsymbol{\mu_{t}},\sigma_{T}^{2}\mathbf{I}) 17: for t=T down to t_{\epsilon} with step -\Delta t do 18: Compute drift: f(\mathbf{s}_{t},\mathbf{x}_{\text{ref}})=\gamma(\mathbf{x}_{\text{ref}}-\mathbf{s}_{t}) 19: Compute diffusion: g(t)=\sigma_{\min}\left(\frac{\sigma_{\max}}{\sigma_{\min}}\right)^{t}\sqrt{2\log\left(\frac{\sigma_{\max}}{\sigma_{\min}}\right)} 20: Estimate score: \text{score}_{\theta}(\mathbf{s}_{t},t,\mathbf{x}_{\text{ref}},\bar{\mathbf{x}}) 21: Sample \mathbf{z}\sim\mathcal{N}(0,\mathbf{I}) 22: Update: 23: \mathbf{s}_{t-\Delta t}=\mathbf{s}_{t}+\left(f(\mathbf{s}_{t},\mathbf{x})-g^{2}(t)\text{score}_{\theta}\right)\Delta t+g(t)\sqrt{\Delta t}\cdot\mathbf{z} 24: end for 25: return \hat{\mathbf{s}}=\mathbf{s}_{0} AMDM-SE is a score-based diffusion model in the complex STFT domain, aiming at speech enhancement using multichannel recordings. Building on SGMSE [ 21 ] , we introduce the main modifications to the original formulation and provide a concise summary of the diffusion process in Algorithm 1. Given the noisy and reverberant multichannel speech, \mathbf{x} , AMDM-SE generates enhanced speech, \hat{\mathbf{s}} , using a conditional diffusion model. The diffusion model comprises three main stages: Forward Process: Gradually adds noise to the clean spectrogram \mathbf{s}_{0} , guided by the noisy observation \mathbf{x} , to produce a noisy version \mathbf{s}_{t} at step t , as originally introduced in [ 23 ] and subsequently applied to speech enhancement in [ 21 ] . Score Estimation: Trains a neural network \text{score}_{\theta}(\cdot) to approximate the gradient of the log probability density of \mathbf{s}_{t} , given \mathbf{x} , namely \nabla_{\mathbf{s}_{t}}\log p(\mathbf{s}_{t}\mathbf{s}_{0},\mathbf{x}_{\text{ref}},\bar{\mathbf{x}})

Read the complete text