BINAURAL SPEECH ENHANCEMENT USING COMPLEX CONVOLUTIONAL RECURRENT NETWORKS
Vikas Tokala, Eric Grinstein, Mike Brookes, Simon Doclo, Jesper Jensen, Patrick A. Naylor \sthanks This work was supported by funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 956369 and the UK Engineering and Physica · 2025
Read BINAURAL SPEECH ENHANCEMENT USING COMPLEX CONVOLUTIONAL RECURRENT NETWORKS by Vikas Tokala, Eric Grinstein, Mike Brookes, Simon Doclo, Jesper Jensen, Patrick A. Naylor \sthanks This work was supported by funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 956369 and the UK Engineering and Physica (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20023.txt.
Introduction
Vikas Tokala and five co-authors present, in a 2025 paper, a speech-cleaning network for two-ear devices, from hearing aids to virtual reality headsets. Such tools must lift speech out of noise while keeping the sense of where sounds come from, which one-ear cleaning destroys. Their network works end to end on the complex sound signal.
The design stacks a convolutional encoder and decoder around a memory block that handles complex values. It outputs a mask for each ear in the time-frequency plane. A new loss rule grades not only clarity and noise cut but also how well the spatial cues survive. Against baseline methods, the system raised estimated intelligibility and cut noise in scenes with one target speaker and several kinds of diffuse background noise.
Catalog record
| CREATOR | Vikas Tokala, Eric Grinstein, Mike Brookes, Simon Doclo, Jesper Jensen, Patrick A. Naylor \sthanks This work was supported by funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 956369 and the UK Engineering and Physica |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-20023
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BINAURAL SPEECH ENHANCEMENT USING COMPLEX CONVOLUTIONAL RECURRENT NETWORKS Vikas Tokala, Eric Grinstein, Mike Brookes, Simon Doclo, Jesper Jensen, Patrick A. Naylor \sthanks This work was supported by funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 956369 and the UK Engineering and Physical Sciences Research Council [grant number EP/S035842/1] Abstract From hearing aids to augmented and virtual reality devices, binaural speech enhancement algorithms have been established as state-of-the-art techniques to improve speech intelligibility and listening comfort. In this paper, we present an end-to-end binaural speech enhancement method using a complex recurrent convolutional network with an encoder-decoder architecture and a complex LSTM recurrent block placed between the encoder and decoder. A loss function that focuses on the preservation of spatial information in addition to speech intelligibility improvement and noise reduction is introduced. The network estimates individual complex ratio masks for the left and right-ear channels of a binaural hearing device in the time-frequency domain. We show that, compared to other baseline algorithms, the proposed method significantly improves the estimated speech intelligibility and reduces the noise while preserving the spatial information of the binaural signals in acoustic situations with a single target speaker and isotropic noise of various types. SNR Signal-to-Noise Ratio fwSegSNR frequency-weighted Segmental SNR SOBM STOI-optimal Binary Mask STOI Short-Time Objective Intelligibility WSTOI Weighted STOI HSWOBM High-resolution Stochastic WSTOI-optimal Binary Mask STFT Short Time Fourier Transform ILD Interaural Level Differences IPD Interaural Phase Differences ITD Interaural Time Differences LSA Log Spectral Amplitude OM-LSA Optimally-modified Log Spectral Amplitude DNN Deep Neural Network RNN Recurrent Neural Network ERB Equivalent Rectangular Bandwidth DFT Discrete Fourier Transform VSSNR Voiced-Speech-plus-Noise to Noise Ratio SPP Speech Presence Probability MBSTOI Modified Binaural STOI ReLU Rectified Linear Unit SNR Signal-to-Noise Ratio ISTFT Inverse STFT TF Time-Frequency WGN White Gaussian Noise RTF Relative Transfer Function EVD Eigen Value Decomposition PSD Power Spectral Density IRM Ideal Ratio Mask NPM Normalized Projection Misalignment CRM Complex Ratio Mask FTB Frequency Transformation Block FTM Frequency Transformation Matrix FAL Frequency Attention Layer MWF Multichannel Wiener Filter MVDR Minimum Variance Distortionless Response MSC Magnitude Squared Coherence VAD Voice Activity Detector PESQ Preceptual Evalution of Speech Quality MLP Multi-Layer Perceptrons AED Auto Encoder-Decoder CED Convolutional Encoder-Decoder CNN Convolutional Neural Network CRN Convolutional Recurrrent Network LSTM Long short-term memory TNN Transformer Neural Networks NLP Natural Language Processing IBM Ideal Binary Mask IRM Ideal Ratio Mask SI-SNR Scale Invariant SNR GRU Gated Recurrent Unit HRIRs Head Related Impulse Responses SSN Speech Shaped Noise HATS Head and Torso Simulator PReLU Parametric Rectified Linear Unit BSOBM Binaural STOI-Optimal Masking BMWF Binaural MWF SegSNR Segmental SNR RIR Room Impulse Responses BiTasNet Binaural TasNet SI-SNR Scale Invariant SNR BCCRN Binaural Complex Convolutional Recurrent Network BRIR Binaural Room Impulse Responses † † address: 1 Department of Electrical and Electronic Engineering, Imperial College London, UK 2 Department of Medical Physics and Acoustics, University of Oldenburg, Germany. 3 Demant A/S, Smørum, Denmark. 4 Department of Electronic Systems, Aalborg University, Denmark 1 Introduction Binaural speech enhancement has gained significant attention as a state-of-the-art approach for enhancing speech in various applications, including hearing aids and augmented/virtual reality devices [ 1 , 2 ] . Binaural signals preserve the spatial characteristics of sounds which help listeners in noisy acoustic environments achieve better speech intelligibility and accurate sound source localization [ 3 ] . The fundamental spatial cues that help in localizing sounds and improving intelligibility are ILD ( ILD ) and IPD ( IPD ) or ITD ( ITD ) [ 4 ] . Previously proposed methods for binaural speech enhancement include multichannel Wiener filters [ 5 , 6 ] , beamforming-based [ 1 ] , and mask-based enhancement methods [ 7 , 8 ] . Binaural speech separation using time-domain CED ( CED ) was proposed in [ 9 ] . Recent advancements in deep learning techniques have led to remarkable improvements in monaural speech enhancement. These methods, whether applied in the time domain [ 10 , 11 ] or the TF ( TF ) domain [ 12 , 13 , 14 ] , demonstrate impressive results. Spectrograms are commonly used as input to networks in the TF domain for speech enhancement [ 7 , 14 , 15 ] . Many methods in this domain focus on enhancing only the magnitude of the spectrogram while using the noisy phase information for reconstructing the enhanced speech signal [ 7 , 16 ] . To address optimal phase estimation and signal reconstruction, some approaches jointly estimate both magnitude and phase by utilizing complex-valued spectrograms. These methods have shown promising results and can outperform real-valued networks in monaural speech enhancement [ 16 , 15 ] . The CRN ( CRN ) introduced in [ 13 ] employed a CED ( CED ) architecture with LSTM ( LSTM ) blocks placed in between the encoder and decoder. In [ 15 ] , a deep complex CRN was trained to optimize the SI-SNR ( SI-SNR ) for monaural speech signals. However, using a similar approach for binaural signals could be damaging to the interaural cues. More specifically, for the case of binaural signals, phase information is vital for preserving the IPD values and the enhanced signals should preserve level differences as the noisy signal to retain ILD . While the model may effectively reduce noise and enhance speech intelligibility, modifying the level and phase information could potentially impact the spatial information of the target, leading to a compromised ability for localization and spatial awareness [ 4 , 17 ] . In [ 18 ] , a complex convolutional attention-based transformer network has been proposed which uses a similar loss function based on interaural cues. Our proposed method, BCCRN ( BCCRN ), uses a complex-valued CED -based recurrent network for binaural speech enhancement and introduces terms in the loss function to improve speech intelligibility while preserving the interaural cues based on human perception of the target speech signal with a smaller complex recurrent network compared to [ 18 ] . 2 BCCRN Model Architecture Figure 1 : Model architecture of BCCRN ( BCCRN ). The proposed BCCRN is trained to estimate individual CRM ( CRM ) for each channel. The block diagram of the model architecture is shown in Fig. 1 . The STFT ( STFT ) blocks transform the input signal into the TF domain. The encoder block is made of 6 complex convolutional layers with PReLU ( PReLU ) activation and employs batch normalization. Separate encoder and decoder blocks are used for the left and right-ear channels to estimate the individual CRM s. The decoder block consists of 6 complex convolutional layers which are symmetric in design to the layers in the encoder to reconstruct the signal. The encoder extracts high-level features from the input spectrograms and reduces the resolution of the input. The decoder rebuilds the low-resolution features back to the original size. The encoded information from the left and right encoder blocks is concatenated and provided to the complex LSTM block. The LSTM layers are designed to model the frequency dependencies. Skip connections are placed between the encoder and decoder layers based on the CRN architecture which improves the information flow and facilitates network optimization [ 13 ] . The