Neuromorphic Speech Enhancement with Dual-Branch Spiking Neural Networks

Taiyu, Wenbin, Haoyi, Yuhan, Haibing · 2026

Read Neuromorphic Speech Enhancement with Dual-Branch Spiking Neural Networks by Taiyu, Wenbin, Haoyi, Yuhan, Haibing (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23761.txt.

Introduction

Meng and colleagues clean noisy speech with a brain inspired spiking network that uses a fraction of the usual parameters, in a 2026 paper on neuromorphic computing. Speech enhancement recovers clean speech from noisy recordings and fronts hearing aids and voice recognition. Spiking neural networks compute with binary spikes, like neurons, and suit low power hardware. Their binary activations have kept quality behind conventional networks, though.

The proposed network, GSU DBNet, runs two branches at once. One models the magnitude spectrum, how strong each frequency is, and the other the complex spectrum, which also carries phase. A gated spiking unit with a dual path design exploits both time and frequency structure. On a standard benchmark it reaches a perceptual quality score just above three with under four hundred thousand parameters, only about five to eleven percent of conventional model sizes.

Catalog record

CREATORTaiyu, Wenbin, Haoyi, Yuhan, Haibing
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.23761

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neuromorphic Speech Enhancement with Dual-Branch Spiking Neural Networks Taiyu, Wenbin, Haoyi, Yuhan, Haibing Abstract Spiking neural network (SNN)-based neuromorphic speech enhancement has emerged as a promising paradigm due to its energy efficiency, yet it still underperforms classical artificial neural network (ANN)-based approaches owing to binary activations and the lack of well-designed network architectures. To overcome this limitation, we propose a novel dual-branch spiking neural network architecture equipped with a gated spiking unit (GSU), termed GSU-DBNet. Specifically, GSU-DBNet simultaneously models the speech magnitude spectrum and complex spectrum, predicting the corresponding magnitude and complex spectral masks. Meanwhile, a dual-path GSU module is adopted to exploit temporal and frequency information for enhanced spatiotemporal feature representation. Experiments on a popular benchmark dataset show that GSU-DBNet achieves a PESQ score of 3.04 with only 394K parameters, outperforming existing SNN-based methods while using only 4.5%–10.6% of the parameters of representative ANN-based models. Meng Jiang Zhang Zhou Yin keywords speech enhancement, spiking neural networks, neuromorphic, dual-branch, parameter efficiency † † address: School of Communication Engineering, Hangzhou Dianzi University, Hangzhou, China † † email: {meng_taiyu, wbjiang, hyzhang11, yhzhou05, yhb}@hdu.edu.cn 1 Introduction Speech enhancement (SE) aims to recover clean speech from noisy observations and serves as a critical front-end for hearing aids, automatic speech recognition, and real-time communication. Deep neural networks have become the dominant paradigm [ 29 , 28 , 12 ] , achieving remarkable speech enhancement performance [ 35 , 13 ] . However, current high-performance models typically require millions of parameters and intensive floating-point operations [ 9 , 15 , 18 ] , whose computational and energy overhead is difficult to accommodate within the low-power, low-latency constraints of edge devices. Spiking neural networks (SNNs) offer a fundamentally different paradigm: binary spikes and event-driven dynamics enable temporal modeling with fewer parameters and are natively compatible with low-power neuromorphic hardware [ 5 ] . With surrogate gradient training [ 21 ] , SNNs have achieved near-ANN performance on speech recognition [ 33 ] , keyword spotting [ 1 ] , and other tasks, with recent extensions to speech enhancement [ 6 ] . The Gated Spiking Unit (GSU) [ 7 ] retains only a forget gate, halving the number of recurrent layer parameters relative to Long Short-Term Memory (LSTM) and providing a foundation for parameter-efficient SNN-based SE models. However, existing SNN-based SE methods generally adopt single branch architectures that model only a single spectral dimension. For example, Spiking-FullSubNet [ 7 ] focuses on complex spectral features and achieves parameter compression, but a notable quality gap with conventional networks remains. DPSNN [ 26 ] constructs a dual-path structure on time-domain waveforms and uniformly applies SNN units without investigating the suitability of spiking dynamics across signal domains. These methods fail to exploit the complementary advantages of magnitude and complex spectra in noise suppression and phase recovery. In the conventional neural network domain, lightweight speech enhancement has developed effective architectural paradigms. Dual-path recurrent networks [ 19 , 20 ] factorize long sequences into time and frequency dimensions for separate modeling, substantially reducing computational complexity. For spectral representation, methods such as complex spectral modeling [ 9 ] , sub-band decomposition, dual-branch estimation [ 34 ] , and multi-domain fusion [ 15 , 18 ] exploit complementary information across different representations to improve performance. However, these gains mainly stem from macro-level architectural changes, while the core recurrent modules remain largely unchanged. LSTM [ 8 ] is still widely used, and even GRU [ 4 ] , despite fewer gates, relies on continuous hidden states and dense floating-point operations, limiting energy efficiency. Integrating spiking units into these architectures could achieve both efficiency and performance gains, yet no prior work has systematically explored SNN modeling across feature dimensions in a dual-path framework. Inspired by dual-path architectures [ 19 , 11 ] and gated spiking modeling [ 7 ] , we propose GSU-DBNet , a dual-branch encoder–separator–decoder architecture for speech enhancement based on gated spiking units. Specifically, the separator employs a bidirectional BiGSU in the frequency path to capture global spectral correlations and a unidirectional GSU in the time path for causal temporal modeling. A dual-branch decoder simultaneously estimates a complex mask for phase-aware reconstruction via DeepFilter and a magnitude mask for energy envelope estimation, with the two outputs fused through weighted averaging. On VoiceBank+DEMAND, GSU-DBNet achieves PESQ 3.04 with only 394K parameters , a parameter count of only 4.5%–10.6% of representative ANN methods, while improving PESQ by 0.84 and 0.38 over DPSNN [ 26 ] and Spiking-FSN [ 7 ] respectively. Overall, the main contributions of this work are threefold: • We propose GSU-DBNet, a dual-branch SNN architecture with gated spiking units, for joint magnitude–complex spectrum modeling and spatiotemporal feature extraction. • GSU-DBNet achieves competitive speech quality with fewer parameters than ANN-based methods and demonstrates significant improvements over existing SNN approaches. • Ablation studies reveal that the binary output bottleneck makes the single-gate design optimal, while additional gates degrade performance. 2 Proposed method 2.1 Architecture overview Noisy STFT Encoder 3 \times Conv2d+CBAM Freq: BiGSU Time: GSU Full-GSU DP Block \times 2 cDecoder TransConv mDecoder TransConv cMask \tanh(\cdot) mMask \sigma(\cdot) Deep Filter \oplus iSTFT Enhanced (3,F,T) (64,F^{\prime},T) Figure 1 : Overview of the GSU-DBNet architecture with dual-path spiking blocks and a complex-magnitude dual-branch decoder. As illustrated in Figure 1 , GSU-DBNet follows an encoder–separator–decoder paradigm. The noisy speech is first transformed via STFT, from which the real part, imaginary part, and magnitude spectrum are extracted and concatenated into a three-channel spectral input. The encoder comprises three convolutional blocks, each consisting of a Conv2d layer, GroupNorm, PReLU, and a CBAM [ 32 ] attention module. The first two layers progressively compress the frequency dimension and increase the channel count through strided convolutions, while the third layer employs a 1{\times}1 convolution to increase the channel count to 64, producing a latent feature map. Subsequently, two stacked dual-path GSU blocks alternately process the encoded features along the frequency and time dimensions. The frequency path uses BiGSU to model cross-frequency dependencies, whereas the time path uses GSU to capture causal temporal dependencies. Each path is followed by a linear projection layer, GroupNorm, and a residual connection. Finally, two independent transposed-convolution decoders incorporate U-Net [ 24 ] skip connections to fuse features from the corresponding encoder layers. The complex decoder applies a tanh activation function to generate DeepFilter [ 25 ] coefficients for filtering the noisy STFT, while the magnitude decoder applies a sigmoid activation function to generate a magnitude mask that is multiplied with the noisy magnitude spectrum. The outputs of the two branches are fused via weighted averaging, and the enhanced speech is reconstructed through the inverse STFT. 2.2 Dual-branch spectral processing GSU-DBNet employs a dual-branch decoding structure that performs parallel modeling of the complex and magnitude spectra, fully exploiting the complementary information provided by these two spectral representations. The complex branch jointly estimates the real and imaginary components of the spectrum, thereby preserving phase information. This branch progressively recovers frequency resolution through a transposed-convolution decoder and fuses features from the corresponding encoder layers via U-Net skip connections. The decoder output is passed through a tanh activation function to generate a complex mask, which serves as DeepFilter [ 25 ] coefficients for filtering the noisy STFT spectrum. The magnitude

Read the complete text