A Survey of Deep Learning for Complex Speech Spectrograms
Yuying Xie, Zheng-Hua Tan · 2025
Read A Survey of Deep Learning for Complex Speech Spectrograms by Yuying Xie, Zheng-Hua Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08694.txt.
Introduction
Yuying Xie and Zheng-Hua Tan wrote a survey in two thousand twenty-five on deep learning for speech shown as complex spectrograms. A spectrogram maps how sound energy varies with time and pitch. The complex form keeps both strength and phase, the timing detail that plain maps throw away.
The survey reviews networks built to handle complex values directly, and the more common practice of feeding complex maps to standard networks. It covers training methods and loss functions suited to this data. It then surveys uses such as recovering phase, cleaning noisy speech, and separating speakers, plus links to generative models. The authors aim to give researchers a single map of a fast-moving field.
Catalog record
| CREATOR | Yuying Xie, Zheng-Hua Tan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08694
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Survey of Deep Learning for Complex Speech Spectrograms Yuying Xie, Zheng-Hua Tan Abstract Recent advancements in deep learning have significantly impacted the field of speech signal processing, particularly in the analysis and manipulation of complex spectrograms. This survey provides a comprehensive overview of the state-of-the-art techniques leveraging deep neural networks for processing complex spectrograms, which encapsulate both magnitude and phase information. We begin by introducing complex spectrograms and their associated features for various speech processing tasks. Next, we examine the key components and architectures of complex-valued neural networks, which are specifically designed to handle complex-valued data and have been applied to complex spectrogram processing. As recent studies have primarily focused on applying real-valued neural networks to complex spectrograms, we revisit these approaches and their architectural designs. We then discuss various training strategies and loss functions tailored for training neural networks to process and model complex spectrograms. The survey further examines key applications, including phase retrieval, speech enhancement, and speaker separation, where deep learning has achieved significant progress by leveraging complex spectrograms or their derived feature representations. Additionally, we examine the intersection of complex spectrograms with generative models. This survey aims to serve as a valuable resource for researchers and practitioners in the field of speech signal processing, deep learning and related fields. Journal: Astronomy \& Computing 1 Introduction Figure 1 : This figure illustrates the overall structure of the survey, centered on the modeling and processing of complex spectrograms. The framework is divided into five main components: (1) complex-valued neural networks specifically designed for complex spectrogram processing; (2) RVNNs applied to complex spectrograms; (3) training strategies and loss functions, encompassing low-latency methods, optimization techniques, and loss designs tailored to complex spectrogram learning; (4) applications, covering speech enhancement, speaker separation, and phase retrieval; and (5) generative models, including approaches based on VAEs, GANs, and diffusion models for complex spectrogram processing. In recent years, deep learning has brought remarkable advancements to speech signal processing. A substantial body of research in this field relies on short-time Fourier transform (STFT) representations, which provide detailed time-frequency analyses of speech signals. The STFT representations are inherently complex-valued and can be represented in either Cartesian or polar coordinate systems, resulting in real-imaginary or magnitude-phase spectrograms, respectively. Among these representations, early studies focused predominantly on the magnitude spectrogram for several key reasons. First, magnitude spectrograms exhibit well-defined structural properties and facilitate effective feature extraction in deep neural network (DNN)-based approaches. Second, magnitude spectrograms have played a significant role in conventional speech signal processing, leading mordern DNN-based approaches to naturally inherit this focus. In contrast, phase spectrograms have received comparatively less attention, primarily due to the challenges associated with their modeling. These difficulties stem from the inherent properties of phase spectrograms, including the lack of well-defined structure, the issue of phase wrapping, and a high sensitivity to temporal shifts in the waveform. Besides, there has been skepticism regarding the significance of phase spectrograms. Early studies, such as Ephraim and Malah (1984) and Wang and Lim (1982) , argued that phase spectrograms are not essential in speech processing. However, subsequent research has led to a reassessment of this view. For example, Paliwal et al. (2011) demonstrated that enhancing phase information can improve perceptual quality. Moreover, Le Roux (2011) showed that combining a magnitude spectrogram with an irrelevant phase yields an unexpected audio waveform. These findings underscore the critical role of phase information and have spurred growing interest in incorporating phase modeling into modern approaches. Moreover, reconstruction of waveforms from estimated magnitude spectrograms commonly relies on classical algorithms such as Griffin–Lim or on neural vocoders, both of which are prone to introducing additional distortion. This further highlights the limitations of magnitude-only approaches and emphasizes the significance of complex spectrogram processing. Meanwhile, time-domain signals, preserving all speech information, have been widely adopted in various studies. The main advantages of processing time-domain signal include bypassing the STFT, thereby avoiding explicit phase spectrogram modeling and enabling low-latency processing. Although representative time-domain models like Conv-TasNet ( Luo and Mesgarani (2019) ) have shown promising performance in speaker separation, recent studies ( Zhang et al. (2024b) ; Wang et al. (2023a) ) indicate that time-frequency domain models can achieve higher performance. Furthermore, Wang et al. (2023b) observed that although Conv-TasNet employs an encoder to generate “pseudo-STFT” features, these features are not narrowband, further limiting Conv-TasNet compatibility with conventional STFT-based enhancement methods (e.g., beamforming, weighted prediction error) that rely on narrowband assumptions. In contrast, complex spectrogram–based models integrate seamlessly with such techniques, making them suitable for applications like multi-channel speech enhancement. Moreover, while complex spectrogram–based methods are limited by the time–frequency resolution trade-off of the STFT and typically incur longer delays than time-domain approaches, recent studies have substantially reduced their latency. The latest models achieve delays as low as 4 ms, enabling their deployment in real-time applications ( Wang et al. (2023b) ; Wu and Braun (2025) ). Increasingly, research has focused on DNN-based methods for complex spectrogram processing, implemented with real-valued (RVNNs) or complex-valued neural networks (CVNNs). From the perspective of deep learning development, RVNNs remain the mainstream and have long been widely applied, while CVNNs has also garnered increasing attention in recent years. Research on CVNNs can date back to Widrow et al. (1975) . Since then, CVNNs have been applied in diverse areas such as audio processing, image denoising ( Quan et al. (2021) ; Rawat et al. (2021) ), radar signal processing ( Yu et al. (2019) ; Fuchs et al. (2021) ), and object discovery ( Löwe et al. (2022) ). Several studies have highlighted theoretical advantages of CVNNs ( Tan et al. (2022) ; Wu et al. (2024) ; Hirose (2011) ; Voigtlaender (2023) ; Nitta (2002) ; Nitta (2013) ). Nevertheless, despite established theoretical benefits, the practical superiority of CVNNs over RVNNs remains inconclusive. Recent studies in speech signal processing are mostly focusing on using RVNNs to handle complex spectrogram processing. This may be because RVNNs have lower computational cost while achieving comparable performance ( Wu et al. (2023) ). Using deep learning for complex spectrogram processing has yielded promising results across various related fields closely linked to perceptual quality, including phase retrieval, speech enhancement, and speaker separation. Progress has been made in architectural design, training strategies, and loss function development throughout the exploration. Moreover, recent studies have increasingly focused on low-latency processing, universal models, further performance improvements, and lower computational cost, among others. There is also growing interest in leveraging generative models for complex spectrogram processing. However, despite these advances, the field remains fragmented and a systematic survey is still lacking. This survey aims to fill the gap and provide a comprehensive summary of DNN-based approaches for complex spectrogram processing, serving as a reference for both future research and practical applications. This work begins by introducing the complex spectrograms and their associated features. It then provides a summary of methods for processing complex spectrograms using CVNNs, primarily from the following perspectives: (1) Research on CVNNs is ongoing, and our understanding of their properties and capabilities continues to evolve, and (2) complex-valued operations possess inherent characteristics, such as the ability to represent data in both Cartesian and polar coordinates, and to simultaneously perform scaling and rotation. Subsequently, this review