A High-Fidelity Speech Super Resolution Network using a Complex Global Attention Module with Spectro-Temporal Loss
Tarikul Islam, Biraj, Rida, Rashedul, Taieba, Nursad, Anomadarshi · 2025
Read A High-Fidelity Speech Super Resolution Network using a Complex Global Attention Module with Spectro-Temporal Loss by Tarikul Islam, Biraj, Rida, Rashedul, Taieba, Nursad, Anomadarshi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00229.txt.
Introduction
Tarikul Islam Tamiti and co-authors published a research paper in 2025 on a network that upgrades poor-quality speech recordings to high fidelity, restoring both loudness patterns and phase. Phase carries much of what makes speech sound natural, yet many enhancers ignore it. Their system works directly in the complex time-frequency domain.
A global attention block tracks relations across sounds and frequencies, while a conformer captures local and long-range structure. Training mixes time-domain and multi-resolution spectral losses. On a standard voice data set it beat four leading systems, even at the extreme jump from telephone-grade 2 kilohertz audio to full 48 kilohertz, rebuilding high frequencies without noisy artifacts. The authors target archives, calls, and media restoration.
Catalog record
| CREATOR | Tarikul Islam, Biraj, Rida, Rashedul, Taieba, Nursad, Anomadarshi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00229
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A High-Fidelity Speech Super Resolution Network using a Complex Global Attention Module with Spectro-Temporal Loss Tarikul Islam, Biraj, Rida, Rashedul, Taieba, Nursad, Anomadarshi Abstract Speech super-resolution (SSR) enhances low-resolution speech by increasing the sampling rate. While most SSR methods focus on magnitude reconstruction, recent research highlights the importance of phase reconstruction for improved perceptual quality. Therefore, we introduce CTFT-Net, a Complex Time-Frequency Transformation Network that reconstructs both magnitude and phase in complex domains for improved SSR tasks. It incorporates a complex global attention block to model inter-phoneme and inter-frequency dependencies and a complex conformer to capture long-range and local features, improving frequency reconstruction and noise robustness. CTFT-Net employs time-domain and multi-resolution frequency-domain loss functions for better generalization. Experiments show CTFT-Net outperforms state-of-the-art models (NU-Wave, WSRGlow, NVSR, AERO) on the VCTK dataset, particularly for extreme upsampling (2 kHz to 48 kHz), reconstructing high frequencies effectively without noisy artifacts. Tamiti Joshi Hasan Hasan Athay Mamun Barua George Mason UniversityUSA Chittagong University of Engineering & TechnologyBangladesh keywords Speech Super-resolution, Complex-valued Network, Complex Global Attention † † email: [email protected] 1 Introduction Speech super-resolution (SSR), also known as bandwidth extension (BWE) [ 1 ] , generates missing high frequencies from low-frequency speech contents to improve speech clarity and naturalness. Therefore, SSR is making its way into different practical applications, where speech quality enhancement [ 2 ] and text-to-speech synthesis [ 3 ] are required. Recently, deep neural networks (DNNs) became the state-of-the-art (SOTA) solutions for SSR, that operate on raw waveforms in time domains [ 4 , 5 , 1 , 6 ] or in full spectral domains [ 7 , 8 , 9 , 10 , 11 ] . Both domains have certain advantages and disadvantages. Time-domain methods don’t need phase prediction but cannot leverage the known auditory patterns from a time-frequency (T-F) spectrogram. Moreover, the length of raw waveforms, especially at high-resolution (HR), is extremely long, hence its modeling is computationally expensive in time-domains. In contrast, spectral methods cannot predict phase and hence, need a vocoder to generate audio from real-valued spectrograms. To solve the problems that exist in both domains, we propose the Complex Time-Frequency Transformation Network (CTFT-Net), which receives complex-valued T-F spectrograms at its input and generates complex-valued T-F spectrograms, subsequently converted to raw waveform at its output. Moreover, motivated by the fact that phase plays a crucial role in speech enhancement [ 12 ] , CTFT-Net adopts joint reconstruction of frequencies and phases from complex T-F spectrograms, providing better results for SSR tasks. We show that our proposed CTFT-Net, a U-Net style model, provides BWE from the lowest 2 kHz input resolution to 48 kHz target resolution (i.e., upsampling ratio 24) by outperforming SOTA models [ 13 , 14 , 15 , 16 , 17 ] in terms of log spectral distance (LSD) without causing artifacts at the verge between existing and generated frequency bands. Moreover, the proposed model’s ability to joint estimation of complex phases and frequencies resolves the following three common issues: our model (i) does not need to utilize the unprocessed phase from the input speech [ 11 ] while reconstructing speech in time-domain, (ii) does not need to reuse the low-frequency bands of the input via concatenation [ 15 , 18 ] at post-processing, and (iii) does not need to flip the existing low-resolution (LR) phase [ 10 ] to reconstruct in time-domain. This paper designs a dual-path attention block in the full complex domain to capture long-range correlations along both the time and frequency axes, referred to as the complex global attention block (CGAB). The CGAB parallelly pays attention to inter-phoneme and inter-frequency dependencies in both time and frequency axes of a complex-valued spectrogram to effectively reconstruct the missing high frequencies and phases. Therefore, CTFT-Net can be termed as a cross-domain framework, which directly uses time, phase, and frequency domain metrics to supervise the network learning. Moreover, a complex-valued conformer is integrated into the bottleneck layer of our CTFT-Net to enhance its capability to provide local and global attention among consecutive spectrograms. Figure 1 : CTFT-Net has complex encoders, decoders, complex skip blocks, CGAB, and real MR-STFT + SI-SDR loss. We combine the scale-invariant signal-to-distortion ratio (SI-SDR) [ 19 ] loss with real-valued multiresolution short-time Fourier transform (STFT) loss [ 20 ] for joint optimization in both time and frequency domains. We show that this combination provides better results for SSR in complex domains. Experimental results show that CTFT-Net outperforms the SOTA baselines, such as NU-Wave, WSRGlow, NVSR, and AERO on LSD for the VCTK multispeaker dataset. Notably, CTFT-Net 1 1 1 Source code of the model will be available after acceptance. performs better for extremely LR speech signals, such as upsampling from a minimum of 2 kHz to 48 kHz. In a nutshell, the technical contributions of our work are: • We propose a cross-domain SSR framework that operates entirely in complex domains, jointly reconstructing both magnitude and phase from the LR speech signal. • We propose CGAB - a dual-path end-to-end attention block in encoders and use conformers in bottleneck layers in the full complex domain to capture the long-range correlations along both the time and frequency axes. • We integrate SI-SDR loss in time-domain with multi-resolution STFT loss in the frequency domain to capture fine-grained and coarse-grained T-F spectral details. • We perform a comprehensive ablation study and evaluate the proposed model on the VCTK multispeaker dataset. Results show that CTFT-Net outperforms the SOTA SSR models. 2 Methodology Here, we discuss our proposed modifications on U-Net that construct complex-valued CTFT-Net for SSR tasks. 2.1 Proposed network architecture in complex-domain The detailed architecture of the proposed CTFT-Net is shown in Fig. 1 . The network consists of four main components: (i) a total of 16 (i.e., 8 + 8) full complex-valued encoder-decoder blocks, (ii) complex-valued skip blocks, (iii) complex-valued conformer in the bottleneck layer, and (iv) complex-valued global attention blocks - CGAB. The complex domain processing by our proposed CTFT-Net has the potential to adopt best practices from two different domains that are explained below. Reasoning behind complex-domain models: Existing SSR methods can be classified broadly into two domains: (i) spectral methods where real-valued T-F spectrograms are provided at model’s input, and (ii) time-domain methods where raw waveforms are provided at model’s input. Spectral methods cannot predict phase and hence, need a vocoder to generate audio from the bandwidth-extended spectrograms. Moreover, spectral methods typically use mean square error (MSE) loss, and cannot directly use time-domain loss functions, such as SI-SDR loss to improve the speech quality while performing BWE. In contrast, time-domain methods avoid phase prediction problems and can include SDR-type loss function but cannot leverage the known auditory patterns from a T-F spectrogram. Our proposed CTFT-Net handles both frequencies and phases simultaneously by receiving complex T-F spectrograms at its input and generates raw waveform at its output without any vocoders. Therefore, it is typically free from the problems that both spectral and time domain methods have and can deliver superior SSR compared to the SOTA models. 2.2 Complex encoders and decoders Each encoder/decoder block is built upon complex-valued convolution to ensure successive extraction and reconstruction of both magnitude and phase from the complex T-F spectrogram. Complex convolution is the key difference between