SEMamba++: A General Speech Restoration Framework Leveraging Global, Local, and Periodic Spectral Patterns
Yongjoon, Jung-Woo · 2026
Read SEMamba++: A General Speech Restoration Framework Leveraging Global, Local, and Periodic Spectral Patterns by Yongjoon, Jung-Woo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11669.txt.
Introduction
Lee and Choi present SEMamba++, a general speech restoration framework tuned to the structure of speech itself. State-space models like SEMamba lead on denoising, yet they miss traits such as spectral periodicity. Their Global, Local, and Periodic module reads frequency bins in a way that uses those traits as built-in biases. A multi-resolution block then processes time and frequency in parallel, catching varied spectral patterns. A learnable mapping adds a final lift. Across baselines, the system posts the best scores while staying light on compute.
Catalog record
| CREATOR | Yongjoon, Jung-Woo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61562 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11669
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SEMamba++: A General Speech Restoration Framework Leveraging Global, Local, and Periodic Spectral Patterns Yongjoon, Jung-Woo Abstract General speech restoration demands techniques that can interpret complex speech structures under various distortions. While State-Space Models like SEMamba have advanced the state-of-the-art in speech denoising, they are not inherently optimized for critical speech characteristics, such as spectral periodicity or multi-resolution frequency analysis. In this work, we introduce an architecture tailored to incorporate speech-specific features as inductive biases. In particular, we propose Frequency GLP, a frequency feature extraction block that effectively and efficiently leverages the properties of frequency bins. Then, we design a multi-resolution parallel time-frequency dual-processing block to capture diverse spectral patterns, and a learnable mapping to further enhance model performance. With all our ideas combined, the proposed SEMamba++ achieves the best performance among multiple baseline models while remaining computationally efficient. Lee Choi keywords General Speech Restoration, GAN † † address: 1 Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea † † email: [email protected], [email protected] 1 Introduction General Speech restoration (GSR) [ 29 ] refers to the task of recovering high-quality speech from signals affected by a range of degradations, such as noise, reverberation, bandwidth limitation, and clipping. This problem is particularly relevant in real-world environments, where speech is often captured under adverse acoustic conditions or with limited recording equipment, resulting in multiple and overlapping types of distortion. Unlike speech denoising or dereverberation that aims to predict clean speech by removing noise or reverberation, GSR not only predicts clean speech but also generates the missing speech fragment so that the resulting speech is perceptually natural. This fragment may be high-frequency bands in bandwidth-limited situations or a high-amplitude signal in clipped scenarios. This hybrid nature has led to various approaches to solving GSR. Generative methods have been used for their high perceptual quality and versatility in generating missing speech fragments. Universe [ 46 ] introduced score-based diffusion [ 47 ] to solve GSR, on which Universe++ [ 44 ] improved using a hybrid of score-matching [ 47 ] and generative adversarial networks (GAN) [ 9 ] . ANYENHANCE [ 56 ] is a multi-task masked generative model (MGM) [ 26 ] utilizing prompt-guidance and self-critic mechanisms to successfully solve voice enhancement. In generative methods, language models (LMs) have also emerged as a viable option due to their generalization capacity. MaskSR [ 27 ] is a masked language model (MLM) capable of restoring full-band speech with discrete acoustic tokens from pretrained neural audio codecs. LLaSE-G1 [ 16 ] addresses multi-task GSR utilizing a language model. However, LM-based methods require substantial amounts of training data. Surprisingly, discriminative methods, although thought to be less perceptually natural and versatile than generative methods, yield impressive results in GSR. Discriminative methods [ 48 , 5 ] achieved high performance in GSR challenges such as URGENT [ 58 ] and AATC [ 57 ] . This success can be attributed to architectural improvements and advanced training techniques (i.e., loss functions) inherited from speech denoising. CMGAN [ 3 ] introduced a Conformer [ 12 ] -based encoder-decoder architecture with dilated DenseNets [ 34 ] . CMGAN used metric discriminator [ 7 ] with reconstruction losses to directly optimize perceptual metrics. MP-SENet extended this framework by introducing parallel magnitude and phase decoding. More recently, SEMamba [ 4 ] replaced the Conformer with Mamba [ 10 ] , leveraging its selective state-space mechanism. For the aforementioned methods, separate feature extraction for time and frequency bins has been a key design; a procedure referred to as time-frequency dual-path (TFDP) processing in this work. Unlike the image domain, where the height and width represent the same characteristics at different locations, the time and frequency bins in a speech spectrum exhibit heterogeneous properties. To this, feature extraction across time and frequency bins has been implemented separately. Although time and frequency features are processed separately, both are often processed with modules of the same architecture [ 3 , 4 , 19 ] . To better reflect the distinction between time and frequency features, the design should be tailored to address a specific domain. In particular, frequency feature extraction modules need to capture global, local, and periodic patterns of the speech spectrum effectively. Conformer [ 3 ] and SpatialNet [ 38 ] -style frequency feature extraction have been popular choices for extracting global and local features by jointly using full-band (global) modules and sub-band (local) modules. However, this frequency feature mixing may not be optimal for GSR due to the lack of local-global selectivity and the spectral periodicity modeling capacity. First, the serially connected local and global modules reduce selectivity: the model’s ability to prioritize either local or global frequency representations depending on the degradation characteristics. This selectivity between local and global representations is critical, as degraded speech in GSR exhibits distinct characteristics across different degradation kernels. For example, the global branch should be prioritized for bandwidth extension, where the distortion is caused by the truncation of a wide spectral region. Moreover, the periodicity in frequency has also been underexplored, despite the importance of near-harmonic structures from vocal excitations. In this regard, a means to focus on global, local, and periodic feature modeling is required. Aside from the frequency feature extraction methods, the single-resolution TFDP processing may not be optimal in GSR. TFDP is often implemented at a single resolution—without aggressive down- or upsampling—to preserve fine details and minimize upsampling artifacts [ 36 ] . Since this design entails a high number of TF bins, the models are necessarily equipped with modules capable of modeling long sequences (or many bins) [ 3 , 4 , 23 , 19 ] . Single-resolution processing faces two limitations. First, scaling model capacity via deeper or wider architectures incurs excessive computational overhead due to the long sequence length. A recent work [ 48 ] resolves this problem by utilizing progressive block expansion strategies. This approach enables efficient training of high-capacity TFDP models, but the training strategy is complex and does not alleviate the heavy computations during inference, which is critical in resource-restrained settings. Second, single-resolution processing misses opportunities for multi-scale feature extraction. For instance, in bandwidth-limited signals, information from low-frequency bins can be better propagated to high-frequency bins when downsampling is applied along the frequency axis. A recent study [ 18 ] uses multi-resolution sequential TFDP processing to resolve this issue, though the sequential TFDP processing does not fully exploit the diverse speech spectral patterns since TFDP-processed output from #1 affects TFDP processing at #2 and #3 in Figure 1 . Figure 1: Various TFDP processing methods. ds and us denote downsampling and upsampling, respectively. To resolve the aforementioned issues, we first propose Frequency GLP (Global, Local, and Periodic), a novel frequency-processing block that employs a parallel connection of the global periodicity (GP) module and the local (L) module. The GP module comprises a series of Fourier Analysis Network (FAN) [ 6 ] applied directly to the frequency bins, whereas the L module comprises a series of convolutional blocks. The method effectively leverages the intrinsic properties of the speech spectrum, achieving the best performance over a range of advanced baselines. Note that FAN has been applied in GSR [ 48 ] to replace the Multi-Layer Perceptron (MLP), but the direct application