HiFi-Stream: Streaming Speech Enhancement with Generative Adversarial Networks

Ekaterina Dmitrieva, Maksim Kaledin · 2025

Read HiFi-Stream: Streaming Speech Enhancement with Generative Adversarial Networks by Ekaterina Dmitrieva, Maksim Kaledin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17141.txt.

Introduction

Ekaterina Dmitrieva and Maksim Kaledin shrink a speech cleaning model for live use on small devices. Their twenty twenty-five paper presents HiFi-Stream, a lean take on the HiFi++ generative model. Speech enhancement is core to phones and voice apps, but top models eat compute.

The new model keeps most of the sound quality of its big parent. Its size and compute load are far lower, among the smallest and fastest around. Tests ran in a streaming setup, where sound must be cleaned as it arrives, not after. It beat modern rival baselines in that mode. The work suits hearing aids, calls, and voice control on weak hardware.

Catalog record

CREATOREkaterina Dmitrieva, Maksim Kaledin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17141

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HiFi-Stream: Streaming Speech Enhancement with Generative Adversarial Networks Ekaterina Dmitrieva, Maksim Kaledin Abstract Speech Enhancement techniques have become core technologies in mobile devices and voice software simplifying downstream speech tasks. Still, modern Deep Learning (DL) solutions often require high amount of computational resources what makes their usage on low-resource devices challenging. We present HiFi-Stream, an optimized version of recently published HiFi++ model. Our experiments demonstrate that HiFiStream saves most of the qualities of the original model despite its size and computational complexity: the lightest version has only around 490k parameters which is 3.5x reduction in comparison to the original HiFi++ making it one of the smallest and fastest models available. The model is evaluated in streaming setting where it demonstrates its superior performance in comparison to modern baselines. I Introduction Speech enhancement (SE) techniques are generally used as a preprocessing tool for speech downstream tasks such as speech recognition. The main goal of speech enhancement is to remove noise and reverberations coming from the actual environment where the audio is recorded. Mobile and wearable smart devices such as smartphones and smartwatches naturally face strongly corrupted speech as an audio input. Studies [ 1 ] , [ 2 ] show that the quality of automatic speech recognition (ASR) solutions benefits from using an additional speech enhancement model making it an important part of speech processing pipeline. In the area of SE, the state-of-the-art Deep Learning (DL) approaches [ 3 ] , [ 4 ] , [ 5 ] demonstrated that DL can manage very hard audio conditions in speech enhancement and source separation problems. Nonetheless, the line of research in these models was mainly achieving quality at all costs resulting in dozens of millions parameters. On the other hand, such models require powerful computational resources which are unlikely to be placed in a mobile or wearable device. To get the scale, DEMUCS [ 3 ] hit the relevant realtime constraints as a result of significant optimization efforts. The architecture has around 60 M parameters and requires 38.1 GMACs (Multiply–ACcumulate operations) per second of audio in 16 kHz [ 6 ] , [ 7 ] . The problem of constructing architectures for various speech processing tasks available on low-resource devices has become the focus of research efforts in recent years [ 8 ] . Generative Adversarial Networks (GANs) quickly showed their superior quality in text-to-speech problems [ 9 ] , but still the right-away application of GANs for SE problems [ 10 ] , [ 11 ] , [ 12 ] resulted in very computationally expensive models. Recently [ 6 ] it was shown that the number of parameters used in GANs for SE can be reduced more than 10 times suggesting that with more careful design there is an opportunity to propose even more efficient solution with almost the same performance on the benchmarks. In our work we propose HiFi-Stream, an optimized version of HiFi++ [ 6 ] introducing structural and technical modifications. The experimental set up, pipeline and model configurations are given in the github repository: https://github.com/KVDmitrieva/source_sep_hifi . 1. We show that the model can be reduced almost 3.5 times in parameters and around 30% in MACs making it one of the available quality-complexity trade-offs. 2. The experiments on VCTK [ 13 ] dataset in offline setting demonstrate that the proposed model does not lose much in performance in comparison to the original model. 3. We also investigate the performance of HiFiStream in an audio stream simulation where the audio is given in chunks. HiFi-Stream demonstrates superior quality to its direct competitors : HiFi++ [ 6 ] and FFC-SE [ 7 ] ; both in performance and computational complexity without additional streaming training. Fig. 1: HiFi++ and Hifi-Stream architectures: the new model reduced the number of parameters, got new MaskNet block, redesigned HiFi upsampler and removed SpectralUNet II Related Work Speech enhancement and speech separation methods generally can be classified into two branches of research. Time-frequency (TF) methods are based on the usage of the spectrograms produced by the Short-Time Fourier Transform (STFT). Models take a spectrogram as input and outputs the cleaned spectrogram which is transformed into a cleaned waveform via inverse STFT (iSTFT). The TF methods are represented in particular by [ 14 , 4 , 1 ] .Time-domain (TD) methods got their popularity just in recent years [ 3 , 15 , 16 , 5 , 17 ] . Most of these models by construction require millions of parameters, the most efficient of these are DPRNN models with around 2-6 M parameters [ 15 ] . For these reason in our research we focus on TF-methods. Streaming processing presents additional constraints [ 8 ] . The first is computational complexity reflected by the number of parameters and processing time. The second is causality: the model is restricted to use only limited context. So, the researchers are focused on constructing streaming-aware versions of state-of-the-art solutions some of which include ReSepformer [ 18 ] and SkiM [ 19 ] . It was recently shown that regression transformer-based solutions can be implemented in low-resource setting but with considerable efforts invested into the attention mechanisms [ 8 ] . The generative look at the problem presented by SEGAN [ 10 ] later developed into HiFi-GAN solutions [ 11 ] and [ 12 ] . It was suggested to train and design the model as GAN where the generator predicts the clean waveform. This was reviewed in HiFi++ [ 6 ] with more structured design. The model achieved the baseline results despite significantly reduced model complexity. Several other solutions of the same scale were proposed later [ 20 , 7 , 21 ] . In [ 6 ] , [ 7 ] the main goal was to reduce the model size but computational complexity was not addressed. Our aim is to improve also the computational complexity of the SE, and we improved the results of [ 6 , 7 ] in this regard, see Fig. 3 . III Methodology The problem of speech enhancement is to get an estimate \hat{x}\in\mathbb{R}^{N_{ch}\times T} of the clean speech signal x\in\mathbb{R}^{N_{ch}\times T} with N_{ch} channels from its corrupted representation y=f\star x+\varepsilon, where f\in\mathbb{R}^{N_{ch}\times k} represents an impulse response kernel (the effects of the environment acoustics, reverberation and recording device) which is convolved with the clean signal and \varepsilon\in\mathbb{R}^{N_{ch}\times T} is an additive noise component. In our work we consider a monaural problem, i.e. the signal has only one channel. Originally GAN approaches mostly [ 10 ] , [ 11 ] , [ 12 ] used time-domain approach requiring direct signal reconstruction. It turned out that such approaches typically have large number of parameters: one needs a sophisticated enough encoder and decoder to obtain a representative set of features from raw audio. So, HiFi++ [ 6 ] and our work in this aspect take a route of TF-methods rising from original vocoding applications [ 9 ] . Classic TF approaches usually propose to estimate a magnitude mask M(y)\in\mathbb{R}^{F\times M} with values from 0 to 1 for signal spectrogram s=STFT(y)\in\mathbb{C}^{F\times M} and then estimate the clean signal as \hat{x}=iSTFT(M(y)\odot s). In such a way, the phase component is bypassed to have a well-defined inverse STFT. On a high

Read the complete text