VoiceRestore: Flow-Matching Transformers for Speech Recording Quality Restoration

Stanislav · 2025

Read VoiceRestore: Flow-Matching Transformers for Speech Recording Quality Restoration by Stanislav (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00794.txt.

Introduction

Stanislav Kirdey presents a model that cleans up poor speech recordings, in a paper from 2025. Real clips often mix noise, echo, squeezed sound, and thin range, and past tools tend to fix one flaw at a time.

His model, VoiceRestore, uses flow matching with transformer blocks and learns from made up flawed clips, so it needs no matched clean and flawed pairs. One model handles short clips and long talks alike. Tests in words and in scores showed broad gains across flaw types. The work points to a single tool for archive, call, and field speech alike.

Catalog record

CREATORStanislav
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-00794

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VoiceRestore: Flow-Matching Transformers for Speech Recording Quality Restoration Stanislav Abstract We present VoiceRestore , a novel approach to restoring the quality of speech recordings using flow-matching Transformers trained in a self-supervised manner on synthetic data. Our method tackles a wide range of degradations frequently found in both short and long-form speech recordings, including background noise, reverberation, compression artifacts, and bandwidth limitations—all within a single, unified model. Leveraging conditional flow matching and classifier-free guidance, the model learns to map degraded speech to high-quality recordings without requiring paired clean and degraded datasets. We describe the training process, the conditional flow matching framework, and the model’s architecture. We also demonstrate the model’s generalization to real-world speech restoration tasks, including both short utterances and extended monologues or dialogues. Qualitative and quantitative evaluations show that our approach provides a flexible and effective solution for enhancing the quality of speech recordings across varying lengths and degradation types. Model code and pre-trained checkpoints are available at: https://github.com/skirdey/voicerestore Kirdey keywords speech restoration, self-supervised learning † † email: [email protected] 1 Introduction The quality of speech recordings is critical to numerous applications, from telecommunications and broadcasting to voice assistants and archival preservation. Degradations such as background noise, reverberation, compression artifacts, and bandwidth limitations can significantly impair intelligibility and listener experience. These problems can occur in both short recordings (e.g., voice commands or prompts) and long-form recordings (e.g., audiobooks, lectures, podcasts). Conventional approaches to speech enhancement and restoration often focus on specific types of degradation or short-duration audio clips, limiting their use in real-world scenarios that involve multiple degradations and varying recording lengths. An effective solution must generalize across different degradation types and be equally adept at handling both short and long-form recordings. In this paper, we introduce VoiceRestore , a novel approach to speech recording quality restoration using flow-matching Transformers trained in a self-supervised manner on synthetic data. Our method leverages recent advances in conditional flow matching [ 1 ] and Transformer architectures to create a unified model capable of restoring speech recordings of varying lengths and degradation types within a single framework. Unlike many supervised methods that require paired clean and degraded datasets, our approach generates degradations on the fly during training. This self-supervised strategy exposes the model to a diverse range of synthetic degradations, eliminating the need for labeled datasets. We detail the training process, the conditional flow matching framework, and our network’s architecture. We then explore the model’s generalization to real-world tasks, including both short and long-form recordings suffering from severe degradations such as heavy background noise and compression artifacts. Through qualitative and quantitative evaluations, we show that our approach effectively enhances speech quality across a broad array of degradation scenarios and recording lengths. Our main contributions are : • A self-supervised training method for speech recording quality restoration using conditional flow matching. • A unified architecture that handles multiple speech degradations in both short and long-form recordings. • Detailed implementation specifics, including model architecture and training configurations, to facilitate reproducibility. • A demonstration of the model’s generalizability to real-world degradation scenarios, including challenging short and extended speech samples. By addressing the challenges of speech recording quality restoration in a self-supervised fashion, VoiceRestore offers a promising solution for practical applications involving multiple and unpredictable degradations. 2 Related Work Speech enhancement and restoration have been extensively studied. Traditional methods often rely on signal processing techniques, such as spectral subtraction and Wiener filtering, to reduce noise and improve speech quality. These methods can be effective for specific types of degradation but typically struggle with complex or multiple degradations, and their performance may degrade on longer recordings due to temporal variability. Deep learning approaches have significantly advanced speech enhancement capabilities. Models like SEGAN [ 2 ] and DCCRN [ 3 ] have demonstrated improved performance over traditional methods. However, these models often require paired datasets of clean and degraded speech and may not generalize well to unseen degradations or longer recordings. Recent work has explored generative models for speech enhancement. Diffusion-based models [ 4 ] have shown promise in modeling complex speech distributions, but they can be computationally intensive and are often evaluated on short utterances. Other approaches leverage self-supervised learning for robustness without labeled data; for example, Transformer-based U-Net architectures have been explored in E2TTS [ 5 ] . Flow matching [ 1 ] provides a flexible framework for generative modeling by learning continuous-time dynamics that transform a simple base distribution into a complex target distribution. Conditional flow matching extends this idea to conditional generation tasks, making it suitable for speech restoration, where the model learns a mapping from degraded to clean speech. Transformers [ 6 ] have revolutionized sequence modeling tasks, including speech processing. Their ability to capture long-range dependencies makes them well suited for modeling both short- and long-form speech recordings. Recent works have applied Transformers to various speech recognition and synthesis tasks [ 7 ] , demonstrating their effectiveness in handling complex temporal dynamics. Our work builds on these advances by integrating conditional flow matching with Transformer architectures, enabling effective speech restoration across varying recording lengths without the need for paired datasets. 3 Proposed Method In this section, we detail our approach to speech recording quality restoration using conditional flow matching and Transformer architectures. Our framework is designed to handle both short and long-form recordings within a single model. 3.1 Problem Formulation Let x\in\mathbb{R}^{T\times F} represent a clean speech spectrogram, where T is the number of time frames (which can vary widely to accommodate both short and long recordings) and F is the number of frequency bins. We denote the degraded version as y , obtained by applying various degradations to x . Our goal is to learn a restoration function f_{\theta}(y) that estimates clean speech \hat{x} given the degraded input y . 3.2 Conditional Flow Matching for Speech Restoration 3.2.1 Mathematical Framework We define a continuous-time flow that transports the degraded speech distribution p(y) to the clean speech distribution p(x) over time t\in[0,1] . At each time t , we have an intermediate state x_{t} : x_{t}=(1-\alpha(t))y+\alpha(t)x, (1) where \alpha(t) is a time-dependent interpolation function. The corresponding vector field u_{t}(x_{t}) is: u_{t}(x_{t})=\frac{dx_{t}}{dt}=\dot{\alpha}(t)(x-y). (2) Our neural network v_{\theta}(x_{t},t,y) aims to approximate this vector field by minimizing: \mathcal{L}(\theta)=\mathbb{E}_{t,x,y}\left[\left\lVert u_{t}(x_{t})-v_{\theta}(x_{t},t,y)\right\rVert_{2}^{2}\right]. (3) Degraded Speech y Intermediate State x_{t} Clean Speech x t=0 t=1 Time t\in[0,1] Figure 1 : Conditional Flow Matching Process for Speech Restoration The interpolation function allows the model to smoothly transition between degraded and clean speech over time, modeling the restoration process as a continuous transformation. 3.2.2 Self-Supervised Training with Synthetic Degradations We employ a self-supervised training strategy where synthetic degradations are applied to clean speech recordings on the fly. This approach enables the model to learn from a diverse range of degradation types and combinations without relying on labeled data. By training on both short and long-form recordings, the model learns to handle varying sequence lengths and temporal dynamics. Clean Speech x Synthetic Degradations Degraded Speech y Training of VoiceRestore Figure 2 : Synthetic Degradation Pipeline for Self-Supervised Training 3.2.3 Handling Variable-Length Recordings To accommodate both short and long-form recordings, we design the model to process variable-length sequences efficiently. Techniques such as chunking for long recordings and padding or masking for batch processing of variable-length

Read the complete text