Modeling strategies for speech enhancement in the latent space of a neural audio codec

Sofiene Kammoun, Xavier Alameda-Pineda, Simon Leglaive · 2025

Read Modeling strategies for speech enhancement in the latent space of a neural audio codec by Sofiene Kammoun, Xavier Alameda-Pineda, Simon Leglaive (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26299.txt.

Introduction

Sofiene Kammoun, Xavier Alameda-Pineda, and Simon Leglaive compare ways to train speech enhancement models inside the latent space of a neural audio codec. A codec compresses speech into a compact latent form, either continuous vectors or discrete tokens. Speech enhancement aims to clean a noisy recording, and working in the codec latent space may be more efficient than working on raw waveforms.

Kammoun and the team test autoregressive and non autoregressive models based on the Conformer architecture, plus a baseline that fine tunes the codec encoder itself. Predicting continuous latents consistently beat discrete token prediction. Autoregressive models gave higher quality but lower intelligibility and efficiency, while encoder fine tuning gave the strongest enhancement scores at the cost of worse codec reconstruction.

Catalog record

CREATORSofiene Kammoun, Xavier Alameda-Pineda, Simon Leglaive
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26299

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modeling strategies for speech enhancement in the latent space of a neural audio codec Sofiene Kammoun, Xavier Alameda-Pineda, Simon Leglaive Abstract Neural audio codecs (NACs) provide compact latent speech representations in the form of sequences of continuous vectors or discrete tokens. In this work, we investigate how these two types of speech representations compare when used as training targets for supervised speech enhancement. We consider both autoregressive and non-autoregressive speech enhancement models based on the Conformer architecture, as well as a simple baseline where the NAC encoder is simply fine-tuned for speech enhancement. Our experiments reveal three key findings: predicting continuous latent representations consistently outperforms discrete token prediction; autoregressive models achieve higher quality but at the expense of intelligibility and efficiency, making non-autoregressive models more attractive in practice; and encoder fine-tuning yields the strongest enhancement metrics overall, though at the cost of degraded codec reconstruction. The code and audio samples are available online. 1 1 1 sofienekammoun.github.io/SE-NAC-25/ † † address: ⋆ CentraleSupélec, IETR (UMR CNRS 6164), France † Inria at Univ. Grenoble Alpes, CNRS, LJK, France 1 Introduction Speech enhancement (SE) addresses the problem of estimating a clean speech signal from a degraded recording affected by additive noise, reverberation, or other distortions. Although the target output is often the clean waveform (for listening or downstream tasks), the enhancement process can be defined in different representation domains. The choice of representation (waveform, time–frequency, continuous latent vectors, or discrete tokens) strongly conditions model design, inference speed, and the quality and intelligibility of the restored signal. Classic SE algorithms worked predominantly in the short-time Fourier transform (STFT) domain, exploiting structure in the time–frequency plane and statistical assumptions about speech and noise [ 1 , 2 , 3 ] . Early deep learning approaches to SE followed the established time–frequency pipeline by estimating time-frequency masks or magnitudes using discriminative training of neural networks [ 4 , 5 ] . Later, time-domain architectures such as Conv-TasNet [ 6 ] showed that learned encoder–separator–decoder pipelines with 1D temporal convolutions can outperform oracle time-frequency masking in speech separation, which removes the need for explicit phase estimation by using trainable filterbanks and time-domain loss functions. The effectiveness of this time-domain approach was also confirmed for SE [ 7 ] . In parallel, neural audio codecs (NACs), initially introduced in [ 8 ] , are being adopted across speech processing frameworks, where they provide an alternative representation space for speech signals [ 9 ] . NACs compress audio through an encoder-decoder framework into compact latent sequences represented as continuous vectors or as discrete tokens produced by residual vector quantization (RVQ) [ 10 ] . The appeal of these representations for downstream tasks is due to two key advantages. First, they offer a very compressed representation of the data, potentially reducing memory and compute for sequence models. Second, the discrete-token view makes it straightforward to reuse modeling techniques from natural language processing (NLP). Recent audio generative models such as [ 11 ] process audio signals as sequences of discrete tokens, using RVQ-aware and often autoregressive (AR) transformer-based architectures [ 12 ] . Inspired by this success, recent works have developed SE methods that generate clean speech tokens conditioned on noisy speech tokens [ 13 , 14 , 15 ] . While transformers are often assumed to require discrete tokens as input, their first layer is an embedding layer that produces continuous representations, making it feasible to directly use the continuous latent representation of the NAC without modifying the rest of the model. Recent works have started exploring the use of transformer-based architectures using continuous representations for image generation [ 16 ] , speech synthesis [ 17 ] , speech separation [ 18 ] , and SE [ 19 , 20 ] . Moreover, in general audio generation tasks, AR models are appropriate when the output length or alignment between input and output is unknown, which is the case for tasks like text-to-speech (TTS) [ 11 ] , prompt-based music [ 21 ] , and image [ 22 ] generation. However, in SE, the input and output signals (noisy and clean speech) are of equal length and perfectly aligned, with no absolute need for diverse predictions. This reduces the intrinsic need for AR modeling. Still, AR models offer the advantage of modeling intra-sequence temporal dependencies. In contrast, non-AR models generate the entire sequence simultaneously, relying only on input-output dependencies. Recent concurrent studies have also begun to explore this trade-off for SE [ 23 ] , [ 24 ] , offering complementary insights. Finally, SE and codecs are two essential components of voice communication systems, making the study of their interaction practically relevant. A natural approach to SE using NACs is to fine-tune the NAC encoder to produce a clean speech latent representation directly from the noisy input. However, this may compromise the NAC’s reconstruction performance. To our knowledge, this simple strategy has only received very little attention in prior work [ 25 ] . In this work, we focus on three central questions regarding SE using NACs: (i) whether to model discrete tokens or continuous vectors in the NAC’s latent space, (ii) how AR and non-AR models compare, and (iii) how the fine-tuning of the NAC’s encoder compares to adding a separate SE model. Our goal is not to optimize any single model for peak performance but to provide a fair comparative study of these signal representation spaces and modeling choices. 2 Method We frame our study of SE in NAC latent spaces along three modeling axes introduced above. To structure the discussion, we design a family of models that systematically vary along these axes while keeping other components comparable. All methods share the following common structure. A pretrained NAC maps noisy and clean waveforms into latent representations, either continuous embeddings or discrete tokens. Enhancement models then learn a conditional distribution of clean latents given noisy latents. Depending on the variant, the predictor may operate autoregressively over time or non-autoregressively, targeting either continuous vectors or discrete tokens. Finally, the NAC decoder reconstructs the waveform from the predicted latent sequence. We also compare these models to a simple fine-tuning of the NAC encoder. In the following, we denote by \theta the model parameters for each SE method. All models are trained in a supervised manner by minimizing the negative log-likelihood. 2.1 Neural audio codecs NACs are encoder-decoder architectures built with 1D convolutional networks, but they further introduce RVQ of the latent representation extracted by the encoder [ 8 ] . Let \mathbf{y}_{\text{wav}} and \mathbf{x}_{\text{wav}}\in\mathbb{R}^{d\cdot f_{s}} denote the noisy and clean speech waveforms of duration d in seconds and sampling rate f_{s} in Hertz, assuming d\cdot f_{s}\in\mathbb{N} without loss of generality. A NAC can be described with the following operations: \displaystyle\text{Tokenize: } \displaystyle\begin{aligned} \mathbf{x}_{\text{wav}}\\ \mathbf{y}_{\text{wav}}\end{aligned}\overset{\displaystyle\mathcal{E}}{\longrightarrow}\begin{aligned} \bar{\mathbf{x}}\\ \bar{\mathbf{y}}\end{aligned}\in\mathbb{R}^{L\times T}\overset{\displaystyle\mathcal{Q}_{\mathcal{C}}}{\longrightarrow}\begin{aligned} \mathbf{x}\\ \mathbf{y}\end{aligned}\in\{1,...,K\}^{N\times T}, (1) \displaystyle\text{Detokenize: } \displaystyle\begin{aligned} \mathbf{x}\\ \mathbf{y}\end{aligned}\overset{\displaystyle\mathcal{Q}_{\mathcal{C}}^{-1}}{\longrightarrow}\begin{aligned} \hat{\bar{\mathbf{x}}}\\ \hat{\bar{\mathbf{y}}}\end{aligned}\in\mathbb{R}^{L\times T}\overset{\displaystyle\mathcal{D}}{\longrightarrow}\begin{aligned} \hat{\mathbf{x}}_{\text{wav}}\\ \hat{\mathbf{y}}_{\text{wav}}\end{aligned}\in\mathbb{R}^{d\cdot f_{s}}, where \mathcal{E} is the encoder that encodes the waveform into an L -dimensional downsampled continuous representation of length T , \mathcal{D} is the decoder that performs the reverse operation, and \mathcal{Q}_{\mathcal{C}} and \mathcal{Q}_{\mathcal{C}}^{-1} denote the direct and inverse RVQ modules, which depend on a set of codebooks \mathcal{C}=\{\mathcal{C}_{n}\in\mathbb{R}^{L\times K}\}_{n=1}^{N} , with K codebook vectors of dimension L for each quantization

Read the complete text