Token-based Audio Inpainting via Discrete Diffusion

Tali Dror ∗ , Iftach Shoham ∗ , Moshe Buchris, Oren Gal, Haim Permuter, Gilad Katz, Eliya Nachmani · 2025

Read Token-based Audio Inpainting via Discrete Diffusion by Tali Dror ∗ , Iftach Shoham ∗ , Moshe Buchris, Oren Gal, Haim Permuter, Gilad Katz, Eliya Nachmani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08333.txt.

Introduction

Tali Dror and six co-authors present a 2025 paper on filling gaps in damaged music recordings, a task called audio inpainting. Earlier diffusion-based tools struggle when the missing stretch runs long. The team instead works with tokens, compact codes for sound produced by a trained audio tokenizer. Their method runs a discrete diffusion process over those tokens, which keeps long repairs stable and musically coherent.

Two training ideas support the method. One loss term rewards smooth change over time, and a structured corruption scheme masks whole spans during training so the model learns to rebuild them. On two music data sets, with gaps up to three quarters of a second, the method beat strong rivals for gaps of moderate length and up. The authors present it as a step toward better music restoration tools.

Catalog record

CREATORTali Dror ∗ , Iftach Shoham ∗ , Moshe Buchris, Oren Gal, Haim Permuter, Gilad Katz, Eliya Nachmani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08333

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Token-based Audio Inpainting via Discrete Diffusion Tali Dror ∗ , Iftach Shoham ∗ , Moshe Buchris, Oren Gal, Haim Permuter, Gilad Katz, Eliya Nachmani Abstract Audio inpainting refers to the task of reconstructing missing segments in corrupted audio recordings. While prior approaches—including waveform and spectrogram-based diffusion models—have shown promising results for short gaps, they often degrade in quality when gaps exceed 100 milliseconds (ms). In this work, we introduce a novel inpainting method based on discrete diffusion modeling, which operates over tokenized audio representations produced by a pre-trained audio tokenizer. Our approach models the generative process directly in the discrete latent space, enabling stable and semantically coherent reconstruction of missing audio. We evaluate the method on the MusicNet dataset using both objective and perceptual metrics across gap durations up to 300 ms. We further evaluated our approach on the MTG dataset, extending the gap duration to 500 ms. Experimental results demonstrate that our method achieves competitive or superior performance compared to existing baselines, particularly for longer gaps, offering a robust solution for restoring degraded musical recordings. Audio examples of our proposed method can be found at https://iftach21.github.io/ 1 1 footnotetext: Equal contribution 1 Introduction Audio inpainting refers to the task of reconstructing missing or corrupted segments of an audio signal [ 1 ] . It is a fundamental inverse problem in audio processing, with applications ranging from restoring damaged recordings and removing artifacts, to filling in gaps caused by data loss in transmission or editing operations [ 24 ] . Traditional approaches to this problem often relied on signal modeling techniques such as autoregressive models [ 1 ] , sparse representations [ 19 ] , or linear predictive coding [ 1 ] . While effective under certain assumptions (e.g., local stationarity), these methods typically perform well only on short gaps and may struggle with long-range dependencies or semantic coherence [ 24 , 37 ] . In recent years, deep generative models have significantly advanced the state of audio inpainting. Models such as Variational Autoencoders (VAEs) [ 20 ] and diffusion probabilistic models [ 15 , 17 ] have demonstrated the ability to learn expressive priors directly from large-scale audio datasets. Notably, diffusion models have emerged as particularly powerful tools for solving ill-posed inverse problems due to their iterative denoising process and strong generative capacity. Approaches like DiffWave [ 15 ] apply diffusion directly on waveform samples, while others such as MAID [ 17 ] and CQT-Diff+ [ 24 ] operate in the continuous time-frequency domain using spectrograms or Constant-Q Transform (CQT) representations [ 24 , 37 ] . In this work, we propose a novel approach to audio inpainting based on discrete diffusion modeling inspired from the work of [ 18 ] . Unlike prior methods that operate in the continuous domain [ 15 , 17 , 24 , 37 ] , our method applies the diffusion process to a discrete latent space. Specifically, we employ the WavTokenizer [ 11 ] to quantize audio signals into compact sequences of discrete tokens, and perform the diffusion process entirely in this categorical space. This formulation allows the model to capture high-level semantic structures in audio, while avoiding the challenges of modeling raw waveforms or spectrograms directly. Our discrete approach provides several key benefits. First, the token representation reduces the sequence length compared to raw time-domain audio, and filters out low-level noise, simplifying the generative task. Second, discrete diffusion enables stable and coherent long-range modeling, making it well-suited for reconstructing larger gaps. Finally, operating in token space mitigates artifacts such as oversmoothing and spectral blurring, which are common in continuous-domain inpainting models [ 24 , 37 ] . This paper presents the first study of Discrete Diffusion Models (DDMs) for audio inpainting. Our method sets a new direction by unifying discrete generative modeling with inpainting tasks, and offers a promising alternative to waveform and spectrogram based diffusion approaches. Experimental results show that our model achieves high perceptual quality across diverse musical inputs and gap durations, even in challenging inpainting scenarios. 2 Related Work 2.1 Methods for Audio Inpainting The term “audio inpainting” was used by Adler et al. [ 1 ] to describe the restoration of missing or corrupted segments in audio signals. Although the terminology is relatively recent, the problem itself has been studied under various names, including audio interpolation [ 19 ] , audio extrapolation [ 16 ] , missing sample reconstruction [ 1 ] , waveform substitution [ 1 ] , and imputation [ 16 ] . Early methods primarily focused on short-gap restoration, typically below 100 ms, by assuming local stationarity in the signal. A major class of such methods is based on Auto-Regressive (AR) modeling [ 1 ] , where each sample is predicted from previous samples. These techniques are effective for short segments, but performance deteriorates with increasing gap length. Another classical approach exploits the sparsity of audio in time–frequency representations, such as the Short-Time Fourier Transform (STFT) or Gabor transforms [ 16 , 1 ] . Sparse reconstruction algorithms like Orthogonal Matching Pursuit (OMP) [ 34 ] are used to find sparse coefficient vectors that match the known parts of the signal. Later improvements employed adaptive dictionary learning instead of fixed transforms. For example, Tauböck et al. [ 32 ] learned Gabor-like atoms from the neighborhood of the gap, improving reconstruction fidelity. Nonnegative Matrix Factorization (NMF) based methods [ 22 ] represent the spectrogram as a product of low-rank matrices, allowing inference of missing time–frequency bins. Probabilistic extensions of NMF further enhance robustness in noisy settings. Beyond linear and sparse priors, more structured models have also been explored. Sinusoidal modeling leverages harmonic regularities in musical or speech signals [ 1 ] , while graph-based methods use audio self-similarity to propagate known content into missing regions. Notably, Perraudin et al. [ 27 ] introduced graph Laplacian regularization on self-similarity matrices to infer missing parts from acoustically similar, uncorrupted regions. Although these classical methods are effective for short-duration gaps, they typically struggle with gaps exceeding 100 ms due to the breakdown of their underlying assumptions. To address longer and more complex gaps, deep learning approaches have emerged. Convolutional neural networks were first applied to spectrogram-based inpainting. Marafioti et al. [ 20 ] proposed a context encoder , using a U-Net–style architecture to restore tens-of-milliseconds gaps from time–frequency representations. Generative Adversarial Networks (GANs) were then introduced to increase generative diversity and temporal coherence. Ebner and Eltelt [ 8 ] employed a Wasserstein GAN (WGAN) with multi-context conditioning to inpaint gaps up to 500 ms, while Marafioti et al. [ 19 ] extended this idea with GACELA, a multi-scale GAN that leverages latent conditioning to handle gaps as long as 1–1.5 seconds. These models outperform traditional methods on long gaps, especially when rich temporal dependencies or musical structures must be captured. Overall, the evolution from linear and sparse signal models to modern deep learning approaches has significantly advanced the state of audio inpainting, particularly for long and structured signal gaps. 2.2 Audio Inpainting using Diffusion Models Diffusion models, also known as score-based generative models, have recently revolutionized generative modeling in vision and audio. These models define a forward process that gradually adds noise to data and learn to reverse it

Read the complete text