DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding

Yang, Yunpeng, George, Shao-Fu, Craig, Alessio, Ramanan · 2025

Read DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding by Yang, Yunpeng, George, Shao-Fu, Craig, Alessio, Ramanan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22362.txt.

Introduction

Speech generators often work from tokens, compact codes for sound, but reading those codes one by one is slow. Yang Yang and co-authors present DiffSoundStream in 2025, a faster route for non-streaming use. Their codec is taught with meaning-based tokens alongside sound tokens, so the two sets overlap less and carry less waste.

A diffusion model then rebuilds the waveform from the meaning tokens plus a coarse layer of sound tokens. At fifty tokens per second, the system matched the quality of a standard codec running at twice that rate. A shortcut cut the diffusion process to four steps with only a small quality loss, making the speedup practical.

Catalog record

CREATORYang, Yunpeng, George, Shao-Fu, Craig, Alessio, Ramanan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-22362

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding Yang, Yunpeng, George, Shao-Fu, Craig, Alessio, Ramanan Abstract Token-based language modeling is a prominent approach for speech generation, where tokens are obtained by quantizing features from self-supervised learning (SSL) models and extracting codes from neural speech codecs, generally referred to as semantic tokens and acoustic tokens. These tokens are often modeled autoregressively, with the inference speed being constrained by the token rate. In this work, we propose DiffSoundStream, a solution that improves the efficiency of speech tokenization in non-streaming scenarios through two techniques: (1) conditioning the neural codec on semantic tokens to minimize redundancy between semantic and acoustic tokens, and (2) leveraging latent diffusion models to synthesize high-quality waveforms from semantic and coarse-level acoustic tokens. Experiments show that at 50 tokens per second, DiffSoundStream achieves speech quality on par with a standard SoundStream model operating at twice the token rate. Additionally, we achieve step-size distillation using just four diffusion sampling steps with only a minor quality loss. Yang Li Sung Shih Dooley Centazzo Rajeswaran keywords speech tokenization, nerual speech codec, latent diffusion model † † email: {yanghm, yunpeng, gsung, shaofu, dooleyc, centazzo, ramanan}@google.com 1 Introduction Recent advances in self-supervised learning [ 1 , 2 , 3 , 4 ] and neural codecs [ 5 , 6 ] have allowed the transformation of speech signals into sequences of discrete tokens, effectively reframing speech modeling as a token-based language modeling problem. This paradigm shift has led to advances in a wide range of speech modeling applications such as speech continuation [ 7 ] , text-to-speech synthesis [ 8 , 9 ] , speech separation [ 10 ] , speech translation [ 11 , 12 ] , and end-to-end speech dialogue systems [ 13 ] . A common thread across these solutions is their reliance on autoregressive (AR) modeling, applied either partially or fully to the token sequences. A key challenge in AR-based speech modeling is balancing computational efficiency and reconstruction quality. The computational cost of AR models scales with the length of token sequences they process, making lower token rates, measured in tokens per second of speech, an attractive strategy for reducing inference latency, lowering computational demands, and simplifying the modeling task. Furthermore, a lower token rate extends the effective audio context length that an AR model can process given a fixed token context window. On the reverse side, aggressive token rate reduction risks degrading speech fidelity, necessitating careful optimization of the rate-distortion trade-off. This balance between efficiency and quality remains a central consideration in the design of token-based speech systems. One way to mitigate distortion is by increasing the size of the codebook to enhance the information capacity per token. However, this approach is inherently unscalable, as the codebook size grows exponentially with the number of bits allocated to each token. In our work, we adhere to a fixed codebook size of 2048 per token, a standard configuration adopted in previous solutions [ 14 , 13 ] , and instead direct our efforts toward optimizing the token rate (tokens per second). In this work, we limit our scope to non-causal/non-streaming tokenization during encoding and decoding, and stick to the hybrid semantic-acoustic tokenization approach [ 7 , 8 , 13 , 11 , 12 ] . Rather than examining specific token-based language modeling schemes, we focus solely on the tokenization process itself, investigating how to reduce the token rate for speech while preserving the perceptual quality of the reconstructed waveform. 1.1 Contributions We propose DiffSoundStream, a tokenization model with three key contributions: Semantic-conditioned acoustic tokenization By conditioning both the encoder and decoder of the SoundStream codec on semantic tokens, we explicitly reduce redundancy between semantic and acoustic token streams. Unlike previous hybrid approaches [ 7 , 8 , 13 , 11 , 12 ] , where semantic tokens merely guide the generation of acoustic tokens without participating in waveform decoding, our method encourages acoustic tokens to capture complementary information from semantic ones, allowing efficient reconstruction with fewer tokens. Latent diffusion-based waveform decoding While standard neural speech codecs [ 6 , 5 ] employ deterministic GAN-trained decoders [ 15 , 16 ] without external stochasticity, we enhance the generative capability of the decoding process by training a latent diffusion model [ 17 , 18 ] conditioned on semantic and coarse acoustic tokens. This approach overcomes the inherent quality ceiling of a deterministic decoding at low token rates, leveraging its iterative sampling process to synthesize perceptually critical waveform details. Distilled 4-step diffusion sampling To mitigate the computational overhead of the generative process, we apply step-size distillation [ 19 ] to reduce sampling iterations to just 4 steps with minor degradation in perceptual quality. 1.2 Related works Previous works such as SpeechTokenizer [ 20 ] and Mimi [ 13 ] unify semantic and acoustic encoding via a single encoder by distilling information from an SSL. While distillation is necessary in transforming non-causal semantic information to a causal one, in our case we focus on non-causal tokenization and directly leverage SSL-derived semantic tokens without distillation. In addition to reducing token rate, hybrid AR and non-autoregressive (NAR) schemes [ 9 , 20 , 21 , 22 ] offer a complementary approach to improving the efficiency of token-based modeling and inference. We consider our scheme orthogonal to these designs. 2 Modeling Our proposed DiffSoundStream model comprises three components, which we detail in the subsequent subsections. • Semantic-conditioned SoundStream (SS-SC) : A modified SoundStream autoencoder [ 6 ] where both the encoder and decoder are conditioned on semantic tokens derived from a pretrained WavLM model [ 3 ] . • Continuous-latent SoundStream (SS-CL) : A SoundStream [ 6 ] variant that replaces the discrete residual vector quantization (RVQ) bottleneck with a continuous latent space, trained using noise augmentation and regularization following the VAE formulation [ 23 ] . • SoundStream latent diffuser : A diffusion model trained to model the continuous latent distribution from SS-CL , while conditioned on both the semantic tokens from WavLM and the acoustic tokens from SS-SC . 2.1 Semantic-conditioned SoundStream model The SS-SC model we propose takes 24kHz audio waveform as input, and process it with four layers of encoder and decoder convolutional blocks that follow the architecture illustrated in Figure. 3 of [ 6 ] . The four blocks resample the temporal dimension at a factor of [8,8,6,5] (from waveform to bottleneck), resulting in a frame rate of 12.5 Hz at the bottleneck. The RVQ bottleneck employs 8 quantizers with 2048 codebook entries each, trained with uniform quantizer dropout to improve codebook utilization. We maintain the original SoundStream training recipes [ 6 ] , combining multi-scale STFT discriminators-based adversarial loss, feature matching loss, and reconstruction losses, as defined by Equation (6) in [ 6 ] . Since we focus on non-causal tokenization, all the convolutions are non-causal with ‘same’ padding. Most token-based speech generation solutions adopt a hybrid tokenization scheme [ 7 , 8 , 13 , 11 , 12 ] that utilizes both semantic tokens derived from a self-supervised learning (SSL) model and acoustic tokens from a speech codec. Semantic tokens are often used as an intermediate modeling target to guide the generation of acoustic tokens and do not participate in the decoding of acoustic

Read the complete text