Next Tokens Denoising for Speech Synthesis

Yanqing Liu, Ruiqing XueChong ZhangYufei LiuGang WangBohan LiYao QianLei HeShujie LiuSheng Zhao Microsoft · 2025

Read Next Tokens Denoising for Speech Synthesis by Yanqing Liu, Ruiqing XueChong ZhangYufei LiuGang WangBohan LiYao QianLei HeShujie LiuSheng Zhao Microsoft (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22746.txt.

Introduction

Yanqing Liu and nine co-authors merge two rival styles of speech generation into one text-to-speech model. Their 2025 paper introduces Dragon-FM. Step-by-step models see only the past and run slowly; diffusion-style models use both directions but cannot cache past work. The team wanted the strengths of both for long speech such as podcasts.

Their model handles sound in chunks. Across chunks it works step by step, which keeps the whole piece coherent and lets it reuse cached work. Inside each chunk, a flow-matching process cleans the sound in parallel with full context. The system runs on compact sound tokens, 12.5 per second at high audio quality. Tests on podcast data show fast, high-quality speech made with no example voice. The bridge between continuous and token-based modeling is a further claim.

Catalog record

CREATORYanqing Liu, Ruiqing XueChong ZhangYufei LiuGang WangBohan LiYao QianLei HeShujie LiuSheng Zhao Microsoft
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-22746

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Next Tokens Denoising for Speech Synthesis Yanqing Liu, Ruiqing XueChong ZhangYufei LiuGang WangBohan LiYao QianLei HeShujie LiuSheng Zhao Microsoft Abstract While diffusion and autoregressive (AR) models have significantly advanced generative modeling, they each present distinct limitations. AR models, which rely on causal attention, cannot exploit future context and suffer from slow generation speeds. Conversely, diffusion models struggle with key-value (KV) caching. To overcome these challenges, we introduce Dragon-FM, a novel text-to-speech (TTS) design that unifies AR and flow-matching. This model processes 48 kHz audio codec tokens in chunks at a compact rate of 12.5 tokens per second. This design enables AR modeling across chunks, ensuring global coherence, while parallel flow-matching within chunks facilitates fast iterative denoising. Thus, the model leverages KV-cache across chunks and utilizes bidirectional context within each chunk. Furthermore, it bridges continuous and discrete feature modeling, demonstrating that continuous AR flow-matching can predict discrete tokens with finite scalar quantizers. This efficient codec and fast chunk-autoregressive architecture also make the model highly effective for generating long-form content, such as podcasts. Experiments 1 1 1 See https://cognitivespeech.github.io/dragon-fm for demos of our work on podcast datasets demonstrate its capability to efficiently generate high-quality zero-shot podcasts. 1 1 footnotetext: These authors contributed equally to this work. Corresponding author: Yanqing Liu, [email protected] Figure 1: Proposed Architecture. Dragon-FM autoregressively predicts chunk-level speech tokens, with text and speech prompt. Within each block, flow-matching enables parallel prediction. 1 Introduction Generative modeling is characterized by a diverse and increasingly fragmented landscape of architectures and feature modeling approaches. Among these, autoregressive (AR) and diffusion models ( Radford et al. 2018 ; Lipman et al. 2022 ; Furuta et al. 2024 ; Wang et al. 2023 ; Ramesh et al. 2021 ) are leading paradigms across audio, image, and video generation. While both share iterative architectures, they exhibit distinct limitations, fundamentally differing in their iteration strategies, attention mechanisms, and token granularity. Diffusion models achieve higher inference throughput by generating multiple tokens in parallel through iterative denoising with bidirectional attention. Conversely, AR models predict tokens sequentially using causal attention, efficiently leveraging key-value (KV) caching for lower latency—a capability diffusion models currently lack. Regarding generation quality, diffusion models excel in continuous feature modeling, capturing detailed reconstructions with full-context attention but often struggle with discrete token modeling. In contrast, AR models, widely adopted in text generation (e.g., GPT series) ( Brown et al. 2020 ) , offer efficient discrete sampling and diverse outputs. However, their reliance on causal attention restricts access to future token information, potentially limiting performance in tasks requiring global context, such as reasoning or multimodal synthesis. Features used in diffusion and autoregressive (AR) models for various modalities are commonly categorized as discrete or continuous. Discrete tokens, whether from byte-pair encoding (BPE) for text or variational autoencoders (VAEs) with quantizers for audio, images, and video, offer advantages in compression and semantic richness, making them well-suited for next-token prediction in language models. A fundamental challenge lies in balancing training efficiency with compression quality. Higher token rates improve reconstruction but complicate long-sequence training for large language models (LLMs), while shorter sequences risk compromising reconstruction quality, potentially reducing intelligibility and increasing word error rates (WER) in speech. Conversely, continuous features convey richer acoustic and visual details but are inherently more difficult to model. While discrete AR language models perform exceptionally well in text generation due to their manageable BPE-encoded sequence lengths, discrete AR models for speech or images frequently struggle with the extensive token sequences required, often resulting in degraded reconstruction quality. In contrast, diffusion models excel at continuous text-to-image or video generation by effectively leveraging full-context modeling. Yet, both autoregressive and diffusion approaches face a compounding computational burden due to their iterative nature. To address the aforementioned limitations, we introduce a novel framework and validate its effectiveness through text-to-speech (TTS) applications. Recent advancements in TTS leverage discrete autoregressive (AR) codec language models and continuous flow-matching approaches to balance prosody modeling with fine-grained audio quality. For example, VALL-E ( Wang et al. 2023 ) employs two separately trained discrete language models: one autoregressively predicts residual codes of the first codec layer, while the other non-autoregressively predicts subsequent layers, leading to high latency. Tortoise ( Betker 2023 ) attempts to reduce latency by using a shorter discrete token sequence via a mel quantizer and replacing its second model with a diffusion-based approach, which improves real-time factor (RTF) compared to VALL-E but struggles with separate model optimization. E2 ( Eskimez et al. 2024 ) , a single-stage flow-matching model, directly predicts mel spectrograms instead of codec tokens, achieving high RTF without KV-cache but with reduced prosodic variation compared to AR models due to its fixed duration. Other efforts in TTS focus on improving codec token rates, developing advanced quantizers, and designing hybrid AR-diffusion systems, which often require extensive fine-tuning or complex architectural engineering. Quantizing speech into discrete sequences is inherently more challenging than text byte-pair encoding (BPE) due to speech’s rich prosody, timbre, and accent variations, necessitating specialized approaches like hierarchical decoders for layered codecs and cascaded speech upsampling with diverse AR and diffusion combinations. ( Jia et al. 2025 ) compresses continuous token patches using an aggregation encoder and employs a language model for inter-patch prediction, while a diffusion transformer handles intra-patch prediction with historical token patches. However, this aggregation and historical patch setting may affect sequence modeling quality due to compression artifacts. ( Liu et al. 2024 ) proposes ARDiT, a decoder-only diffusion transformer model that autoregressively generates continuous speech features. However, ARDiT needs a given total duration of target speech during inference and incurs larger latency due to longer speech sequences. Table 1 summarizes the comparison of existing AR and diffusion-based TTS models. Model AR Diffusion E2E Low Frame Rate Low Latency Low RTF VALL-E ✓ \times \times \times \times \times TorToise ✓ ✓ \times \times \times \times E2 \times ✓ ✓ \times \times ✓ ARDiT ✓ ✓ ✓ \times ✓ ✓ Dragon-FM ✓ ✓ ✓ ✓ ✓ ✓ Table 1: Comparison between the proposed model and state-of-the-art AR and diffusion-based TTS systems. Here, AR refers to autoregressive modeling, and diffusion indicates diffusion-based models. End-to-end (E2E) denotes joint training of the acoustic model. Low latency indicates small first-byte latency, approaching streaming systems, while low real-time factor (RTF) reflects strong parallel processing capability. We define a low frame rate as fewer than 15 tokens per second. In this work, we propose a novel generative modeling and text-to-speech (TTS) framework, Dragon-FM (flow matching), designed to overcome the limitations of individually optimized autoregressive (AR) and diffusion models. Our framework incorporates a high-quality 48kHz codec that compresses audio into a discrete token sequence at 12.5 tokens per second using a finite scalar quantizer ( Mentzer et al. 2023 ; Parker et al. 2024 ) , ensuring high reconstruction quality and manageable sequence lengths. A key innovation of the proposed model lies in its integration of AR next-token prediction with parallel flow-matching denoising within the same token chunk. This significantly reduces iterative steps while simultaneously preserving AR’s crucial key-value (KV) caching efficiency and leveraging flow-matching’s bidirectional context attention to boost scalability and sampling diversity. Recognizing analogous challenges in text modeling, where long-sequence large language models (LLMs) hinder real-time applications and diffusion models improve performance at an efficiency

Read the complete text