Text-To-Speech with Chain-of-Details: modeling temporal dynamics in speech generation

Jianbo Ma, Richard Cartwright · 2026

Read Text-To-Speech with Chain-of-Details: modeling temporal dynamics in speech generation by Jianbo Ma, Richard Cartwright (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19330.txt.

Introduction

Ma and Cartwright extend the coarse-to-fine idea in text-to-speech synthesis into the time domain with Chain-of-Details. Their cascaded framework refines temporal detail stage by stage, with each stage aimed at a specific time granularity. All detail predictions share one decoder, which keeps parameter use efficient across resolutions. Notably, the coarsest level plans phonetics on its own, with no separate phoneme duration predictor, and the method is tested on several datasets.

Catalog record

CREATORJianbo Ma, Richard Cartwright
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26180
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19330

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Text-To-Speech with Chain-of-Details: modeling temporal dynamics in speech generation Jianbo Ma, Richard Cartwright Abstract Recent advances in Text-To-Speech (TTS) synthesis have seen the popularity of multi-stage approaches that first predict semantic tokens and then generate acoustic tokens. In this paper, we extend the coarse-to-fine generation paradigm to the temporal domain and introduce Chain-of-Details (CoD), a novel framework that explicitly models temporal coarse-to-fine dynamics in speech generation using a cascaded architecture. Our method progressively refines temporal details across multiple stages, with each stage targeting a specific temporal granularity. All temporal detail predictions are performed using a shared decoder, enabling efficient parameter utilization across different temporal resolutions. Notably, we observe that the lowest detail level naturally performs phonetic planning without the need for an explicit phoneme duration predictor. We evaluate our method on several datasets and compare it against several baselines. Experimental results show that CoD achieves competitive performance with significantly fewer parameters than existing approaches. Our findings demonstrate that explicit modeling of temporal dynamics with the CoD framework leads to more natural speech synthesis. I Introduction Text-to-Speech (TTS) is a fundamental technology in modern Artificial Intelligence, powering a wide range of applications across diverse domains. Over the past several decades, TTS research has evolved through multiple paradigms. Early efforts focused on articulatory synthesis [ 6 ] , followed by the development of statistical parametric synthesis systems [ 48 ] , such as those based on Hidden Markov Models (HMM) [ 38 , 39 ] . In the late 2010s, the introduction of neural vocoders that leverage architectures like SampleRNN [ 23 ] and WaveNet [ 41 ] , were employed by models like the Deep Voice series [ 2 , 12 , 27 ] , Tacotron series [ 45 , 32 ] , and Char2wav [ 36 ] , which has improved quality of TTS. More recently, the powerful generative capabilities of diffusion models [ 34 , 13 , 35 ] have been applied to TTS, as seen in works such as Glow-TTS [ 17 ] , Grad-TTS [ 29 ] and naturalSpeech 2 [ 33 ] . The proliferation of large-scale speech datasets and advances in generative modeling have greatly enhanced the naturalness of synthesized speech. More recent state-of-the-art TTS systems can be broadly categorized into two groups: continuous feature generation and token prediction. Recent models like VoiceBox [ 20 ] and E2-TTS [ 9 ] , which utilize flow-matching techniques, fall into the first category, while models such as VALL-E [ 42 ] , NaturalSpeech 3 [ 15 ] , and MaskGCT [ 44 ] represent the latter. Within the token prediction paradigm, approaches can be further divided into autoregressive (AR) and non-autoregressive (NAR) models. AR models, which generate tokens sequentially in a manner akin to large language models (LLMs), have attracted considerable attention; Parler-TTS [ 22 ] is a notable example. In contrast, NAR models offer the advantage of parallel decoding, resulting in faster inference. Notably, SPEAR-TTS [ 16 ] introduced a two-stage, coarse-to-fine generation process, where the first stage predicts semantic information and the second stage refines acoustic details. This multi-stage paradigm has been adopted by subsequent works such as NaturalSpeech 3 [ 15 ] and MaskGCT [ 44 ] . Additionally, [ 11 ] proposed a single-stage model employing Masked Token Modeling inspired by MaskGIT [ 5 ] , though some trade-offs in performance has been observed. In this paper, we extend the coarse-to-fine generation paradigm by explicitly incorporating temporal dynamics through the introduction of the Chain-of-Detail (CoD) method. Inspired by VAR [ 37 ] , which models the progressive increase of image resolution in an autoregressive manner, our CoD approach decomposes speech generation into multiple progressive stages. At each stage, the model generates a version of the speech representation at a particular temporal resolution—starting from a temporal coarse, low-detail representation that captures broad timing and structure, and progressively refining it to higher temporal resolutions that add finer-grained acoustic details. Unlike VAR in the image domain, each temporal level in CoD employs masked generative modeling. Crucially, in CoD, a unified codebook and a single model architecture can be shared to all stages. This design allows the model to efficiently reuse parameters across all temporal resolutions, improving both efficiency and consistency. During inference, CoD-TTS operates in a non-autoregressive manner, generating all tokens in parallel at each temporal level. Experimental results show that CoD-TTS achieves competitive performance with fewer parameters. II Related Works As the proposed CoD model is a discrete token-based model, we primarily focus on related work in this category. Modern TTS models typically involve two main stages: speech representation and generative modeling. The first stage employs a VQ-GAN architecture where an encoder maps continuous time-domain waveforms into lower-dimensional representation embeddings, followed by vector quantization using techniques such as Residual Vector Quantization (RVQ) [ 46 ] . The quantized representations are subsequently converted back to continuous time-domain waveforms by a decoder. Representative works in this area include SoundStream [ 46 ] , EnCodec [ 8 ] , and DAC [ 18 ] . Figure 1 upper figure shows this first stage. The subsequent generative models operate on the discrete token space. Parler-TTS, which is a public implementation based on [ 22 ] , is an autoregressive token prediction model that utilizes the delay pattern introduced in MusicGen [ 7 ] . It follows a decoder-only architecture similar to [ 4 ] . We refer to this type of model that directly predicts acoustic tokens from transcript conditions as one-stage models. Fig. 1: Pipeline Overview of masked audio token modeling approach. It consists of two stages with 1) tokenization that tokenizes audio into audio tokens, and 2) masked audio token modeling predicts audio tokens masked at random with a transformer decoder. Masked Audio Token Modeling: Inspired by MaskGIT [ 5 ] , Masked Audio Token Modeling (MATM) learns to predict audio tokens that are randomly masked during training. Figure 1 presents the pipeline overview of this approach. During training, random audio tokens are masked and fed into a bidirectional transformer decoder, which learns to predict the masked tokens using cross-entropy loss. During inference, a cosine masking schedule is employed to iteratively predict audio tokens across multiple steps, where each step samples predicted tokens based on their confidence probabilities. We encourage reader to refer [ 5 ] for details. [ 11 ] proposed a single-stage masked audio token modeling approach that leverages knowledge distillation from a two-stage teacher model. This single-stage framework shares structural similarities with each level of our proposed CoD method, as both employ masked token prediction for speech generation. However, the temporal modeling aspect is not explicitly addressed in this work. Multi-stage generative models: The concept of two-stage TTS models was first introduced in SPEAR-TTS [ 16 ] . The core idea involves first predicting semantic tokens from text, followed by generating acoustic tokens conditioned on these semantic representations. This approach draws inspiration from AudioLM [ 3 ] , which proposed generating acoustic tokens through a coarse-to-fine progression from semantic tokens to detailed acoustic representations using a three-stage design. While originally developed for autoregressive (AR) language model-based generative models, this coarse-to-fine approach has been successfully adapted to non-autoregressive (NAR) models as well. Several language model-based

Read the complete text