HoliTok:A Coutinuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding

Bohan Li, Shi Lian, Hankun Wang, Yiwei Guo, Yu Xi, Zhihan Li, Da Zheng, Colin Zhang, Kai Yu · 2026

Read HoliTok:A Coutinuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding by Bohan Li, Shi Lian, Hankun Wang, Yiwei Guo, Yu Xi, Zhihan Li, Da Zheng, Colin Zhang, Kai Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29948.txt.

Introduction

Bohan Li and eight colleagues build HoliTok, a speech tokenizer serving both speech making and understanding in one model. It squeezes high-rate speech into a stream of compact latents, staged to keep signal truth, sense, and ease of learning. One model both speaks and listens from the same latents, with no extra training tricks. Tests show strong rebuild quality, easier learning for controlled synthesis, and the only steady run among rivals in the joint setup. HoliTok offers a shared base for spoken models that must both hear and speak.

Catalog record

CREATORBohan Li, Shi Lian, Hankun Wang, Yiwei Guo, Yu Xi, Zhihan Li, Da Zheng, Colin Zhang, Kai Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48687
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29948

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HoliTok:A Coutinuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding Bohan Li, Shi Lian, Hankun Wang, Yiwei Guo, Yu Xi, Zhihan Li, Da Zheng, Colin Zhang, Kai Yu Abstract Unified speech foundation models require a holistic tokenization space that is both learnable by language models and decodable into high-quality waveforms. Existing speech tokenizers, however, often fail to satisfy these requirements simultaneously, leading to increased architectural complexity and more involved training designs. We propose HoliTok, a continuous Holi stic speech Tok enization model designed for unified generation-understanding modeling. HoliTok encodes 48 kHz speech into a compact 25 Hz sequence of 128-dimensional latents. It is trained with a progressive strategy that jointly preserves signal-level fidelity, incorporates semantic information, and maintains strong latent learnability. Based on this tokenization, we build a unified AR+DiT model for speech synthesis and recognition, where the same latent sequence supports both generation-specific and unified generation-understanding tasks. Experiments show that HoliTok achieves competitive reconstruction fidelity, improves generative learnability for high-quality and controllable synthesis, and, among the evaluated representations, is the only one that operates robustly in our unified generation-understanding architecture without additional optimization tricks. These results suggest that HoliTok serves as an effective speech tokenizer and a foundational representation interface for unified spoken language modeling. The code is available at: https://github.com/bovod-sjtu/HoliTok . 1 1 footnotetext: is the corresponding author. 1 Introduction Recent progress in multimodal foundation models is moving toward unified understanding and generation 58 ; 29 ; 21 ; 19 ; 49 ; 50 . Rather than treating downstream tasks separately, emerging systems seek to build all-in-one architectures that can understand, reason over, and generate within a shared parameter space. In the speech domain, this direction places a stronger requirement on the tokenizer: speech should be represented in a continuous space that is simultaneously decodable, learnable, and informative, so that it can serve as the interface for unified generation-understanding modeling. However, such a holistic continuous speech tokenizer remains underdeveloped. In its absence, downstream models must compensate through incremental architectural designs, such as task-specific encoders, multiple token streams, or decoupled modules. Consequently, the burden of unification is shifted from the representation itself to increasingly complex model design 51 ; 52 ; 54 . Conventional acoustic front-end features, such as mel spectrograms, Fbank features, and MFCCs 1 , retain local signal structure, but they produce dense frame-level sequences that are redundant and difficult to model for downstream understanding and generation. In contrast, self-supervised speech representations 5 ; 26 ; 10 expose richer semantic information, but they are not naturally decodable into high-fidelity waveforms and often present a challenging target for generative modeling. Thus, existing representations typically satisfy only part of the requirements for unified continuous speech modeling, leaving a gap between semantic abstraction, acoustic fidelity, and model learnability. Current speech tokenizers address this challenge only partially. Discrete codec-based tokenizers 13 ; 27 ; 18 ; 24 compress speech into language-model-friendly symbols, but quantization and multi-codebook designs may introduce information loss and additional modeling complexity. Continuous tokenizers 32 ; 38 ; 11 avoid quantization and are favorable for generation, yet many are optimized mainly for reconstruction or synthesis rather than as a shared tokenization space for unified generation-understanding models. Existing “unified” representations 15 ; 55 are also often evaluated in task-specific systems separately, leaving the consistency of the shared modeling space unclear. Recent AR+DiT architectures offer a simple downstream framework for unified continuous speech generation and understanding. For example, Ming-UniAudio 54 proposes MingTok-Audio to connect a compact variational autoencoder (VAE) latent with richer semantic features via an additional semantic module. While this improves tokenizer usability, the low-level latent remains fixed as higher-level semantics are introduced, resulting in an inconsistent modeling space and limited generative capacity. In this work, we propose HoliTok , a Holi stic Tok enization model for unified continuous speech generation and understanding. HoliTok encodes 48 kHz speech into a compact 25 Hz sequence of 128-dimensional continuous latents. Its training follows a progressive recipe that gradually shapes a learnable and semantically informative latent space. We first train an autoencoder to ground the representation in faithful waveform reconstruction. We then introduce a sequence-aware variational bottleneck to regularize the latent distribution, making the sequence smoother and easier to predict while preserving signal-level fidelity. Finally, we strengthen variational regularization and refine the latent space through high-level feature distillation and audio-language supervision, enabling the resulting tokenization to retain information useful for spoken language understanding while remaining highly learnable for diverse speech synthesis tasks. We build a unified generation-understanding model based on an AR+DiT architecture to evaluate whether a continuous speech tokenizer can serve as a unified modeling interface. The latent sequence is first encoded into patch embeddings for autoregressive modeling by the LLM. For generation, the LLM predicts semantic hidden states, which condition a DiT-based flow-matching head to predict the next latent patch. For understanding, the LLM predicts the next text token through an LM head. This evaluation is intentionally downstream-aware: beyond measuring reconstruction quality, it examines whether the tokenizer facilitates unified AR+DiT modeling. We evaluate HoliTok from three complementary perspectives: reconstruction, speech synthesis, and unified generation-understanding modeling. Empirically, HoliTok achieves competitive reconstruction fidelity with a highly compact latent sequence, while supporting high-quality, diverse, and controllable TTS. In unified spoken language modeling, instantiated with ASR and TTS, HoliTok-Base already provides a more modeling-friendly continuous latent space than existing alternatives. HoliTok-Unite further improves both synthesis and recognition by incorporating the causal semantic encoder trained in the final stage, demonstrating substantially better usability than the baselines. These results show that HoliTok is not only an effective speech tokenizer, but also a principled representation interface that bridges the modeling-space gap between unified continuous speech understanding and generation. 2 Related Work Audio representation for unified generation and understanding. Audio tokenization for unified modeling has been studied through both discrete and continuous representations. Discrete codecs and speech tokenizers 13 ; 27 ; 18 , provide compact language-model-friendly units. Continuous tokenizers avoid quantization and have been explored in 32 ; 38 ; 15 ; 55 for speech synthesis or unified audio modeling. These works improve different aspects of acoustic fidelity, semantic accessibility, and downstream usability. Compared with these works, HoliTok emphasizes holistic evaluation of the tokenization space within a single unified generation–understanding model, directly testing whether the same continuous representation is modelable as a shared interface for both speech generation and understanding. Unified generation-understanding architecture with continuous tokens. Continuous-token architectures have recently emerged for unified generation and understanding. In vision, recent works 19 ; 49 ; 50 perform autoregressive generation and understanding with continuous visual tokens. Similar in audio, DiTAR 28 uses an autoregressive backbone with a DiT-based flow-matching head for continuous speech patches, and Ming-UniAudio 54 extends this idea to unified speech understanding, generation, and editing. It shows that continuous tokens can support unified modeling, but also make the representation space itself a bottleneck. Our work adopts the AR+DiT setting as a downstream-aware evaluation protocol and shows that HoliTok better balances generation and understanding under the same architecture. 3 Methodology Figure 1: An Overview. Left side is the three-stage training strategy of HoliTok; Right side is our downstream architecture for unified generation-understanding tasks. 3.1 Main Architecture HoliTok is a speech tokenizer built on a low-latency

Read the complete text