OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models
Han Zhu, Lingxuan Ye, Wei Kang, Zengwei Yao, Liyong Guo, Fangjun Kuang · 2026
Read OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models by Han Zhu, Lingxuan Ye, Wei Kang, Zengwei Yao, Liyong Guo, Fangjun Kuang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00688.txt.
Introduction
Zhu and colleagues present OmniVoice, a zero-shot text-to-speech model for over 600 languages. Its core is a discrete non-autoregressive diffusion architecture. Older discrete models stall in two-step pipelines from text to meaning to sound, but OmniVoice maps text to multi-codebook sound tokens. Two tricks enable this: full-codebook random masking for training, and starting from a pre-trained language model for clearness. Trained on 581,000 hours of multilingual data gathered from open sources, it claims the broadest language reach to date with top results.
Catalog record
| CREATOR | Han Zhu, Lingxuan Ye, Wei Kang, Zengwei Yao, Liyong Guo, Fangjun Kuang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59831 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.00688
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models Han Zhu, Lingxuan Ye, Wei Kang, Zengwei Yao, Liyong Guo, Fangjun Kuang Abstract We present OmniVoice, a massively multilingual zero-shot text-to-speech (TTS) model that scales to over 600 languages. At its core is a novel diffusion language model-style discrete non-autoregressive (NAR) architecture. Unlike conventional discrete NAR models that suffer from performance bottlenecks in complex two-stage (text-to-semantic-to-acoustic) pipelines, OmniVoice directly maps text to multi-codebook acoustic tokens. This simplified approach is facilitated by two key technical innovations: (1) a full-codebook random masking strategy for efficient training, and (2) initialization from a pre-trained LLM to ensure superior intelligibility. By leveraging a 581k-hour multilingual dataset curated entirely from open-source data, OmniVoice achieves the broadest language coverage to date and delivers state-of-the-art performance across Chinese, English, and diverse multilingual benchmarks. Our code and pre-trained models are publicly available 1 1 1 https://github.com/k2-fsa/OmniVoice . 1 Introduction Zero-shot text-to-speech (TTS) models trained on large-scale datasets have demonstrated a remarkable ability to generate high-quality speech conditioned on only a few seconds of reference audio chen2025neural ; anastassiou2024seed . Despite these advances, most existing models support only a limited set of languages, often leaving hundreds of low-resource languages behind. Expanding language coverage is not merely a technical challenge but also a crucial step toward extending speech technologies to languages across the globe. To address this gap, we aim to develop a massively multilingual zero-shot TTS model supporting hundreds of languages. Realizing such extensive scaling requires a TTS architecture with exceptional capacity and robustness to handle diverse linguistic patterns. In the pursuit of such scalable and high-quality TTS, current research primarily adheres to two paradigms: autoregressive (AR) du2024cosyvoice2 ; guo2024fireredtts ; zhou2025indextts2 ; jia2025ditar ; wang2025spark ; ye2025llasa ; song2025distar ; zhou2025voxcpm ; cui2025glm and non-autoregressive (NAR) models le2023voicebox ; eskimez2024e2 ; chen2024f5 . NAR models offer advantages in both inference speed (via parallel decoding) and robustness (owing to bidirectional context) le2023voicebox ; chen2024f5 ; yang2025pseudo . Within NAR frameworks, models can be broadly categorized into continuous-latent-based chen2024f5 ; zhu2025zipvoice ; zhu2025zipvoicedialog ; jiang2025megatts and discrete-token-based wang2025maskgct . The latter was shown to yield superior prosodic diversity yang2025measuring . However, state-of-the-art (SOTA) discrete-token NAR systems wang2025maskgct typically rely on complex two-stage cascaded pipelines (text-to-semantic followed by semantic-to-acoustic). While such decoupling simplifies the training of individual modules, it also introduces significant drawbacks: (1) error propagation, where inaccuracies in semantic prediction degrade the final audio quality; and (2) information bottlenecks, where low-bitrate semantic representations sacrifice fine-grained acoustic details. While single-stage alternatives gallego2025single attempt to bypass these issues, they have historically lagged behind two-stage systems in terms of speech intelligibility. To bridge this gap, we introduce OmniVoice, an architecturally streamlined yet highly effective discrete NAR TTS framework. OmniVoice employs a discrete masked diffusion objective sahoo2024simple with a bidirectional Transformer vaswani2017attention to directly map text to multi-codebook acoustic tokens, thereby bypassing the complexity and limitations of cascaded pipelines. Its core modeling philosophy extends the success of diffusion language models nie2025large ; ye2025dream to the speech domain. We demonstrate that the potential of this minimalist architecture can be fully unleashed through two technical innovations: (1) Full-Codebook Random Masking: Conventional multi-codebook acoustic prediction methods adopt "per-layer" masking schedules borsos2023soundstorm ; wang2025maskgct , suffering from inefficient convergence. We propose a fully stochastic masking strategy across all codebooks that significantly enhances training efficiency and generative quality. (2) LLM Initialization: To resolve the intelligibility issues in single-stage discrete NAR models, we initialize our backbone with pre-trained AR LLM weights to inherit linguistic knowledge, making OmniVoice the first NAR TTS model to successfully benefit from LLM initialization du2024cosyvoice2 . The architectural advantage of OmniVoice makes it uniquely suited for scaling across diverse linguistic contexts. We curated a 581k-hour multilingual dataset encompassing more than 600 languages, derived exclusively from open-source resources. Training on this dataset allows OmniVoice to achieve the most extensive language coverage reported to date, bridging the gap for hundreds of previously under-served low-resource languages. Beyond its extensive language coverage, OmniVoice supports multi-dimensional controllability, including prompt denoising zhang2025advanced ; wang2024investigation , speaker attribute-based voice design hu2026voicesculptor , and fine-grained paralinguistic and phonetic control liao2025nvspeech ; deng2025indextts . These features significantly enhance its versatility, making OmniVoice well-suited for a wide range of real-world applications. Comprehensive experimental evaluations across Chinese, English, and massively multilingual benchmarks (covering up to 102 languages) validate that OmniVoice delivers SOTA performance in intelligibility, speaker similarity and naturalness, setting a new frontier for high-quality, high-coverage multilingual TTS. 2 Proposed Method This section elaborates the design of OmniVoice. We first introduce the streamlined single-stage architecture that serves as the high-performance backbone for massively multilingual modeling, followed by two core technical optimizations to boost model training efficiency and speech intelligibility. On this basis, we further present the large-scale multilingual data construction and balancing strategies to realize extensive language coverage, and finally equip the model with multi-dimensional controllability for practical deployment. 2.1 Architecture Figure 1: Illustration of OmniVoice architecture. OmniVoice is a single-stage NAR TTS model with a diffusion language model-style architecture nie2025large ; ye2025dream . Specifically, the model is trained with discrete diffusion objective and adopts a bidirectional Transformer backbone. OmniVoice directly maps text to multi-codebook acoustic tokens, eliminating the error propagation and information bottlenecks issues in conventional two-stage cascaded pipelines. This end-to-end streamlined architecture lays the foundation for large-scale multilingual modeling, with targeted optimizations and scaling strategies detailed in the subsequent sections. The architecture of OmniVoice is illustrated in Fig. 1 . The input of OmniVoice comprises: • Text token sequence ( Y ): A concatenated sequence of instruct and transcript tokens, providing the linguistic and task-oriented guidance. • Acoustic token matrix ( X ): A multi-codebook matrix X\in\mathbb{R}^{T\times C} , where T is the number of time steps and C is the number of codebooks. The acoustic matrix X is partitioned along the temporal dimension into two segments: the prompt segment X_{\mathrm{prompt}} , which contains the prefix acoustic context, and the target masked segment X_{\mathrm{target}} , where tokens are randomly replaced with a special mask token [M] . The model is designed to leverage the text conditions Y , the prompt X_{\mathrm{prompt}} and unmasked tokens in X_{\mathrm{target}} to recover the original tokens in the masked position in X_{\mathrm{target}} . The text tokens are embedded via a text embedding layer, and the acoustic tokens are embedded via codebook-specific embedding layers. The embeddings of all C codebooks at the same temporal position are summed to form a unified embedding, which is then fed into a bidirectional Transformer. On the output side, to reconstruct the multi-codebook tokens, OmniVoice employs C independent, codebook-specific prediction heads. Each head projects the final hidden states to output a probability distribution over the vocabulary of its corresponding codebook. The training loss is computed on the masked acoustic token positions, aiming to optimize the model for accurate token recovery. Let \mathcal{M} denote the set of indices (t,c) corresponding to masked positions within the target segment, where t\in\{T_{p}+1,\dots,T\} and c\in\{1,\dots,C\} . The training loss \mathcal{L} is formulated as: \mathcal{L}=-\sum_{(t,c)\in\mathcal{M}}\log P(x_{t,c}\mid X,Y;\theta) (1) where x_{t,c} is the ground-truth acoustic token at time step t and codebook index c , and P(x_{t,c}\mid\dots;\theta) is the probability distribution predicted by the model parameterized by \theta . 2.1.1 Full-Codebook Random Masking for Training