Bayesian Speech synthesizers Can Learn from Multiple Teachers

Ziyang Zhang, Yifan Gao, Xuenan Xu, Baoxiangli, Wen Wu, Chao Zhang · 2025

Read Bayesian Speech synthesizers Can Learn from Multiple Teachers by Ziyang Zhang, Yifan Gao, Xuenan Xu, Baoxiangli, Wen Wu, Chao Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24372.txt.

Introduction

Ziyang Zhang and five co-authors propose BELLE, a Bayesian text to speech system that can learn from several teachers. Speech synthesis is one to many: one text can be spoken many valid ways, so outputs form a distribution whose spread depends on the input. Most systems leave that spread implicit or fix it in advance rather than modeling it.

BELLE places a Bayesian prior over the mean and variance of each mel spectrogram frame and predicts it from the input, while keeping single pass sampling of standard autoregressive models. Learning the variance needs several recordings per text, which normal corpora lack, so learning from multiple teachers supplies that signal. The method models delivery variation explicitly.

Catalog record

CREATORZiyang Zhang, Yifan Gao, Xuenan Xu, Baoxiangli, Wen Wu, Chao Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24372

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bayesian Speech synthesizers Can Learn from Multiple Teachers Ziyang Zhang, Yifan Gao, Xuenan Xu, Baoxiangli, Wen Wu, Chao Zhang Abstract Codec-based text-to-speech (TTS) models have recently gained traction for their efficiency and strong performance in voice cloning. However, codec-based TTS faces limitations due to the challenges of pretraining robust speech codecs and the quality degradation introduced by quantization errors. Emerging evidence suggests that continuous-valued generative models can alleviate these issues and serve as a promising alternative. Yet, effectively modelling diverse speech patterns and developing reliable sampling strategies for continuous-valued autoregressive (AR) TTS remains underexplored. In this work, we propose BELLE, B ayesian e vidential l earning with l anguag e modelling for TTS, a novel continuous-valued AR framework that directly predicts mel-spectrograms from textual input. BELLE treats each mel-spectrogram frame as a Gaussian distribution sampled from a learned hyper-distribution, enabling principled uncertainty estimation, particularly in scenarios with parallel data ( i.e. , one text-audio prompt paired with multiple speech samples). To obtain such data, diverse speech samples are synthesized using multiple pre-trained TTS models given the same text-audio prompts, which are distilled into BELLE via Bayesian evidential learning. Experimental results indicate that BELLE demonstrates highly competitive performance compared with the current best open-source TTS models, even though BELLE is trained on a large amount of synthetic data and uses only approximately one-tenth of their training data. Audio samples generated by BELLE are available at https://belletts.github.io/Belle/ . The code, checkpoints, and synthetic data will be released after the paper is accepted. 1 Introduction Recently, autoregressive (AR) next-token prediction models based on discrete audio codecs have gained significant attention in the audio generation and text-to-speech (TTS) communities, driven by the success of models such as AudioLM Borsos et al. 2023 and VALL-E ( Wang et al. 2023 ) , among others. However, continuous-valued prediction has emerged as a promising alternative due to its ability to eliminate quantization errors introduced by token-based codecs ( Meng et al. 2024 ; Liu et al. 2024 ; Lin & He 2025 ; Chen et al. 2024b ; Zhu et al. 2024 ; Eskimez et al. 2024 ; Chen et al. 2024a ; Wang et al. 2025a ; Jia et al. 2025 ; Lee et al. 2025 ) . Generative models typically rely on sampling mechanisms to introduce stochasticity and improve synthesis quality. In audio-codec-based AR TTS, this is often achieved using mature top- p sampling techniques, similar to those employed in token-based language models (LMs). Diffusion-based models, by contrast, introduce randomness by injecting noise into the input, thereby embedding variability directly into the generation process. However, effective sampling strategies for continuous-valued AR models remain underexplored. Recent studies have begun to address this gap. In image generation, MAR proposed a hybrid approach where an AR model predicts conditioning signals, followed by diffusion-based sampling ( Li et al. 2024a ) . In zero-shot TTS, MELLE introduced Gaussian sampling ( Meng et al. 2024 ) , while FELLE adopted flow-matching techniques ( Lipman et al. 2022 ; Wang et al. 2025a ) to enhance sampling efficiency and output diversity. In this paper, we propose BELLE: B ayesian e vidential l earning with l anguag e modelling for TTS Synthesis—a continuous-valued AR model that directly predicts mel-spectrograms from text inputs based on the evidential deep learning (EDL) framework ( Amini et al. 2020 ) . This represents the first application of Bayesian deep learning to TTS, enabling controllable and theoretically grounded sampling at test time. Specifically, BELLE models each predicted mel-spectrogram frame as a sample from a Gaussian distribution, whose parameters themselves are drawn from hyper-distributions. Metaphorically, the model first predicts the parameters of this hyper-distribution, then samples a specific Gaussian distribution from it, and finally generates mel-spectrogram frames from the sampled Gaussian. However, learning accurate Gaussian posterior estimates from limited data poses a challenge, as typical TTS datasets contain only a single audio recording per text prompt. To overcome this, the training corpus is augmented with synthetic audio samples generated by several publicly available pre-trained zero-shot TTS models ( Du et al. 2024 ; Deng et al. 2025 ; Wang et al. 2025c ; Wang et al. 2024 ; Casanova et al. 2024 ) . Our main contributions are as follows: • We present BELLE, the first AR TTS model using evidential deep learning to train and sample from continuous-valued Mel-Spectrogram space. The results show that Bayesian evidential sampling outperforms conventional Gaussian sampling. • We propose the first multi-teacher knowledge distillation approach for TTS model training, integrating it with evidential deep learning. • BELLE achieves competitive performance compared with leading open-source TTS systems trained on approximately 50k hours of speech data, while BELLE uses only around 5k hours of data that largely consist of synthetic samples. Additionally, its streaming variant, BELLE-stream, attains an effective balance between audio generation quality and latency. 2 Related Work Continuous-valued Zero-Shot TTS and Streaming TTS. Continuous-valued TTS models can be broadly categorized into AR and non-autoregressive (NAR) approaches. AR methods include MELLE ( Meng et al. 2024 ) , which generates mel spectrogram frames using decoder-only Transformer-based AR modelling combined with Gaussian sampling. FELLE ( Wang et al. 2025a ) , ARDiT ( Liu et al. 2024 ) and DiTAR ( Jia et al. 2025 ) first generate continuous-valued hidden states autoregressively, then employ flow-matching or diffusion-based modules, such as DiT ( Peebles & Xie 2023 ) , to sample and produce mel spectrograms or a continuous latent space learned by a VAE ( Pinheiro Cinelli et al. 2021 ) . Moreover, VAE-based AR methods, including GMM-LM ( Lin & He 2025 ) and KALLE ( Zhu et al. 2024 ) , initially leverage a VAE to estimate a latent distribution (mean and variance), and subsequently autoregressively predict latent representations sampled from this distribution. NAR models such as E2 ( Eskimez et al. 2024 ) and F5 ( Chen et al. 2024a ) , on the other hand, predict mel-spectrograms directly via iterative mask-based refinement. Recently, several streaming TTS systems have emerged ( Du et al. 2024 ; Yang et al. 2024 ; Sun et al. 2025 ; Sheng et al. 2025 ; Wang et al. 2025b ) . Most of these approaches model discrete features using a language model, followed by a flow-matching module or other similar components to generate speech features. Such architectures are relatively complex and may introduce potentially higher latency. Bayesian Methods in Sequence Modelling and TTS. Evidential Deep Learning (EDL) ( Amini et al. 2020 ) is commonly used in recurrent neural networks (RNNs) and Transformers to provide per-time-step uncertainty ( Cui et al. 2024 ; Khot et al. 2024 ; Marvi et al. 2025 ) , benefiting tasks like time-series forecasting ( Khot et al. 2024 ; Marvi et al. 2025 ) . Other Bayesian approaches, such as Bayesian neural networks ( Kononenko 1989 ) in Long Short-Term Memory (LSTMs) and Transformers ( Muñoz et al. 2024 ) , and frameworks like SYMHnet ( Abduallah et al. 2024 ) , also address sequential uncertainty. Previous Bayesian TTS work focused on HMM-based models ( Hashimoto et al. 2009b ; Hashimoto et al. 2009a ; Lu & King 2012 ) ; to our knowledge, we are the first to apply

Read the complete text