BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation – Challenges and Insights
Chan-Jan Hsu, Yi-Cheng Lin, Chia-Chun Lin, Wei-Chih Chen, Ho Lam Chung, Chen-An Li, Yi-Chang Chen, Chien-Yu Yu, Ming-Ji Lee, Chien-Cheng Chen, Ru-Heng Huang, Hung-yi Lee, Da-Shan Shiu · 2025
Read BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation – Challenges and Insights by Chan-Jan Hsu, Yi-Cheng Lin, Chia-Chun Lin, Wei-Chih Chen, Ho Lam Chung, Chen-An Li, Yi-Chang Chen, Chien-Yu Yu, Ming-Ji Lee, Chien-Cheng Chen, Ru-Heng Huang, Hung-yi Lee, Da-Shan Shiu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17790.txt.
Introduction
Chan-Jan Hsu and co-authors present BreezyVoice in a 2025 paper on text to speech for Taiwanese Mandarin. Many characters in the language can be read in several ways, so choosing the right sound is a central challenge. The system builds on an existing speech model and adds phonetic control.
Components include a speech tokenizer, a large language model, a flow matching model, and a grapheme to phoneme predictor. Tests showed strong performance in general and code switching contexts, with realistic, high fidelity speech. The team also discuss hard cases, such as rare speakers and polyphone choices, and what they reveal about neural speech systems.
Catalog record
| CREATOR | Chan-Jan Hsu, Yi-Cheng Lin, Chia-Chun Lin, Wei-Chih Chen, Ho Lam Chung, Chen-An Li, Yi-Chang Chen, Chien-Yu Yu, Ming-Ji Lee, Chien-Cheng Chen, Ru-Heng Huang, Hung-yi Lee, Da-Shan Shiu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-17790
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation – Challenges and Insights Chan-Jan Hsu, Yi-Cheng Lin, Chia-Chun Lin, Wei-Chih Chen, Ho Lam Chung, Chen-An Li, Yi-Chang Chen, Chien-Yu Yu, Ming-Ji Lee, Chien-Cheng Chen, Ru-Heng Huang, Hung-yi Lee, Da-Shan Shiu Abstract We present BreezyVoice, a Text-to-Speech (TTS) system specifically adapted for Taiwanese Mandarin, highlighting phonetic control abilities to address the unique challenges of polyphone disambiguation in the language. Building upon CosyVoice, we incorporate a S 3 S^{3} tokenizer, a large language model (LLM), an optimal-transport conditional flow matching model (OT-CFM), and a grapheme to phoneme prediction model, to generate realistic speech that closely mimics human utterances. Our evaluation demonstrates BreezyVoice’s superior performance in both general and code-switching contexts, highlighting its robustness and effectiveness in generating high-fidelity speech. Additionally, we address the challenges of generalizability in modeling long-tail speakers and polyphone disambiguation. Our approach significantly enhances performance and offers valuable insights into the workings of neural codec TTS systems. 1 Introduction Text-to-Speech (TTS) technology has been widely adopted in customer service and accessible tools. Recently, the significant advancements in virtual assistants Hurst et al. (2024) increased the frequency of breadth of TTS utilization, which has prompted a shift away from the mechanical voice outputs towards speech that closely mimics the nuances of human conversation. Aside from immediate practical scenarios, we also highlight the emerging use case of synthetic data generation, as it offers extensive coverage for spontaneous speech in controllable written domains. The benefits of using these otherwise inaccessible synthetic data for training purposes have been well-documented in previous literature Lin et al. (2022) ; Yang et al. (2024) . In response to the increasing expectations for high-quality outcomes, it is particularly advantageous to address localized TTS needs, from adaptation of an existing system, leveraging the knowledge transfer potential from pretrained large-scale corpora. Recent realistic TTS systems have reported training data of approximately 170,000 hours Du et al. (2024a) , which is comparable to the scale of training large ASR systems Radford et al. (2023) , where adaptation efforts for specialization are common Tseng et al. (2024) . In this work, we build upon CosyVoice to develop a Taiwanese Mandarin TTS system, which we name as BreezyVoice. BreezyVoice demonstrates superior performance compared to commercial TTS systems, both in general and code-switching contexts. Beyond the training process, we tackle generalizability challenges in the adapted system from lower-resourced training, primarily involving the correct modeling of long-tail speaker characteristics and the polyphonic characters inherent in the Taiwanese Mandarin language. Our detailed analysis provide valuable insights into the workings of neural codec TTS systems. 2 Background 2.1 Unit-TTS Systems Advanced speech compression codec systems Défossez et al. (2022) ; Du et al. (2024b) have successfully encapsulated verbose speech information into quantized speech units. These units are seamlessly adopted for pursuing Text-to-Speech (TTS) systems Ju et al. (2024) , via incorporating a module that predicts the compressed units from text along with an optional speaker embedding. The design of the text-to-unit model is highly dependent on the granularity of the units to predict. For example, multi-layered codes are often predicted recursively using a non-autoregressive model Wang et al. (2023) ; Chen et al. (2024) , whereas single-layered codes are structured as a sequence-to-sequence generation task Du et al. (2024a) . From this perspective, we identified the benefits of single-layered supervised semantic tokens from Coisyvoice, which dynamically encode essential content and prosody information with a high compression rate. 2.2 The Necessity of Taiwanese Mandarin Conversion The default language of the Cosyvoice model is Mandarin, which is mutually intelligible with Taiwanese Mandarin, but also features various lexical and phonological differences Bradley (1992) . Further, Taiwanese Mandarin employs Traditional Chinese characters as the surface form, which significantly overlap with Cantonese in written form but differ greatly in pronunciation, and there have been scholars that regard them as separate languages Mair (1991) . As the CosyVoice model predominantly associates Traditional Chinese characters with Cantonese, it is crucial to ensure that the model is adequately trained with Taiwanese Mandarin data to guarantee performance on this domain. 2.3 Polyphone Disambiguation and Phonetic Control for Boosting TTS Accuracy Polyphone disambiguation is crucial for Mandarin grapheme-to-phoneme (G2P) conversion, essential for accurate text-to-speech (TTS) systems. Mandarin characters can have multiple pronunciations based on context 1 1 1 For instance, the character “行” has different pronunciations: “ㄒㄧㄥˊ” (to walk) or “ㄏㄤˊ” (a row). , making disambiguation vital, as incorrect pronunciations make phrases unintelligible. To alleviate potential errors in a speech synthesis system, the phonetic controllability of system is highly desired, as it allows us to preamptively augment the accurate Mandarin Phonetic Symbols predicted from a Neural Model g2pW Chen et al. (2022) . We perform phonteic augmenetations using g2pW in both training and inference to boost TTS accuracy. 3 Methodology 3.1 BreezyVoice Architecture The BreezyVoice Framework consists of four components: A Supervised Semantic Speech (S3) Tokenizer, a large language model (LLM), an optimal-transport conditional flow matching model (OT-CFM) Mehta et al. (2024) , and a grapheme to phoneme prediction model g2pW. The following sections describe the components in detail. 3.1.1 Supervised Semantic Speech Tokenizer The speech tokenizer is a Speech Encoder with a vector quantization layer Zeghidour et al. (2021) to output discrete units. The semantic supervision is derived from training these units on a supervised Automatic Speech Recognition (ASR) task. In the one-shot speech cloning setting, the speech tokenizer is used to generate units of the conditioning speaker sample. 3.1.2 Large Language Model for TTS In our framework, the text-to-unit generation task is fulfilled with a large language model (LLM). The model takes tokenized text data and a speaker embedding as inputs, and outputs speech units. During training, the LLM uses teacher forcing technique, taking advantage of the decoding structure to minimizing the cross-entropy losses of the predicted speech sequences. At inference, the LLM generates speech tokens in an auto-regressive manner, providing a foundation for high-fidelity speech synthesis. 3.1.3 Optimal-transport Conditional Flow Matching In our framework, the transformation of speech tokens into a Mel spectrogram is achieved using an optimal-transport conditional flow matching (OT-CFM) model. OT-CFM models leverage optimal transport (OT) to learn a dynamic gradual transformation, iteratively refining a prior into the meaningful data distribution. In particular, our OT-CFM model generates continuous spectrograms that accurately reflect the time-frequency structure of an utterance, from conditioning on speaker characteristics. The input to this stage includes LLM-generated speech tokens, a speaker embedding, and a masked Mel spectrogram where the masked part is to be reconstructed. 3.1.4 Phoneme Prediction Model In this work, we utilize g2pW as our phoneme prediction model. g2pW is an open-source model that currently holds the highest performance on the CPP benchmark Park and Lee (2020) . This model employs learnable softmax weights to condition the outputs of BERT Kenton and Toutanova (2019) . The authors demonstrate that using a conditional weighted softmax, which is conditioned on the polyphonic character of interest and its part-of-speech (POS) tagging, enhances the performance of polyphone disambiguation. 3.2 Implementation Details We now describe implementation details of our approach, covering data preparation and training. 3.2.1 Data Preparation We use a diverse collection of publicly available speech data and utilize its paired transcriptions. When transcriptions are unavailable, we use