Speaker-Conditioned Phrase Break Prediction for Text-to-Speech with Phoneme-Level Pre-trained Language Model Journal: Speech Communication

Dong Yang, Yuki Saito, Takaaki Saeki, Tomoki Koriyama, Wataru Nakata, Detai Xin, Hiroshi Saruwatari · 2025

Read Speaker-Conditioned Phrase Break Prediction for Text-to-Speech with Phoneme-Level Pre-trained Language Model Journal: Speech Communication by Dong Yang, Yuki Saito, Takaaki Saeki, Tomoki Koriyama, Wataru Nakata, Detai Xin, Hiroshi Saruwatari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00675.txt.

Introduction

Dong Yang leads seven authors in total, with Yuki Saito, Takaaki Saeki, Tomoki Koriyama, Wataru Nakata, Detai Xin, and Hiroshi Saruwatari, on phrase break prediction for text-to-speech. Phrase breaks, also called phrasing, decide where a synthetic voice pauses, and in multi-speaker systems the right pauses vary by speaker.

Yang and his co-authors add speaker embeddings, numeric voice fingerprints, to the phrasing model and show those embeddings capture speaker traits from the phrasing task alone. They adapt embeddings to unseen speakers with few examples, and they pioneer phoneme-level pre-trained language models, models trained on speech sounds, for this front-end task, which boosts accuracy greatly. Objective and subjective tests confirm the gains.

Catalog record

CREATORDong Yang, Yuki Saito, Takaaki Saeki, Tomoki Koriyama, Wataru Nakata, Detai Xin, Hiroshi Saruwatari
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-00675

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Speaker-Conditioned Phrase Break Prediction for Text-to-Speech with Phoneme-Level Pre-trained Language Model Journal: Speech Communication Dong Yang, Yuki Saito, Takaaki Saeki, Tomoki Koriyama, Wataru Nakata, Detai Xin, Hiroshi Saruwatari Abstract This paper advances phrase break prediction (also known as phrasing) in multi-speaker text-to-speech (TTS) systems. We integrate speaker-specific features by leveraging speaker embeddings to enhance the performance of the phrasing model. We further demonstrate that these speaker embeddings can capture speaker-related characteristics solely from the phrasing task. Besides, we explore the potential of pre-trained speaker embeddings for unseen speakers through a few-shot adaptation method. Furthermore, we pioneer the application of phoneme-level pre-trained language models to this TTS front-end task, which significantly boosts the accuracy of the phrasing model. Our methods are rigorously assessed through both objective and subjective evaluations, demonstrating their effectiveness. 1 Introduction Text-to-speech (TTS) systems [ 1 , 2 ] aim to generate intelligible, natural-sounding artificial speech from a given text with a specific speaker’s voice. Typically, TTS systems comprise two primary components: the front-end for text processing and the back-end for speech synthesis. Recent advancements in neural end-to-end generative TTS models (acoustic models [ 3 , 4 , 5 , 6 , 7 , 8 , 9 ] and vocoders [ 10 , 11 , 12 , 13 , 14 ] ), have enabled synthetic speech to closely resemble natural human voice and effectively capture the speaker’s acoustic features. Consequently, enhancements to the front-end of TTS systems are receiving increased attention, intending to refine synthetic speech’s naturalness by exploiting richer linguistic and semantic information and capturing more human speech patterns. Notably, pause insertion is indispensable to improve synthetic speech’s naturalness, prosody, and rhythm, especially in long-form TTS synthesis [ 15 ] . Inserting pauses in speech is a natural phenomenon in human communication, often done to enhance expressiveness or to take a breath. In TTS, pauses can broadly be classified into two types: respiratory pauses (RPs) [ 16 , 15 ] and punctuation-indicated pauses (PIPs) . RPs are inserted at word transitions without punctuation, whereas PIPs occur at punctuation marks, aligned with textual indications. Since speakers commonly insert PIPs at punctuation marks during reading, predicting the position of RPs for a given text is more challenging for TTS systems, termed phrase break prediction or phrasing . Many TTS systems [ 17 , 18 ] have integrated a predictive model for phrasing (referred to as a phrasing model) into their front-end, which utilizes word representations derived from a pre-trained language model (PLM). Since Transformer [ 19 ] encoder-based PLMs with self-supervised learning (SSL), such as BERT [ 20 ] , have demonstrated superior word representations across various natural language processing (NLP) tasks, they become increasingly adopted in TTS models [ 21 , 22 , 23 , 24 ] and phrasing models [ 25 , 26 , 27 ] . From our empirical observations, we identify two key challenges in current phrasing models: Challenges 1. In speech corpora, RPs are sparsely distributed, necessitating the use of large datasets to develop robust phrasing models. However, large speech corpora typically involve multiple speakers. Although RP insertion follows some latent rules [ 15 ] , different speakers have different styles of inserting RPs. The phrasing models that ignore speaker identity are forced to learn an “average” phrasing behavior, treating individual stylistic variations as noise, which inherently limits their predictive accuracy. Challenges 2. While mainstream PLMs, such as BERT BASE , have enhanced the performance of phrasing models, they typically generate representations at the subword level. Their effectiveness in phrasing tasks suggests that RP insertion is correlated with syntactic and semantic features. However, RP insertion is fundamentally a spoken-language phenomenon, governed by acoustic and articulatory mechanisms, whereas subword tokens are merely abstract textual units that lack direct information about acoustic features. To tackle these two challenges, we put forward two main hypotheses and propose several methods not only to verify them but also to explore further implications and applications: Hypothesis 1. The performance limitations of the phrasing model on multi-speaker datasets stem from the absence of speaker-specific features. Method 1. Incorporating speaker embeddings into the phrasing model. In multi-speaker TTS models, speaker embeddings are widely used to capture and control the timbre and prosody characteristics of speakers, represented as fixed-length vectors. We propose inserting the speaker embedding between the encoder and decoder of the phrasing model to enable the model to learn the speaker-specific features. Besides, in TTS models, speaker embeddings can be either randomly initialized or derived from a pre-trained speaker verification model (PSVM). The former approach is restricted to the speakers included in the training set, referred to as seen speakers . The latter can generalize to unseen speakers which were not present during training. These methods present an opportunity to investigate whether the prosody and rhythm features encoded in PSVM embeddings can be effectively leveraged in phrasing tasks. Hypothesis 2. As the fundamental units of speech, phonemes have proven to be highly effective inputs in TTS synthesis models [ 28 , 8 ] . Consequently, phoneme representations, which inherently encode richer acoustic attributes than subword representations, could potentially enhance the predictive accuracy of phrasing models. Method 2. Applying phoneme-level PLMs in phrasing models. Within the BERT framework, PLMs that use phoneme tokens as input have been developed: mixed-phoneme BERT [ 28 ] (MP BERT) and phoneme-level BERT [ 29 ] (PL BERT). They have primarily served as the phoneme encoder of TTS models, where the included semantic information contributes to the naturalness of synthesized speech. However, their potential applications in other domains remain unexplored. The unique design of pre-training objectives, which enrich generated representations with both word and phoneme information, suggests that these phoneme-level PLMs could be utilized in phrasing tasks to validate our hypothesis. In our previous work [ 30 ] , we proposed a speaker-conditioned phrasing model and a duration-aware pause insertion model. In this paper, we focus on the speaker-conditioned phrasing model. We first made several critical improvements: 1. Dataset : We construct the phrasing dataset based on the LibriTTS-R corpus [ 31 ] , facilitating pause detection and speech synthesis due to its higher sound quality (Section 4.1 ). 2. Model and training : To address the overfitting problem and instability during training, we improve the architecture of both the baseline and proposed models and optimize the training configurations (Section 3 , Section 4.2 , and Section 4.5 ). 3. Synthesis quality : In the subjective evaluations of our previous work, we found FastSpeech 2 [ 5 ] with HiFi-GAN [ 12 ] struggled to synthesize breath sounds, which are important components of pauses and significantly affect the listeners’ perception. Therefore, we instead use both VITS [ 7 ] and Matcha-TTS [ 9 ] (see A for more details) as the backbone TTS models, which offer natural breath synthesis and benefit the evaluation. Then, we extend our previous work with the following contributions: 1. Exploration of PSVM embeddings : We incorporate speaker embeddings from various PSVMs into our proposed phrasing models. This exploration aims to determine whether the prosody and rhythm features encoded in these embeddings can be leveraged in phrasing tasks (Section 5.1.2 ). 2. Leverage of various PLMs : We experiment with advanced subword-level PLMs

Read the complete text