Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis
Intelligence, Computing, Tianjin University, Nanyang Technological University · 2025
Read Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis by Intelligence, Computing, Tianjin University, Nanyang Technological University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24629.txt.
Introduction
Synthetic voices can sound happy or sad for a whole sentence, but shifting emotion word by word inside one sentence is far harder. Tianrui Wang and 12 co-authors built WeSCon, a self-training framework for word-level control of emotion and speaking rate in a pre-trained zero-shot speech model. No data set with in-sentence emotion transitions was needed, which has been the field's bottleneck.
A transition-smoothing strategy and a dynamic speed control guide the model through multi-round inference. Self-training then folds that skill into the model end to end, with a dynamic emotional attention bias to simplify use. Results reached the top level for word-level expressive control while keeping zero-shot quality.
Catalog record
| CREATOR | Intelligence, Computing, Tianjin University, Nanyang Technological University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-24629
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis Thanks: Corresponding Author. This work was supported by the National Natural Science Foundation of China under Grant U23B2053 and Grant 62176182. Affiliation: Tianrui Wang, Haoyu Wang, Meng Ge, Cheng Gong, Chunyu Qiang, Affiliation: Ziyang Ma, Zikang Huang, Guanrou Yang, Xiaobao Wang, Affiliation: Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence, Computing, Tianjin University, Nanyang Technological University Abstract While emotional text-to-speech (TTS) has made significant progress, most existing research remains limited to utterance-level emotional expression and fails to support word-level control. Achieving word-level expressive control poses fundamental challenges, primarily due to the complexity of modeling multi-emotion transitions and the scarcity of annotated datasets that capture intra-sentence emotional and prosodic variation. In this paper, we propose WeSCon, the first self-training framework that enables word-level control of both emotion and speaking rate in a pretrained zero-shot TTS model, without relying on datasets containing intra-sentence emotion or speed transitions. Our method introduces a transition-smoothing strategy and a dynamic speed control mechanism to guide the pretrained TTS model in performing word-level expressive synthesis through a multi-round inference process. To further simplify the inference, we incorporate a dynamic emotional attention bias mechanism and fine-tune the model via self-training, thereby activating its ability for word-level expressive control in an end-to-end manner. Experimental results show that WeSCon effectively overcomes data scarcity, achieving state-of-the-art performance in word-level emotional expression control while preserving the strong zero-shot synthesis capabilities of the original TTS model. 1 Introduction Humans possess the ability to regulate emotional expression during speech flexibly scherer1995expression . To simulate this expressive capability, recent advances in text-to-speech synthesis (TTS) have increasingly focused on controllable generation of various aspects of speech, such as timbre, emotion, and speaking rate xie2024towards . Such control is a key objective in the development of human-like and expressive TTS. Most current TTS models exhibit zero-shot capabilities, enabling them to synthesize speech from text while cloning attributes such as timbre, emotion, and speaking rate from a reference speech sample kharitonov2023speak ; wang2023neural ; ju2024naturalspeech . Despite these advances, as shown in Figure 1 , emotional and speaking rate control in current models is typically limited to the utterance level. Figure 1: Word-level control of emotion and speaking rate aims to modulate both attributes within an utterance, guided by multiple emotional prompts and emotion-speed-tagged text. Our approach, WeSCon, achieves this using only a small-scale public dataset without emotion transitions. This differs significantly from how humans naturally express emotion in speech. Unlike global speaker identity, emotional expression and speaking rate are dynamic and often vary within a single sentence mozziconacci2002prosody ; guan2018speech . Therefore, word-level control of these factors is essential for achieving more natural and expressive speech synthesis 9746323 . To address this limitation, some approaches have proposed phoneme-level emotion prediction from target text to guide expressive synthesis tang2024ed ; 9383524 ; du21b_interspeech . While these methods show potential for word-level emotion control, relying solely on text makes it difficult to capture essential acoustic cues such as prosody and intensity, which are vital to emotional expression control chen2022fine ; chandra2024exploring ; zhao2023emotion . To address this limitation, recent studies such as ELaTE kanda2024making and EmoCtrl-TTS wu2024laugh have demonstrated that reference speech with emotional content can support intra-utterance control of time-varying expressive patterns, such as transitions from laughter to crying. These works reflect a growing interest in TTS with word-level control over both emotion and speaking rate, but they also underscore several fundamental challenges. First, word-level expression control requires multiple emotional speech prompts, which introduces the challenge of guiding the model to attend to the appropriate emotion at each word. In addition, current methods for fine-grained expression control rely on large-scale emotional speech datasets with time-aligned emotion transitions. However, such datasets are limited in both scale and accessibility rathi2024analyzing , making fine-grained control even more difficult to realize in practice. These challenges lead us to ask: Is it possible to achieve effective word-level control of both emotion and speaking rate without relying on speech datasets containing emotion or speed transitions? In this work, motivated by the zero-shot potential of pretrained TTS models, we propose WeSCon, a two-stage self-training framework that achieves W ord-level E motion and S peed Con trol for TTS using only a small amount of public speech data without emotion or speed transitions. In the first stage, we design a multi-round inference framework that incorporates a transition-smoothing module and a dynamic speed control mechanism. Without relying on any emotional training data, this approach enables a pretrained zero-shot TTS model to perform high-quality word-level emotional expression control in TTS. In the second stage, the original TTS model is repurposed as a student and trained under the supervision of the 1st-stage teacher. A dynamic emotional attention bias is introduced, enabling the student to acquire word-level control of emotion and speed through a simplified end-to-end inference process, without the need for complex iterative generation or smoothing. Experimental results show that WeSCon achieves state-of-the-art performance on the task of word-level emotional expression control in TTS, while preserving the zero-shot generalization and generation capabilities of the pretrained TTS model. Our contributions are summarized as follows: • We propose a multi-round inference mechanism equipped with transition smoothing and dynamic speaking rate control, which is the first to achieve word-level control of both emotion and speaking rate in TTS without relying on any emotional training data. • We further introduce a novel self-training framework with a dynamic emotional attention bias mechanism that empowers a pretrained TTS model with end-to-end word-level emotion and speaking rate control, using limited data without intra-sentence emotion or speed transitions. • We conduct comprehensive experiments to validate the effectiveness of our proposed framework. Results show that our method enables a pretrained zero-shot TTS model to achieve SOTA performance in word-level emotional expression control, while preserving its original zero-shot capabilities. Ablation studies further confirm the contribution of each key design component. Our samples are available at https://anonymousdemo999.github.io/ . 2 Related Work Scarcity of Emotional Dataset The development of controllable TTS, particularly for emotional expression control, depends heavily on high-quality emotional speech datasets zhu2024metts_emodata ; guo2023emodiff_emodata ; yang2025emovoicellmbasedemotionaltexttospeech_emodata . While public corpora such as ESD zhou2021seen , IEMOCAP busso2008iemocap , and CREMA-D cao2014crema are available, they primarily provide utterance-level annotations and lack word-level or time-aligned emotional labels. These datasets are also limited in size and diversity, often consisting of scripted speech and covering a narrow range of emotions and speakers ma24b_interspeech . More importantly, emotional datasets with intra-sentence variation, which are essential for learning word-level control, remain extremely scarce and are typically restricted to private use kanda2024making . Creating such datasets is expensive, requiring detailed word- or frame-level annotation and subjective emotional labeling liu2024emphasisrenderingconversationaltexttospeech . This lack of fine-grained emotional data poses a major challenge for training models capable of word-level expressive TTS. From Utterance-Level to Word-Level Controllability of Emotion and Speaking Rate Most controllable TTS systems support only utterance-level control, where a single label or reference speech governs the entire sentence anastassiou2024seed ; du2024cosyvoice . To achieve word-level control, some methods attempt to predict frame- or phoneme-level emotional indicators from text alone phone_level_tts0 ; phone_level_tts1 ; phone_level_tts2 , but they often fail to capture expressive variability due to the