TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet

2025

Read TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04349.txt.

Introduction

Jaeseok Jeong and co-authors present a way to add feeling to machine-made speech in this 2025 paper. Their system, TTS-CtrlNet, builds on a text-to-speech model that turns written words into spoken voice. Unlike older tools, it lets emotion shift over the course of a single spoken line.

The method borrows a trick from image software: keep the base model frozen and train a copy to handle the new control signal. That way the voice keeps its natural sound and its skill at copying a new speaker. The team shares design choices that made the control work well. Tests show the speech matches target feelings more closely than prior systems.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04349

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet Abstract Recent advances in text-to-speech (TTS) have enabled natural speech synthesis, but fine-grained, time-varying emotion control remains challenging. Existing methods often allow only utterance-level control, require full model fine-tuning with a large emotion speech dataset, which can degrade performance. Inspired by [ 1 ] , we propose the first ControlNet-based approach for controlling flow-matching TTS (TTS-CtrlNet), which freezes the original model and makes a trainable copy of it to process additional condition. We show that TTS-CtrlNet can boost the pretrained large TTS model by adding intuitive, scalable, and time-varying emotion control while inheriting the ability of the original model (e.g., zero-shot voice cloning & naturalness). Furthermore, we provide practical recipes for adding emotion control: 1) optimal architecture design choice with block analysis, 2) emotion-specific flow step, and 3) flexible control scale. Experiments show that ours can effectively adds an emotion controller to existing TTS, and achieves state-of-the-art performance with emotion similarity scores: Emo-SIM and Aro-Val SIM. Project page is available at: https://curryjung.github.io/ttsctrlnet_project_page 1 Introduction Recent advances in text-to-speech (TTS) technology have enabled highly natural and expressive speech synthesis, allowing users to generate lifelike utterances from arbitrary text [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . As these systems mature, user demand for greater controllability, especially over emotional expression, has grown rapidly. While various approaches have been proposed to condition TTS on emotion, most existing methods [ 2 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ] support only utterance-level control and lack the ability to finely modulate time-varying emotional cues within an utterance. This limitation is particularly critical in applications such as speech-to-speech translation, where accurate emotional transfer aligned over time is essential. In addition, most of them are trained on a limited number of speakers and samples (in an extreme case, only 1 speaker) due to the high cost of collecting a labeled emotion speech dataset, resulting in a lack of control over emotions with arbitrary speakers. This significantly hinders their practical applicability. Wu et al. 2024 shows remarkable success in time-varying emotion conditioning by fine-tuning pre-trained large-scale TTS. However, it 1) requires a huge training cost while fine-tuning the entire pre-trained model with an 87k hours of large-scale speech dataset, which includes 27k hours of in-house emotion dataset, 2) compromises the performance of the pre-trained model, and 3) cannot intuitively adjust emotion intensity. To address the problem, we first adapt the ControlNet paradigm to flow-matching-based TTS in order to synthesize time-varying emotions of any zero-shot speaker. Inspired by Zhang et al. 2023 , we freeze the original large-scale pre-trained model and utilize it as an encoder to leverage its deep prior knowledge in speech synthesis. The original model and its trainable copy are connected with zero-convolution. It minimizes damage during fine-tuning by progressively increasing the controlling features from a zero tensor. By training the control network while freezing the backbone TTS model, our approach preserves the ability of the original model and enables emotion control with a relatively small amount of unlabeled speech data ( \sim 400 hours) compared to the data used for the original model. In addition, we provide practical recipes for effective emotion control: 1) analyzing layer-wise impact on WER, 2) identifying optimal flow step interval for emotion-specific condition, and 3) using the proper temporal window required to capture emotion and 3) flexible interpolation between behaviors of the original model and ControlNet-applied version using a control scale. In conclusion, we show that it is possible to efficiently and effectively incorporate desired control capabilities into a large pre-trained system without modifying its original parameters, and without requiring extensive computational resources or costly labeled data. Our contributions are as follows. • We first introduce a ControlNet-based conditioning method for large-scale & flow-matching TTS, enabling fine-grained, time-varying emotion control with zero-shot TTS. • Our approach preserves the quality, zero-shot capability and naturalness of the original TTS model by freezing its parameters during the control network training with careful design choices. • We propose effective training and inference strategies to maximize emotional expressiveness with 1) minimal computational cost and 2) a balance between word error and emotion control. 2 Related work 2.1 Emotion-controllable text-to-speech There have been a number of zero-shot TTS models [ 22 , 5 , 3 ] that enable voice cloning using a short reference audio clip. During the voice cloning stage, these models can transfer the emotion expressed in the reference audio, allowing for natural emotion control. [ 20 ] achieves emotion control by incorporating an additional emotion condition into the model. While previous studies have demonstrated the ability to control emotion at the utterance level, achieving fine-grained, time-varying emotion control remains a significant challenge. [ 21 ] supports time-varying emotional control, it requires full model finetuning, which incurs a substantial computational cost. 2.2 Controlling large-scale generative model Both flow-matching & diffusion models are a family of score-based generative models which synthesize the target distribution by iteratively denoising random Gaussian noise. Accompanied by diffusion transformer [ 23 , 24 , 25 ] , they achieve huge success in scaling the generative model in both image [ 23 , 26 , 27 , 28 , 29 ] and audio modalities, enabling product-level models. Based on the success, [ 1 , 30 ] propose to add controllability to the pre-traineded model while leveraging the prior of the model with fine-tuning. [ 1 ] chooses a carefully designed conditional network (ControlNet), by making a trainable copy only at downblocks of the original UNet and connecting it with the original branch through zero-convolution. It leads to huge success in spatial control. On the other hand, [ 31 ] analyzes the influence of each layer in a DiT trained via flow matching, identifies the vital layers, and demonstrates the possibility of training-free editing during inference. 2.3 Flow matching Flow matching [ 32 ] learns a vector field \mathbf{v}_{t}(\mathbf{x};\theta) which transfer the known distribution (e.g., Gaussian noise) to the target distribution p_{1} , approximating the data distribution q with a neural network \theta{} . In conditional flow matching (CFM), we have flow map \boldsymbol{\psi}_{t}(\mathbf{x})=\sigma_{t}(\mathbf{x}_{1})x+\mu_{t}(\mathbf{x}_{1}) conditioned on \mathbf{x}_{1} . With \mu_{0}(\mathbf{x}_{1})=0 and \sigma_{0}(\mathbf{x}_{1})=1 , \mu_{1}(\mathbf{x}_{1})=\mathbf{x}_{1} and \sigma_{1}(\mathbf{x}_{1})=0 , a vector field of the flow is d\boldsymbol{\psi}_{t}(\mathbf{x}_{0})/dt=\mathbf{u}_{t}\left(\boldsymbol{\psi}_{t}(\mathbf{x}_{0})\mid\mathbf{x}_{1}\right) . Then, CFM loss becomes : \mathcal{L}_{\text{CFM}}(\theta)=\mathbb{E}_{t,\,q(\mathbf{x}_{1}),\,p(\mathbf{x}_{0})}\left\|\mathbf{v}_{t}(\boldsymbol{\psi}_{t}(\mathbf{x}_{0}))-\frac{d}{dt}\boldsymbol{\psi}_{t}(\mathbf{x}_{0})\right\|^{2}. (1) Optimal transport further provides the OT-CFM loss: \mathcal{L}_{\text{CFM}}(\theta)=\mathbb{E}_{t,\,q(\mathbf{x}_{1}),\,p(\mathbf{x}_{0})}\left\|\mathbf{v}_{t}\left((1-t)\mathbf{x}_{0}+t\,\mathbf{x}_{1}\right)-(\mathbf{x}_{1}-\mathbf{x}_{0})\right\|^{2}, (2) where the flow \boldsymbol{\psi}_{t}(\mathbf{x}_{0}) is interpolation between \mathbf{x}_{0} and \mathbf{x}_{1} with flow step t\in[0,1] . With the learned \mathbf{v}t(\cdot;\theta) , we synthesize x_{1} from randomly sampled \mathbf{x}_{0}\sim N(0,I) by solving the ordinary differential equation (ODE) with the predicted flow d\boldsymbol{\psi}_{t}(\mathbf{x}_{0})/dt . \mathbf{x}_{0} is progressively transferred into \mathbf{x}_{1} with the number of function evaluations (NFE) from 0 to 1. 2.4 Speech emotion recognition Speech Emotion Recognition (SER) has received increasing attention in recent years, with diverse modeling approaches proposed to better capture emotional cues from speech [ 33 , 34 , 35 ] . One prominent line of work [ 36 ] leverages self-supervised learning to pretrain models on large-scale, unlabeled speech corpora. Models like wav2vec 2.0 [ 24 ] are trained to learn contextualized audio representations

Read the complete text