TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet
2025
Read TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04349.txt.
Introduction
Jaeseok Jeong and co-authors present a way to add feeling to machine-made speech in this 2025 paper. Their system, TTS-CtrlNet, builds on a text-to-speech model that turns written words into spoken voice. Unlike older tools, it lets emotion shift over the course of a single spoken line.
The method borrows a trick from image software: keep the base model frozen and train a copy to handle the new control signal. That way the voice keeps its natural sound and its skill at copying a new speaker. The team shares design choices that made the control work well. Tests show the speech matches target feelings more closely than prior systems.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-04349
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet Abstract Recent advances in text-to-speech (TTS) have enabled natural speech synthesis, but fine-grained, time-varying emotion control remains challenging. Existing methods often allow only utterance-level control, require full model fine-tuning with a large emotion speech dataset, which can degrade performance. Inspired by [ 1 ] , we propose the first ControlNet-based approach for controlling flow-matching TTS (TTS-CtrlNet), which freezes the original model and makes a trainable copy of it to process additional condition. We show that TTS-CtrlNet can boost the pretrained large TTS model by adding intuitive, scalable, and time-varying emotion control while inheriting the ability of the original model (e.g., zero-shot voice cloning & naturalness). Furthermore, we provide practical recipes for adding emotion control: 1) optimal architecture design choice with block analysis, 2) emotion-specific flow step, and 3) flexible control scale. Experiments show that ours can effectively adds an emotion controller to existing TTS, and achieves state-of-the-art performance with emotion similarity scores: Emo-SIM and Aro-Val SIM. Project page is available at: https://curryjung.github.io/ttsctrlnet_project_page 1 Introduction Recent advances in text-to-speech (TTS) technology have enabled highly natural and expressive speech synthesis, allowing users to generate lifelike utterances from arbitrary text [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . As these systems mature, user demand for greater controllability, especially over emotional expression, has grown rapidly. While various approaches have been proposed to condition TTS on emotion, most existing methods [ 2 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ] support only utterance-level control and lack the ability to finely modulate time-varying emotional cues within an utterance. This limitation is particularly critical in applications such as speech-to-speech translation, where accurate emotional transfer aligned over time is essential. In addition, most of them are trained on a limited number of speakers and samples (in an extreme case, only 1 speaker) due to the high cost of collecting a labeled emotion speech dataset, resulting in a lack of control over emotions with arbitrary speakers. This significantly hinders their practical applicability. Wu et al. 2024 shows remarkable success in time-varying emotion conditioning by fine-tuning pre-trained large-scale TTS. However, it 1) requires a huge training cost while fine-tuning the entire pre-trained model with an 87k hours of large-scale speech dataset, which includes 27k hours of in-house emotion dataset, 2) compromises the performance of the pre-trained model, and 3) cannot intuitively adjust emotion intensity. To address the problem, we first adapt the ControlNet paradigm to flow-matching-based TTS in order to synthesize time-varying emotions of any zero-shot speaker. Inspired by Zhang et al. 2023 , we freeze the original large-scale pre-trained model and utilize it as an encoder to leverage its deep prior knowledge in speech synthesis. The original model and its trainable copy are connected with zero-convolution. It minimizes damage during fine-tuning by progressively increasing the controlling features from a zero tensor. By training the control network while freezing the backbone TTS model, our approach preserves the ability of the original model and enables emotion control with a relatively small amount of unlabeled speech data ( \sim 400 hours) compared to the data used for the original model. In addition, we provide practical recipes for effective emotion control: 1) analyzing layer-wise impact on WER, 2) identifying optimal flow step interval for emotion-specific condition, and 3) using the proper temporal window required to capture emotion and 3) flexible interpolation between behaviors of the original model and ControlNet-applied version using a control scale. In conclusion, we show that it is possible to efficiently and effectively incorporate desired control capabilities into a large pre-trained system without modifying its original parameters, and without requiring extensive computational resources or costly labeled data. Our contributions are as follows. • We first introduce a ControlNet-based conditioning method for large-scale & flow-matching TTS, enabling fine-grained, time-varying emotion control with zero-shot TTS. • Our approach preserves the quality, zero-shot capability and naturalness of the original TTS model by freezing its parameters during the control network training with careful design choices. • We propose effective training and inference strategies to maximize emotional expressiveness with 1) minimal computational cost and 2) a balance between word error and emotion control. 2 Related work 2.1 Emotion-controllable text-to-speech There have been a number of zero-shot TTS models [ 22 , 5 , 3 ] that enable voice cloning using a short reference audio clip. During the voice cloning stage, these models can transfer the emotion expressed in the reference audio, allowing for natural emotion control. [ 20 ] achieves emotion control by incorporating an additional emotion condition into the model. While previous studies have demonstrated the ability to control emotion at the utterance level, achieving fine-grained, time-varying emotion control remains a significant challenge. [ 21 ] supports time-varying emotional control, it requires full model finetuning, which incurs a substantial computational cost. 2.2 Controlling large-scale generative model Both flow-matching & diffusion models are a family of score-based generative models which synthesize the target distribution by iteratively denoising random Gaussian noise. Accompanied by diffusion transformer [ 23 , 24 , 25 ] , they achieve huge success in scaling the generative model in both image [ 23 , 26 , 27 , 28 , 29 ] and audio modalities, enabling product-level models. Based on the success, [ 1 , 30 ] propose to add controllability to the pre-traineded model while leveraging the prior of the model with fine-tuning. [ 1 ] chooses a carefully designed conditional network (ControlNet), by making a trainable copy only at downblocks of the original UNet and connecting it with the original branch through zero-convolution. It leads to huge success in spatial control. On the other hand, [ 31 ] analyzes the influence of each layer in a DiT trained via flow matching, identifies the vital layers, and demonstrates the possibility of training-free editing during inference. 2.3 Flow matching Flow matching [ 32 ] learns a vector field \mathbf{v}_{t}(\mathbf{x};\theta) which transfer the known distribution (e.g., Gaussian noise) to the target distribution p_{1} , approximating the data distribution q with a neural network \theta{} . In conditional flow matching (CFM), we have flow map \boldsymbol{\psi}_{t}(\mathbf{x})=\sigma_{t}(\mathbf{x}_{1})x+\mu_{t}(\mathbf{x}_{1}) conditioned on \mathbf{x}_{1} . With \mu_{0}(\mathbf{x}_{1})=0 and \sigma_{0}(\mathbf{x}_{1})=1 , \mu_{1}(\mathbf{x}_{1})=\mathbf{x}_{1} and \sigma_{1}(\mathbf{x}_{1})=0 , a vector field of the flow is d\boldsymbol{\psi}_{t}(\mathbf{x}_{0})/dt=\mathbf{u}_{t}\left(\boldsymbol{\psi}_{t}(\mathbf{x}_{0})\mid\mathbf{x}_{1}\right) . Then, CFM loss becomes : \mathcal{L}_{\text{CFM}}(\theta)=\mathbb{E}_{t,\,q(\mathbf{x}_{1}),\,p(\mathbf{x}_{0})}\left\|\mathbf{v}_{t}(\boldsymbol{\psi}_{t}(\mathbf{x}_{0}))-\frac{d}{dt}\boldsymbol{\psi}_{t}(\mathbf{x}_{0})\right\|^{2}. (1) Optimal transport further provides the OT-CFM loss: \mathcal{L}_{\text{CFM}}(\theta)=\mathbb{E}_{t,\,q(\mathbf{x}_{1}),\,p(\mathbf{x}_{0})}\left\|\mathbf{v}_{t}\left((1-t)\mathbf{x}_{0}+t\,\mathbf{x}_{1}\right)-(\mathbf{x}_{1}-\mathbf{x}_{0})\right\|^{2}, (2) where the flow \boldsymbol{\psi}_{t}(\mathbf{x}_{0}) is interpolation between \mathbf{x}_{0} and \mathbf{x}_{1} with flow step t\in[0,1] . With the learned \mathbf{v}t(\cdot;\theta) , we synthesize x_{1} from randomly sampled \mathbf{x}_{0}\sim N(0,I) by solving the ordinary differential equation (ODE) with the predicted flow d\boldsymbol{\psi}_{t}(\mathbf{x}_{0})/dt . \mathbf{x}_{0} is progressively transferred into \mathbf{x}_{1} with the number of function evaluations (NFE) from 0 to 1. 2.4 Speech emotion recognition Speech Emotion Recognition (SER) has received increasing attention in recent years, with diverse modeling approaches proposed to better capture emotional cues from speech [ 33 , 34 , 35 ] . One prominent line of work [ 36 ] leverages self-supervised learning to pretrain models on large-scale, unlabeled speech corpora. Models like wav2vec 2.0 [ 24 ] are trained to learn contextualized audio representations