FC-TTS: Style and Timbre Control in Zero-Shot Text-to-Speech with Disentangled Speech Representations

Yoonhyung Lee, Hyunsin Park, Jinhwan Park, Jinkyu Lee · 2026

Read FC-TTS: Style and Timbre Control in Zero-Shot Text-to-Speech with Disentangled Speech Representations by Yoonhyung Lee, Hyunsin Park, Jinhwan Park, Jinkyu Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24618.txt.

Introduction

Yoonhyung Lee and three colleagues present FC-TTS, a 2026 zero-shot speech system that controls speaking style and speaker timbre on their own from two voice clips. Earlier systems inherit limits from pre-trained disentangled representations, but FC-TTS adds design choices, training setup, and extra goals that make attribute separation more reliable. Tests show high-quality speech that sounds natural in zero-shot use. Only this system supports steady, independent control of style and timbre from those clips.

Catalog record

CREATORYoonhyung Lee, Hyunsin Park, Jinhwan Park, Jinkyu Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55691
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24618

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FC-TTS: Style and Timbre Control in Zero-Shot Text-to-Speech with Disentangled Speech Representations Thanks: Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Yoonhyung Lee, Hyunsin Park, Jinhwan Park, Jinkyu Lee Abstract Recent advances in zero-shot text-to-speech (TTS) have enabled accurate imitation of reference speech in terms of both speaking style and speaker timbre. However, achieving disentangled control over these aspects from separate references remains a challenging task. Several studies have proposed disentangled speech representations that decompose speech into interpretable attributes ( e.g. , timbre, prosody, and content), providing a promising foundation for TTS with attribute control from separate references. Yet, how to effectively integrate such representations into TTS systems to achieve independent and precise control remains underexplored. In this paper, we present FC-TTS, a zero-shot TTS framework that enables disentangled control of style and timbre by conditioning on two distinct reference utterances. Unlike existing systems that inherit limitations from those pre-trained disentangled representations, FC-TTS introduces key design strategies, including architectural choices, training framework, and auxiliary training objectives, which improve the reliability of attribute separation and dual-reference control. Experiments show that FC-TTS achieves high-fidelity synthesis and competitive zero-shot naturalness, while uniquely supporting consistent and independent manipulation of style and timbre. Audio samples are available at https://qualcomm-ai-research.github.io/fc-tts 1 Introduction With recent breakthroughs in text-to-speech (TTS) technology ( 18 ; 2 ; 3 ) , the focus has expanded from simply generating natural-sounding speech to enabling expressive and personalized synthesis for diverse applications such as virtual assistants, audiobooks, accessibility tools, and interactive media ( 42 ) . This evolution has brought a growing demand for fine-grained control over attributes such as speaking style and speaker timbre ( 2 ; 6 ) . To support such control, existing TTS systems have explored various strategies. Supervised models trained on labeled datasets ( 18 ; 5 ; 24 ) offer reliable conditioning but scale poorly due to costly annotations. In contrast, reference-based zero-shot approaches ( 1 ; 2 ; 20 ; 12 ; 3 ) offer greater flexibility by conditioning on example utterances, but typically entangle style and timbre in a single reference, hindering independent control. To overcome this limitation, recent studies have explored disentangled speech representations ( 7 ; 13 ) , leading to various TTS systems that exploit such factorization. In particular, these systems employ disentangled attributes together with jointly pre-trained decoders to enable more interpretable control and, in principle , the possibility of conditioning on style and timbre separately. Nevertheless, disentanglement is often imperfect in practice, and the decoders inherently bound synthesis quality, with no guarantee of robustness to unseen style-timbre combinations. In this paper, we investigate how to effectively leverage disentangled speech representations to build TTS models that support separate control over style and timbre using distinct references. To this end, we present FC-TTS, which leverages factorized speech representations but adds: (1) a two-stage spectrogram generation pipeline (timbre-conditioned blurry spectrogram, then prosody refinement) for robustness to unseen combinations; (2) a VQ-VAE-based style encoder that captures fine-grained and intra-utterance style variability; and (3) a conditioning-aware consistency loss that extends conventional regularization to multi-condition settings by enforcing joint coherence across timbre and style, providing more precise guidance for disentangled control. In our experiments, we show that FC-TTS delivers competitive zero-shot TTS performance compared to state-of-the-art models that lack explicit support for separate control of style and timbre, as evaluated by objective metrics including UTMOS ( 37 ) , word error rate, and speaker similarity. Furthermore, we conduct detailed evaluations on the RAVDESS dataset ( 30 ) —a highly expressive emotional speech corpus—focusing on both timbre and prosody controllability through both objective and subjective evaluations. Specifically, for timbre control, we compare FC-TTS with a factorized codec-based system ( 13 ) , and for prosody control, we compare against a state-of-the-art zero-shot TTS model, F5-TTS ( 3 ) . These evaluations highlight that FC-TTS not only maintains high synthesis quality but also enables precise and independent manipulation of style and timbre, which existing zero-shot systems do not explicitly support. 2 Related Work 2.1 Conditional Text-to-Speech Conditional TTS incorporates auxiliary signals to control speaker or style attributes, enabling expressive and personalized speech. A common approach is to use labeled datasets with speaker or style annotations ( 18 ; 5 ) , which support multi-condition models capable of controlling multiple attributes simultaneously ( 27 ; 28 ; 15 ) . More recently, prompt-guided methods ( 24 ; 11 ) have emerged, where models interpret free-form descriptions of style or tone. While these approaches offer disentangled controllability and flexibility, these approaches still rely heavily on annotation coverage and granularity, limiting scalability in open-domain or fine-grained control settings. 2.2 Disentangled Speech Representation Learning A complementary direction focuses on learning disentangled speech representations that disentangle speech into interpretable components ( e.g. , timbre, pitch, or linguistic content), opening promising directions for developing controllable TTS systems without labeled datasets. NANSY++ ( 7 ) leverages various information perturbation functions to isolate pitch, linguistic, and timbre features. FACodec ( 13 ) enforces strong information bottlenecks and uses supervision from phoneme and speaker labels to learn factorized representations. LSCodec ( 10 ) further improves timbre isolation via speaker perturbation and achieves strong results in voice conversion. Despite progress, most systems are trained in autoencoding setups with paired inputs and targets, meaning they are primarily optimized for naturally co-occurring factors. As a result, their ability to generalize to mismatched conditioning—such as combining timbre and prosody from different references—remains underexplored. 2.3 Reference Speech-based Style and Timbre Control Recently, there have been studies to control the style and timbre of TTS models based on two reference samples. For instance, IndexTTS 2 ( 45 ) achieves this by relying on the disentanglement property of semantic codec representations derived from MaskGCT ( 41 ) . EmoSphere++ ( 6 ) trains a dedicated emotion encoder and introduces a regularization loss to enforce orthogonality between emotion and timbre vectors. Although promising, these methods rely on representations with only empirically observed disentanglement, or on pre-trained encoders that may discard other attributes such as accent. In contrast, our approach builds on systematically factorized representations that preserve reconstruction quality while enabling explicit decomposition, providing a more reliable basis for separate style and timbre control. Figure 1: FC-TTS architecture. First stage : a phoneme sequence \bm{y} is conditioned on timbre embedding z_{\text{spk}} via the timbre adapter to generate a blurry log-mel spectrogram \bm{h} , anchoring timbre characteristics. Second stage : \bm{h} is refined into a clean spectrogram \hat{\bm{x}} by a flow-matching decoder conditioned on style embedding \bm{z}_{\text{sty}} , obtained from prosody tokens \mathbf{c_{p}} through hierarchical TCF modules, imprinting prosodic characteristics. During training, target speech provides both timbre and style references; at inference, separate references enable disentangled attribute control. Blue modules indicate trainable components. 3 Methodology 3.1 Preliminaries 3.1.1 Factorized Speech Codec To build a timbre and style controllable TTS system, we adopt FACodec 1 1 1 https://github.com/open-mmlab/Amphion/tree/main/models/codec/ns3_codec ( 13 ) , a neural speech codec that decomposes speech into interpretable components while supporting faithful reconstruction. It factorizes a speech signal into multiple disentangled streams of discrete tokens, each capturing a distinct speech attribute: prosody tokens \mathbf{c_{p}} , content tokens \mathbf{c_{c}} , and acoustic detail tokens \mathbf{c_{d}} . Each stream is represented as \mathbb{Z}^{N_{*}\times T} ,

Read the complete text