Characteristic-Specific Partial Fine-Tuning for Efficient Emotion and Speaker Adaptation in Codec Language Text-to-Speech Models Journal: Speech Communication, * is Corresponding author

Tianrui Wang 1,2 , Meng Ge 1 , Cheng Gong 3 , Chunyu Qiang 1 , Haoyu Wang 1 , Zikang Huang 1 , Yu Jiang 1 Xiaobao Wang 1,2 , Xie Chen 4 , Longbiao Wang 1,5,∗ , Jianwu Dang 6 · 2025

Read Characteristic-Specific Partial Fine-Tuning for Efficient Emotion and Speaker Adaptation in Codec Language Text-to-Speech Models Journal: Speech Communication, * is Corresponding author by Tianrui Wang 1,2 , Meng Ge 1 , Cheng Gong 3 , Chunyu Qiang 1 , Haoyu Wang 1 , Zikang Huang 1 , Yu Jiang 1 Xiaobao Wang 1,2 , Xie Chen 4 , Longbiao Wang 1,5,∗ , Jianwu Dang 6 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14273.txt.

Introduction

Tianrui Wang and co-authors proposed in 2025 a surgical way to fine-tune speech synthesis models: adjust only the two layers that matter most for emotion and voice identity. Full fine-tuning on small data is slow and erases old skills. Their method scores every layer's contribution, then trains the strongest layer and the weakest one together.

Across a combined corpus of eleven datasets, the partial approach matched or beat full tuning on sound quality and clarity. It updated only about eight percent of the weights, trained about twice as fast, and forgot far less of the base model's skill. The authors call the trick characteristic-specific partial fine-tuning. They argue adaptation should be a scalpel, not a bath. The recipe suits any large speech model facing a new voice or mood.

Catalog record

CREATORTianrui Wang 1,2 , Meng Ge 1 , Cheng Gong 3 , Chunyu Qiang 1 , Haoyu Wang 1 , Zikang Huang 1 , Yu Jiang 1 Xiaobao Wang 1,2 , Xie Chen 4 , Longbiao Wang 1,5,∗ , Jianwu Dang 6
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14273

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Characteristic-Specific Partial Fine-Tuning for Efficient Emotion and Speaker Adaptation in Codec Language Text-to-Speech Models Journal: Speech Communication, * is Corresponding author Tianrui Wang 1,2 , Meng Ge 1 , Cheng Gong 3 , Chunyu Qiang 1 , Haoyu Wang 1 , Zikang Huang 1 , Yu Jiang 1 Xiaobao Wang 1,2 , Xie Chen 4 , Longbiao Wang 1,5,∗ , Jianwu Dang 6 Abstract Recently, emotional speech generation and speaker cloning have garnered significant interest in text-to-speech (TTS). With the open-sourcing of codec language TTS models trained on massive datasets with large-scale parameters, adapting these general pre-trained TTS models to generate speech with specific emotional expressions and target speaker characteristics has become a topic of great attention. Common approaches, such as full and adapter-based fine-tuning, often overlook the specific contributions of model parameters to emotion and speaker control. Treating all parameters uniformly during fine-tuning, especially when the target data has limited content diversity compared to the pre-training corpus, results in slow training speed and an increased risk of catastrophic forgetting. To address these challenges, we propose a characteristic-specific partial fine-tuning strategy, short as CSP-FT . First, we use a weighted-sum approach to analyze the contributions of different Transformer layers in a pre-trained codec language TTS model for emotion and speaker control in the generated speech. We then selectively fine-tune the layers with the highest and lowest characteristic-specific contributions to generate speech with target emotional expression and speaker identity. Experimental results demonstrate that our method achieves performance comparable to, or even surpassing, full fine-tuning in generating speech with specific emotional expressions and speaker identities. Additionally, CSP-FT delivers approximately 2× faster training speeds, fine-tunes only around 8% of parameters, and significantly reduces catastrophic forgetting. Furthermore, we show that codec language TTS models perform competitively with self-supervised models in speaker identification and emotion classification tasks, offering valuable insights for developing universal speech processing models. 1 Introduction With advancements in text-to-speech (TTS) technology, the focus has expanded beyond achieving high content intelligibility to include emotional expression and speaker cloning ( Triantafyllopoulos et al. 2023 ; Qin et al. 2023 ). To improve task support and model performance, TTS models are being developed with increasingly larger parameters and trained on larger datasets ( Lyth and King 2024 ; Peng et al. 2024 ). As many large-scale general codec language TTS models are now open-sourced, adapting them to generate speech that meets specific emotional expression and speaker cloning requirements has become a growing area of research interest ( Moss et al. 2020 ; Neekhara et al. 2021 ). Adapting general pre-trained TTS models to generate speech with specific emotional expression and speaker identity is a domain adaptation (DA) problem ( Singhal et al. 2023 ). Based on the availability of target domain data, domain adaptation methods can be categorized into three types: data-scarce, task-irrelevant-data, and target-data-available DA ( Singhal et al. 2023 ). Data-scarce DA, such as few-shot DA, focuses on adapting a pre-trained model to a specific task using only a few examples. This is typically achieved through in-context learning or fine-tuning with limited data ( Choi et al. 2020 ; Huang et al. 2022 ). On the other hand, task-irrelevant-data DA, including zero-shot DA, relies on the pre-trained model’s inherent generalization capabilities to adapt to new tasks without fine-tuning ( Xin et al. 2024 ; Casanova et al. 2022 ; Wang et al. 2023a ). However, both few-shot and zero-shot strategies often suffer from performance instability due to the limited availability of target-domain data ( Singhal et al. 2023 ). In contrast, target-data-available DA employs supervised fine-tuning with labeled target domain data. By aligning the model with the specific target domain, this approach consistently delivers stable and high-performance results on target tasks ( Motiian et al. 2017 ; Neekhara et al. 2021 ). Currently, many general-purpose codec language TTS models have been open-sourced. While these models demonstrate zero-shot capability to generate speech with prompted emotion and speaker identity, their performance is often unstable, particularly when dealing with unseen speakers or unseen emotional expressions ( Zhang et al. 2023 ). In contrast, fine-tuning these pre-trained models with a small amount of data that includes specific emotional expressions or speaker characteristics has been shown to significantly improve their ability to control emotion and speaker identity in speech generation, offering a more stable and reliable performance ( Huang et al. 2024 ; Lou et al. 2024 ). Fine-tuning methods can be broadly categorized into two types: full fine-tuning ( Jain et al. 2022 ; Inoue et al. 2020 ; Yang et al. 2021 ) and parameter-efficient fine-tuning ( Hu et al. 2022 ; Houlsby et al. 2019 ; Wang et al. 2023b ). Full fine-tuning continually trains a pre-trained model on the target dataset with a small learning rate for a few steps ( Moss et al. 2020 ), thereby adapting the entire model’s parameters to the target task or data domain. In contrast, parameter-efficient fine-tuning freezes all pre-trained parameters and only fine-tunes lightweight inserted parameters ( Hu et al. 2022 ). This approach is often employed for large parameter models, where full fine-tuning may require unaffordable computational resources. While full fine-tuning can demonstrate superior domain adaptation performance when computational resources permit, it is prone to overfitting the target data domain, which can result in catastrophic forgetting ( Chen and Garner 2024 ). Therefore, effective learning of target knowledge, along with resource consumption and catastrophic forgetting, are key challenges in target-data-available DA, which also hinder the practical deployment and application of open-sourced codec language TTS models. Effective learning of target knowledge is the primary goal of fine-tuning, and is often considered alongside the two challenges of resource consumption and catastrophic forgetting. The solution to the issue of resource consumption is straightforward: under the premise of ensuring effective learning of target knowledge, we need to minimize the number of parameters involved in the training process as much as possible. On the other hand, catastrophic forgetting, sometimes referred to as knowledge forgetting, occurs when a model loses a significant portion of the knowledge it acquired during pre-training after being fine-tuned for a specific target domain ( McCloskey and Cohen 1989 ; French 1999 ; Goodfellow et al. 2013 ). This degradation severely compromises the model’s ability to generalize to fine-tuning-unseen data, for example, in TTS tasks, fine-tuning a model for a specific speaker can impair other critical capabilities, such as emotion control and word accuracy ( Fan et al. 2015 ). To mitigate such issues, few-shot learning studies have demonstrated that transferring knowledge from specific modules of a pre-trained model can achieve effective performance on target data domains ( Shen et al. 2021 ). Similarly, fine-tuning approaches focusing solely on the attention layer parameters of visual transformers have been shown to produce results comparable to full fine-tuning, while reducing catastrophic forgetting ( Touvron et al. 2022 ). Furthermore, a comprehensive analysis across various datasets and model architectures highlights the importance of understanding different model modules’ specific contributions for efficiently adapting to target tasks ( Ye et al. 2023 ). By analyzing the contributions of individual modules to target tasks, selectively fine-tuning only partial parameters allows for efficient learning of domain-specific knowledge with a minimal set of trainable parameters (

Read the complete text