When Fine-Tuning Fails and when it Generalises: Role of Data Diversity and Mixed Training in LLM-based TTS

Anupam Purwar, Aditya Choudhary[2pt] Sprinklr AIGurugram, India · 2026

Read When Fine-Tuning Fails and when it Generalises: Role of Data Diversity and Mixed Training in LLM-based TTS by Anupam Purwar, Aditya Choudhary[2pt] Sprinklr AIGurugram, India (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10904.txt.

Introduction

Anupam Purwar and Aditya Choudhary test whether fine-tuning the language-model backbone of a text-to-speech system helps it clone voices faithfully. Across several speakers, LoRA fine-tuning of a compact Qwen model beat the frozen base model on perceptual quality, speaker similarity, and signal-to-noise ratio. The gains hinge on training data diversity: speakers with more varied recordings improved on all three measures at once, while narrow data limited the benefit.

Catalog record

CREATORAnupam Purwar, Aditya Choudhary[2pt] Sprinklr AIGurugram, India
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45688
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10904

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Fine-Tuning Fails and when it Generalises: Role of Data Diversity and Mixed Training in LLM-based TTS Anupam Purwar, Aditya Choudhary[2pt] Sprinklr AIGurugram, India Abstract Large language models are increasingly adopted as semantic backbones for neural text-to-speech systems. However, frozen LLM representations are insufficient for modeling speaker-specific acoustic and perceptual characteristics. Our experiments involving fine tuning of the Language Model backbone of TTS show promise in improving the voice consistency and Signal to Noise ratio (SNR) in voice cloning task. Across multiple speakers, LoRA fine-tuning consistently outperforms the non–fine-tuned base Qwen-0.5B model across three complementary dimensions of speech quality. First, perceptual quality improves significantly, with DNS-MOS gains of up to +0.42 points for speakers whose training data exhibits sufficient acoustic variability. Second, speaker fidelity improves for all evaluated speakers, with consistent increases in voice similarity, indicating that LoRA effectively adapts speaker identity representations without degrading linguistic modeling. Third, signal-level quality improves in most cases, with signal-to-noise ratio increasing by as much as 34 percent. Crucially, these improvements are strongly governed by the characteristics of the training data. Speakers with high variability in acoustic energy and perceptual quality achieve simultaneous gains in DNS-MOS, voice similarity, and SNR. In contrast, speakers trained on acoustically homogeneous data experience limited gains or perceptual degradation, even when voice similarity improves. This reveals that LoRA can faithfully clone speaker identity while also amplifying noise characteristics and recording artifacts present in narrow training distributions. We further identify a loss–quality divergence phenomenon in which training and validation loss continue to improve during fine-tuning while perceptual quality degrades for low-variability speakers. Besides, we show that optimal inference temperature of the language model backbone depends on training data variability, with conservative sampling benefiting low-variability speakers but degrading quality for high-variability ones. Overall, this work establishes that LoRA fine-tuning is not merely a parameter-efficient optimization technique but an effective mechanism for better speaker-level adaptation in compact LLM-based TTS systems. When supported by sufficiently diverse training data, LoRA-adapted Qwen-0.5B consistently surpasses its frozen base model in perceptual quality, speaker similarity with low latency using GGUF model hosted in quantized form. 1 1 footnotetext: Equal contribution by both authors 2 2 footnotetext: Corresponding author: [email protected] . Project page: https://anupam-purwar.github.io/page/. 1 Introduction Voice to voice architecture are integral to multi-modal agent development. Different voice to voice architectures viz. cascaded, end to end and hybrid architectures have been proposed. Cascaded systems are easier to build initially whereas end-to-end requires more specialized expertise. End-to-end native voice models are being prioritized for consumer applications where naturalness and latency matter most, while cascaded approaches remain popular in enterprise settings where tool calling and advanced reasoning capabilities provided by frontier LLMs are priority. Cascading multi-modal agents which support voice input and output, comprise of three principle components, Automated Speech Recognition (ASR), Text based Agent (LLM) and Text-to-Speech (TTS), each of which are susceptible to errors of different kinds. TTS powered by language model backbones have shown promise in using Language model backbone to predict sequences of acoustic tokens conditioned on text and a speaker prompt 6 17 11 23 8 . Language model based TTS systems collapse linguistic modeling, prosody planning, and long-range acoustic coherence into a single autoregressive backbone. However, despite their LM-centric design, existing literature does not report systematic LoRA fine-tuning of these language model backbones for TTS adaptation, leaving its impact on perceptual audio quality in voice cloning largely unexplored. Parameter-efficient fine-tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), have emerged as a practical solution for adapting large text-to-speech (TTS) models under memory and latency constraints. Existing LoRA-based TTS literature predominantly applies adaptation to downstream synthesis components viz. acoustic decoders, speaker embeddings, or style and emotion control modules, while keeping the linguistic or semantic modeling backbone frozen. Representative works including LoRP-TTS, StyleSpeech, EELE, and LoRA-based multi-speaker VITS adaptations follow this paradigm, focusing on efficient control or personalization without modifying the core generative model (Table ). UtterTune represents one of the only reported research which inject LoRA into LM layers; however, its scope is primarily limited to pronunciation and pitch-accent control, without broader analysis of perceptual quality, inference latency, or stability trade-offs (Table ). As a result, several fundamental questions remain unanswered in the literature. First, it is unclear how LoRA adaptation of an LM-based TTS backbone interacts with the pretrained acoustic prior encoded in large-scale speech-trained language models. Second, the reliability of validation loss as a proxy for perceptual quality parameters like MOS, SNR, and speaker similarity has not been rigorously examined in token-level generative TTS models. Third, the role of training data characteristics, such as acoustic variability, energy spread, and linguistic diversity, in determining the success or failure of LM-level LoRA adaptation is poorly understood. Finally, while inference-time decoding controls such as temperature and top- k sampling are known to influence perceptual outcomes in language models, their joint optimization with LoRA-adapted LM backbones for TTS has not been systematically studied, particularly in the context of balancing naturalness, stability, and latency. In contrast to prior approaches summarized in Table , our work directly addresses these gaps. First, by conducting a comprehensive empirical study of LoRA fine-tuning applied to the Qwen-0.5B language model backbone for TTS. Next, we evaluated its performance across multiple datasets and speakers, with following main contributions: • LM-Backbone LoRA for TTS: We apply LoRA directly to attention layers of a language model backbone (Qwen-0.5B) used for acoustic token prediction, moving beyond synthesis-layer-only adaptation. • Loss–Quality Decoupling Analysis: We identify and characterize a failure mode where validation loss improves monotonically while perceptual quality (DNS-MOS) degrades, challenging conventional early-stopping criteria for LM-based TTS. • Training Data Variability Study: Through controlled experiments across speakers and datasets, we demonstrate that acoustic variability, is a strong predictor of successful LM-level LoRA adaptation. • Hyperparameter Optimization: We show that decoding hyperparameters (temperature and top- k sampling) can partially mitigate perceptual degradation induced by LoRA adaptation, enabling post-training control over quality stability trade-offs. • Latency Optimization: We show that model quantization and use of GGUF to store model weights, enables faster inference by significantly reducing first chunk latency. Table 1: Comparison of LoRA-based Methods in Text-to-Speech Systems Paper What was implemented (2-line summary) LoRA on LLM TTS Backbone LoRP-TTS 5 Bondaruk & Kubiak (2025) Applies LoRA to TTS model layers for rapid speaker personalization from a single noisy reference utterance. Low-rank updates adapt speaker characteristics while keeping the synthesis backbone frozen. No TTS-Hub 2 OpenReview (ICLR submission) Introduces modular LoRA adapters trained for individual speech attributes and composed arithmetically. Enables controllable TTS without modifying the underlying language or synthesis backbone. No StyleSpeech 16 Lou et al. (ACM MM Asia 2024) Employs parameter-efficient LoRA fine-tuning on pre-trained TTS components for phoneme and style control. Focuses on efficiency and controllability rather than LM adaptation. No LoRA-based Multi-Speaker TTS IEEE Access (2024) Integrates LoRA and residual adapters into VITS-based multi-speaker TTS architectures. Achieves efficient speaker adaptation with substantially fewer trainable parameters. No EELE 19 arXiv:2408.10852 Applies LoRA modules to emotional and expressive control layers in TTS systems. Targets parameter-efficient emotion adaptation without altering linguistic modeling layers. No UtterTune 12 (2025) Injects LoRA directly into the Transformer-based language model backbone of an LLM-TTS system. Enables pronunciation and pitch-accent control by adapting semantic/phoneme prediction layers. Yes

Read the complete text