GOAT-TTS: Expressive and Realistic Speech Generation via A Dual-Branch LLM

Yaodong Song, Hongjie Chen, Jie Lian, Yuxin Zhang, Guangmin Xia, Zehan Li, Genliang Zhao, Jian Kang, Jie Li, Yongxiang Li, Xuelong Li Institute of Artificial Intelligence (TeleAI), China Telecom, Beijing [email protected] · 2025

Read GOAT-TTS: Expressive and Realistic Speech Generation via A Dual-Branch LLM by Yaodong Song, Hongjie Chen, Jie Lian, Yuxin Zhang, Guangmin Xia, Zehan Li, Genliang Zhao, Jian Kang, Jie Li, Yongxiang Li, Xuelong Li Institute of Artificial Intelligence (TeleAI), China Telecom, Beijing [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12339.txt.

Introduction

Yaodong Song and co-authors (2025) present GOAT-TTS, a text-to-speech system built on a large language model with a dual-branch design. Such systems usually turn sound into discrete tokens, and the authors list three problems with that route. Quantizing a speech prompt throws away acoustic detail beyond recovery. Systems lean on exactly aligned prompt speech and text, which real deployments rarely have. Tuning a model to emit speech tokens can damage its original gift for understanding text, a failure called catastrophic forgetting.

GOAT-TTS answers with two branches. One aligns the modalities: a speech encoder and projector turn a prompt into continuous acoustic features, keeping tone and other paralinguistic detail that tokens lose. The other protects the language model's text skills while it learns to speak. The goal is expressive, realistic speech without the brittleness of prompt-dependent, token-only systems.

Catalog record

CREATORYaodong Song, Hongjie Chen, Jie Lian, Yuxin Zhang, Guangmin Xia, Zehan Li, Genliang Zhao, Jian Kang, Jie Li, Yongxiang Li, Xuelong Li Institute of Artificial Intelligence (TeleAI), China Telecom, Beijing [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12339

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GOAT-TTS: Expressive and Realistic Speech Generation via A Dual-Branch LLM Yaodong Song, Hongjie Chen, Jie Lian, Yuxin Zhang, Guangmin Xia, Zehan Li, Genliang Zhao, Jian Kang, Jie Li, Yongxiang Li, Xuelong Li Institute of Artificial Intelligence (TeleAI), China Telecom, Beijing [email protected] Abstract While large language models (LLMs) have revolutionized text-to-speech (TTS) synthesis through discrete tokenization paradigms, current architectures exhibit fundamental tensions between three critical dimensions: 1) irreversible loss of acoustic characteristics caused by quantization of speech prompts; 2) stringent dependence on precisely aligned prompt speech-text pairs that limit real-world deployment; and 3) catastrophic forgetting of the LLM’s native text comprehension during optimization for speech token generation. To address these challenges, we propose an LLM-based text-to-speech G eneration approach O ptimized via a novel dual-branch A rchi T ecture (GOAT-TTS). Our framework introduces two key innovations: (1) The modality-alignment branch combines a speech encoder and projector to capture continuous acoustic embeddings, enabling bidirectional correlation between paralinguistic features (language, timbre, emotion) and semantic text representations without transcript dependency; (2) The speech-generation branch employs modular fine-tuning on top-k layers of an LLM for speech token prediction while freezing the bottom-n layers to preserve foundational linguistic knowledge. Moreover, multi-token prediction is introduced to support real-time streaming TTS synthesis. Experimental results demonstrate that our GOAT-TTS achieves performance comparable to state-of-the-art TTS models while validating the efficacy of synthesized dialect speech data. † † footnotetext: † Equal contribution. † † footnotetext: ∗ Corresponding author. 1 Introduction In the era of large language models (LLMs) [ 1 ; 2 ; 3 ] , the construction of end-to-end speech dialogue systems [ 4 ; 5 ; 6 ; 7 ] necessitating dialect comprehension and affective awareness typically demands the acquisition of large-scale dialect-specific and/or rich-emotion speech-text alignment datasets. This requirement incurs significant resource expenditures due to the labor-intensive nature of multi-modal data curation. However, given the relative abundance of textual data and the accelerated progress in LLM-driven speech synthesis architectures, we posit that this data scarcity challenge will be ameliorated through synthetic data augmentation. In particular, the field of speech synthesis has undergone transformative advances driven by breakthroughs in text-based language modeling paradigms. Empirical studies across industry and academia have demonstrated that speech language models (SLMs) based on decoder-only transformers exhibit remarkable instruction-following capabilities, enabling granular control over vocal attributes such as speaker identity, environmental acoustics, affective prosody, and semantic coherence through speech prompt conditioning. Pioneering works such as SPEAR-TTS [ 8 ] , VALL-E [ 9 ] , and CosyVoice [ 10 ] have established foundational methodologies for few-shot/zero-shot speaker adaptation via autoregressive modeling of discrete speech tokens. Recent architectural innovations exemplified by OuteTTS [ 11 ] , CosyVoice2 [ 12 ] , and Llasa [ 13 ] have further advanced the state-of-the-art by integrating pre-trained text language models as core architectural components. These approaches leverage the next-token prediction paradigm to speech tokens within the LLM’s generative framework, achieving high-fidelity speech synthesis while maintaining linguistic coherence. Collectively, these advancements underscore the cross-modal potential of large-scale pre-trained models, demonstrating their capacity to simultaneously enhance semantic understanding and high acoustic fidelity during the speech generation process. However, our preliminary investigations reveal three intrinsic challenges in these methods. First, discrete tokenization of the speech prompt induces irreversible loss of acoustic characteristics, resulting in measurable degradation in terms of synthesized speech quality metrics including natural prosody and emotional expressiveness. Second, the rigid association between audio signals and their corresponding transcripts in the prompt imposes significant constraints on the flexibility of real-world deployment. Third, we observed a drastic deterioration of the LLM’s native text comprehension capability during training, suggesting an inherent incompatibility between the preservation of pre-trained linguistic knowledge and the optimization of speech generation objectives. To address these challenges, we propose a novel TTS G eneration approach O ptimized by a dual-branch LLM A rchi T ecture (GOAT-TTS). The framework comprises three core components, a shared pre-trained LLM backbone, a modality-alignment branch, and a speech-generation branch, achieving prompt-driven synthesis of high-quality speech through a two-stage training paradigm. Specifically, the model employs a pretrained speech encoder to extract multidimensional acoustic features from a speech prompt. Conditioned on the acoustic features, the LLM processes textual inputs to generate corresponding discrete speech tokens, which are subsequently mapped to mel-spectrograms via a flow-matching model. Our proposed GOAT-TTS framework differs from other existing LLM-based TTS systems in terms of architectural design and training methodologies. • First, through the modality-alignment branch, we extract continuous representations of the speech prompt by the combination of a speech encoder and a projector module. Unlike conventional approaches that rely on discrete tokenization for speech prompt encoding, our method simultaneously captures rich paralinguistic features (e.g., timbral characteristics, prosodic contours, and affective nuances) and establishes bidirectional correlations between continuous acoustic embeddings and semantic text representations. This deep alignment process equips the LLM backbone with the ability to directly interpret speech prompts through their continuous acoustic embeddings, eliminating the dependency on precise transcriptions of speech prompts during inference, which significantly enhances flexibility for industrial applications. • Second, to preserve the pre-trained LLM’s intrinsic knowledge structure, the generation-branch adopts a layer-wise parameter-freezing strategy: only the top-k layers’ parameters undergo task-specific fine-tuning while foundational layers remain frozen. This strategy ensures retention of the LLM’s original contextual comprehension capabilities while enabling targeted optimization for speech synthesis objectives. Additionally, we introduce a multi-token prediction mechanism [ 14 ] , which reduces frequency discrepancies between the original LLM outputs and the fine-tuned model’s speech token predictions. This design not only accelerates the inference speed but also inherently supports streaming text token input and streaming speech token output, offering general compatibility for real-world applications. 2 Method In this section, we detail the training methodology underpinning our proposed framework. As depicted in Figure 1 , the framework employs a hierarchical processing pipeline. First, a pretrained Whisper-Small speech encoder is utilized to extract latent acoustic representations from input speech prompts. These latent features are then mapped to LLM-compatible embeddings through a projector module, which incorporates a 3-layer CNN-based architecture followed by a linear transformation layer. Subsequently, the LLM generates speech tokens in an iterative fashion, conditioned jointly on both the adapted acoustic embeddings and the provided textual input. Finally, the discrete speech tokens are converted into mel-spectrogram representations via a flow-matching model, which facilitates subsequent waveform synthesis through probabilistic density estimation. Figure 1: Left: Model architecture of GOAT-TTS. Initially, LLM_{S} and LLM_{T} share weight parameters. Right: Illustration of the two-stage training strategy for GOAT-TTS. The training regimen for the complete framework is structured into two sequential stages. Stage I centers on modality-alignment training, which establishes cross-modal semantic coherence between acoustic and linguistic features by optimizing the speech encoder and projector module. Stage II subsequently transitions to speech-generation training, where the full pipeline is fine-tuned to refine the end-to-end speech synthesis capability. During this stage, the flow-matching model is further optimized to enhance waveform fidelity by minimizing the discrepancy between predicted and target mel-spectrograms. This phased approach ensures robust modality alignment while enabling high-quality speech generation through iterative parameter optimization. 2.1 Modality-Alignment Training Inspired by previous work on modality alignment [ 15 ; 16 ] , our framework adopts speech-text continuation pairs as the core training data for speech-text alignment. In this stage,

Read the complete text