Differentiable Reward Optimization for LLM based TTS system
Changfeng Gao, Zhihao Du, Shiliang Zhang · 2025
Read Differentiable Reward Optimization for LLM based TTS system by Changfeng Gao, Zhihao Du, Shiliang Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05911.txt.
Introduction
Changfeng Gao, Zhihao Du, and Shiliang Zhang fine-tune text-to-speech systems without ever rendering the audio during training. Modern speech models speak through neural codec tokens. Standard feedback training scores the finished sound, a slow loop. Their method scores the tokens directly.
A trick called Gumbel-Softmax makes that token reward smooth enough to train by gradient. A multi-task reward model grades several angles at once, such as clarity and emotion. The result topped a standard benchmark for word error rate, meaning cleaner pronunciation. The extra reward heads also steered tone and quality in zero-shot use. The authors pitch it as faster, sharper speech tuning.
Catalog record
| CREATOR | Changfeng Gao, Zhihao Du, Shiliang Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05911
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Differentiable Reward Optimization for LLM based TTS system Changfeng Gao, Zhihao Du, Shiliang Zhang Abstract This paper proposes a novel Differentiable Reward Optimization (DiffRO) method aimed at enhancing the performance of neural codec language models based text-to-speech (TTS) systems. In contrast to conventional reinforcement learning from human feedback (RLHF) approaches applied to TTS, DiffRO directly compute the rewards based on neural codec tokens, rather than relying on synthesized audio. Furthermore, we employ the Gumbel-Softmax technique to render the reward function differentiable, thereby streamlining the RLHF training process. Additionally, we introduce a multi-task reward (MTR) model which can provide feedback from different perspectives and find that it can augment the system’s capability to follow instructions effectively. Experimental results indicate that DiffRO significantly improves the pronunciation accuracy of the TTS system, achieving state-of-the-art (SOTA) WER results on the seed-tts-eval benchmark. Moreover, with the integration of the MTR model, we demonstrate the ability to control emotional and quality attributes in a zero-shot manner. † † address: Speech Team, Tongyi Lab, Alibaba Group † † email: {gaochangfeng.gcf, neo.dzh}@alibaba-inc.com Index Terms : Text-to-speech (TTS) synthesis, large language models, RLHF 1 Introduction Inspired by the success of text-based large language models (LLMs), neural codec token language modeling (LM) has emerged as a leading approach for text-to-speech (TTS) generation. This methodology employs a pre-trained speech tokenizer to encode speech into a discrete sequence of tokens, which are subsequently modeled using a decoder-only LM that predicts these tokens based on textual input. Ultimately, a flow matching (FM) [ 1 ] model and a vocoder are applied to transform the tokens into audible speech. Leveraging the capabilities of zero-shot and in-context learning inherent in LLMs, neural codec-based TTS systems [ 2 , 3 ] have demonstrated enhanced quality and naturalness, enabling applications such as zero-shot TTS and instruction-based TTS. In the training of LLMs, Reinforcement Learning from Human Feedback (RLHF) [ 4 ] represents a critical step in aligning LLMs with human preferences, contributing to the success of numerous state-of-the-art models. Recent TTS systems have also endeavored to incorporate RLHF to enhance the quality of generated speech; however, a widely applicable methodology remains elusive. The challenges can be summarized as follows: Firstly, unlike traditional natural language processing (NLP) tasks, TTS systems require additional backend FM and vocoder models to convert discrete neural codec tokens into waveform audio. The computational demands posed by these backend models are substantial, which hinders the large-scale production of RLHF data. Secondly, the diversity of generated TTS speech is often insufficient. Although the neural codec LLM can generate different token sequences from the same text and speech prompt by employing various sampling strategies and random seeds, the resultant voices consistently exhibit high similarity. Consequently, it becomes challenging to differentiate between positive and negative feedback for training the reward model. Finally, the evaluation methods for TTS systems are inherently complex. Typically, the mean opinion score (MOS) serves as the gold standard for evaluating TTS quality. However, more critical aspects such as accuracy, naturalness, and speaker similarity are ultimately determined by listener perception. This issue is exacerbated in instruction-based TTS scenarios, where certain instructions may conflict with the need for precise pronunciation accuracy. Several existing studies [ 5 , 6 , 7 , 8 , 9 ] have attempted to address one or more of the aforementioned challenges through various approaches. For instance, Chen et al. [ 6 ] introduced a uncertainty-aware optimization (UNO) with unpaired negative and positive samples which can mitigate the impact of insufficient diversity within TTS systems. They further proposed a Reverse Input Output (RIO) strategy to generate additional positive samples by reversing the prompts and their corresponding responses [ 7 ] . Anastassiou et al. [ 8 ] utilized automatic speech recognition (ASR) and speaker recognition systems to measure word error rate (WER) and speaker similarity as reinforcement learning (RL) rewards. Moreover, Gao et al. [ 9 ] demonstrated that employing the accuracy of the speech emotion recognition (SER) model as a reward can enhance emotional controllability. Collectively, these studies illustrate that leveraging objective indicators can yield significant improvements in TTS systems across various dimensions. In this work, we propose a novel Differentiable Reward Optimization (DiffRO) method specifically designed for neural codec-based TTS systems. Unlike existing approaches, DiffRO predicts rewards directly from the neural codec tokens rather than from the synthesized speech, thereby reducing the computational burden associated with backend frame-matching and vocoder models. Furthermore, we utilize the Gumbel-Softmax technique to ensure that the loss function of the reward model is differentiable with respect to the input text of the TTS system. Finally, we construct a multiple downstream tasks reward (MTR) model, including ASR, speech emotion recognition (SER), speech quality assessment (SQA), and age and gender prediction. Our findings indicate that the MTR model can effectively regulate speech attributes in accordance with preset preferences or input instructions. 2 Related work 2.1 Neural Codec LLM based TTS systems \includegraphics [width=0.7]figure1.png Figure 1: Comparison between the DPO an the proposed DiffRO. DiffRO can directly predict the reward score from the codec token sequence rather than the audio, and it can directly optimize the LM with back-propagation rather than the RL training loop like DPO. Most neural codec LLM based TTS systems comprise four primary building blocks: a pre-trained speech tokenizer, a neural codec token LM, a FM model, and an acoustic vocoder. Given a paired input of speech X_{1:T} and text Y_{1:N} , the speech tokenizer encodes the speech into a token sequence U_{1:T} . Subsequently, the LM is trained using a next-token prediction task: U_{t}=\text{argmax}_{1:Q}P_{\pi}(U_{t}=q|U_{1:t-1},Y_{1:N}) (1) where Q is the code-book size of the tokenizer, \pi stand for the LM model. In general, the token sequence U_{1:T} carries most information of the text and plays a crucial role in the TTS system. The TTS system will recover the audios X_{1:T} from U_{1:T} with the FM and the vocoder. X_{1:T}=\text{Vocoder}(\text{FM}(U_{1:T};\theta_{f});\theta_{v}) (2) where \theta_{f} and \theta_{v} are the parameters of the FM model and the vocoder. 2.2 RLHF for TTS To obtain human or AI feedback, one input Y should be inputted into the LM more than once, then different sample strategy and random seed are used to obtain different audios {X^{1},X^{2},...,X^{K}} . The positive and negative X^{pos} and X^{neg} are chosen from the \{X^{i}\}_{i=1}^{K} and their neural codec tokens U^{pos} and U^{neg} are used to train a reward model R(X,Y) , which can provide feedback to the LM during the RL phase. \mathcal{L}_{R}=-\mathbb{E}\left[\log\sigma(R(U^{pos},Y)-R(U^{neg},Y))\right] (3) \pi_{\theta}^{*}=\max_{\pi_{\theta}}\mathbb{E}\left[R(U,Y)\right]-\beta D_{\text{KL}}\left[\pi_{\theta}(U|Y)\|\pi_{\text{ref}}(U|Y)\right] (4) Proximal policy optimization (PPO) [ 10 ] are used to optimize the eq 4 . Direct Preference Optimization (DPO) [ 11 ] can merge the reward modeling and RL into one phase and optimize the eq 3 and 4 with U^{pos} and U^{neg} directly. \begin{split}\mathcal{L}_{DPO}(\pi_{\theta},\pi_{ref})=-\mathbb{E}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(U^{pos}|Y)}{\pi_{ref}(U^{pos}|Y)}\right.\right.\\ \quad\left.\left.-\beta\log\frac{\pi_{\theta}(U^{neg}|Y)}{\pi_{ref}(U^{neg}|Y)}\right)\right]\end{split} (5) where \pi_{\theta} is the optimized LM model and \pi_{ref} denotes a frozen reference model. Although DPO can simplify RLHF into a single phase, constructing a preferred and dispreferred training set remains a complex task for TTS systems. On one hand, to derive meaningful feedback, the predicted token sequence must be decoded into waveform audio via the FM model and the vocoder. On the other hand, synthesized audio samples often exhibit high similarity, making it challenging to accurately distinguish between positive