Direct Preference Optimization for Speech Autoregressive Diffusion Models
Zhijun Liu, Dongya Jia, Xiaoqiang Wang, Chenpeng Du, Shuai Wang, Zhuo Chen, Haizhou Li · 2025
Read Direct Preference Optimization for Speech Autoregressive Diffusion Models by Zhijun Liu, Dongya Jia, Xiaoqiang Wang, Chenpeng Du, Shuai Wang, Zhuo Chen, Haizhou Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18928.txt.
Introduction
Zhijun Liu and six co-authors work on speech that a computer builds from text. Recent models build speech sound step by step, with each new piece of sound spread out by a diffusion process. These models reach top results in zero-shot text-to-speech, where the voice is copied from a short sample. Little work had yet tuned them with feedback on which outputs people prefer.
The team applies direct preference optimization, a feedback method, to the DiTAR speech model. They call the result ARDM-DPO. Fine-tuning with paired good and bad outputs makes the speech more expressive, meaning it sounds less flat and more human. It also holds up better on long texts, where weaker models often break down or lose the thread.
Catalog record
| CREATOR | Zhijun Liu, Dongya Jia, Xiaoqiang Wang, Chenpeng Du, Shuai Wang, Zhuo Chen, Haizhou Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-18928
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Direct Preference Optimization for Speech Autoregressive Diffusion Models Zhijun Liu, Dongya Jia, Xiaoqiang Wang, Chenpeng Du, Shuai Wang, Zhuo Chen, Haizhou Li Abstract Autoregressive diffusion models (ARDMs) have recently been applied to speech generation, achieving state-of-the-art (SOTA) performance in zero-shot text-to-speech. By autoregressively generating continuous speech tokens with next-token diffusion, these models offer a promising alternative to next-token prediction, avoiding the technical complexities associated with discrete speech tokenization. As a relatively new paradigm, research on reinforcement learning (RL)-based fine-tuning of speech ARDMs remains limited. In this paper, we propose Autoregressive Diffusion-Direct Preference Optimization (ARDM-DPO) to advance this research. By fine-tuning the recently proposed zero-shot text-to-speech model DiTAR with DPO, we achieve significant improvements in terms of speech expressiveness and robustness for long texts. † † address: 1 School of Data Science, The Chinese University of Hong Kong, Shenzhen 2 School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen 3 Nanjing University 4 ByteDance Seed 1 Introduction A growing body of work in multimodal generation, including audio [ 1 , 2 , 3 ] , image [ 4 , 5 , 6 ] , and video synthesis [ 7 , 8 , 9 ] , now employs autoregressive diffusion models (ARDMs) as the underlying architecture. ARDMs encode continuous modalities into sequences of continuous latent vectors (“continuous tokens”) and synthesize sequences by autoregressively predicting the next token with a diffusion model. Compared to next-token prediction over discrete token sequences [ 10 , 11 ] , this next-token diffusion approach [ 12 ] preserves fine details while avoiding excessively long sequences, thanks to the compactness of continuous latent representations. Notably, recent studies [ 2 , 12 , 13 , 14 ] applying ARDMs to speech generation report state-of-the-art zero-shot text-to-speech (TTS) performance. Modern TTS systems trained on large datasets can sample from the data distribution with high fidelity. However, the generated speech may not always align with human preferences after pretraining. For example, when prompted with emotional speech, TTS models can still produce undesirable monotone outputs, requiring additional post-filtering by users [ 15 ] . Preference alignment algorithms [ 16 , 17 , 18 ] bias the output distribution of generative models toward human-preferred samples, making them an essential post-training step for powerful speech generation systems [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ] . Direct Preference Optimization [ 17 ] (DPO) was originally proposed for aligning language models and was later adapted for fine-tuning diffusion models [ 30 ] . In this work, we extend DPO to ARDMs (ARDM-DPO) and apply it to fine-tune the recently proposed DiTAR model [ 2 ] , which achieves state-of-the-art performance in zero-shot TTS. DiTAR features a highly efficient ARDM architecture that separates computation for encoding the generation history from denoising the next token. To our knowledge, this is the first preference-alignment method tailored to ARDMs for TTS. We evaluate ARDM-DPO on two benchmarks: (A) F0 variance for expressiveness and (B) text likelihood for robustness on hard texts challenging autoregressive TTS. ARDM-DPO nearly doubles F0 variance with minimal speaker-similarity loss and reduces CER by 25%. Audio samples and further details can be found in the online supplement 1 1 1 https://zjlww.github.io/ardm-dpo/ . 2 Background Figure 1: ARDM sampling viewed as a Markov chain. Each state contains both the history-generated tokens and the current noisy token. In this work, we define x_{a:b}:=\left\{x_{n}:a\leq n<b,n\in\mathbb{Z}\right\} and x_{a..b}:=\left\{x_{n}:a\leq n\leq b,n\in\mathbb{Z}\right\}. 2.1 Diffusion Models Suppose q(x_{0}) is the data distribution. For each diffusion time index t\in\{1,\ldots,T\} , define the positive decreasing sequence (\alpha_{t})_{t=1}^{T} and the positive increasing sequence (\sigma_{t})_{t=1}^{T} . For each t define the Gaussian perturbation distribution q(x_{t}|x_{0}):=\mathcal{N}(x_{t};\alpha_{t}x_{0},\sigma_{t}^{2}I_{d}) . Define the noise perturbed distribution for each t\in\{1,\ldots,T\} as q(x_{t}):=\int q(x_{0})q(x_{t}|x_{0})\mathrm{d}x_{0}. (1) A well-trained diffusion model can be considered as a score estimator \nabla_{x_{t}}\log q(x_{t}) trained with denoising score matching [ 31 ] . In the DDPM [ 32 , 33 ] sampler, the diffusion model generates samples starting from random noise x_{T}\sim p(x_{T})=\mathcal{N}(x_{T};0,I) . Then it samples x_{t-1} given x_{t} iteratively. x_{t-1}=a_{t}x_{t}+b_{t}\nabla_{x_{t}}\log q(x_{t})+c_{t}\epsilon, (2) where \epsilon\sim\mathcal{N}(0,I) is an independent noise, (a_{t})_{t=1}^{T} , (b_{t})_{t=1}^{T} and (c_{t})_{t=1}^{T} are positive sequences depending on (\alpha_{t})_{t=1}^{T} and (\sigma_{t})_{t=1}^{T} . 2.2 Autoregressive Diffusion Models For simplicity, we assume that the ARDM always generates N continuous tokens. The ARDM sampling process can be viewed as a Markov chain, as illustrated in Fig. 1 . Each state s_{n}^{t} is indexed by the pair (n,t)\in\{1,\ldots,N\}\times\{1,\ldots,T\} , where n is the token index and t is the diffusion time index. Each state s_{n}^{t} contains the history tokens already denoised x^{0}_{<n} and the current noisy token x_{n}^{t} that is being denoised, where we define x_{<n}^{0}:=x_{1..n-1}^{0} . Given the state s_{n}^{t} , an ARDM estimates the conditional score \nabla\log q(x_{n}^{t}|x^{0}_{<n}) and transitions from s_{n}^{t} to s_{n}^{t-1} when t>1 . Upon reaching state s_{n}^{0} , where a token is fully denoised, a random Gaussian noise x_{n+1}^{T} is sampled, and a new DDPM sampling process starts from state s_{n+1}^{T}=(x_{\leq n}^{0},x_{n+1}^{T}) . 3 ARDM-DPO Let \mathbf{x} denote an ARDM sampling trajectory that contains all intermediate states in Fig. 1 . The reward function r(x_{1..N}^{0}) is defined over the terminal states. For a trajectory \mathbf{x} , we define r(\mathbf{x}):=r(x_{1..N}^{0}) . Consider the following KL divergence-constrained policy optimization problem: \max_{\pi}\left[\mathbb{E}_{\pi(\mathbf{x})}\left[r(\mathbf{x})\right]-\beta D_{\mathrm{KL}}(\pi(\mathbf{x})\|\mu(\mathbf{x}))\right], (3) where \beta>0 is the weight of the KL constraint. The KL-divergence constraint ensures that the learned policy \pi(\mathbf{x}) does not deviate too far from the reference distribution \mu(\mathbf{x}) . The unique optimal policy \pi_{r} given reward r is [ 17 , 34 ] : \pi_{r}(\mathbf{x}):=\frac{1}{Z_{r}}\mu(\mathbf{x})\exp\left(\frac{1}{\beta}r(\mathbf{x})\right), (4) where Z_{r} is defined as Z_{r}:=\int\mu(\mathbf{x})\exp\left(\frac{1}{\beta}r(\mathbf{x})\right)\mathrm{d}\mathbf{x} . Then, we have: r(\mathbf{x})=\beta\log\frac{\pi_{r}(\mathbf{x})}{\mu(\mathbf{x})}+\beta\log Z_{r}. (5) Marginalizing over all ARDM intermediate states gives: r(x_{1..N}^{0})=\beta\mathbb{E}_{\pi_{r}(\mathbf{x}|x_{1..N}^{0})}\left[\log\frac{\pi_{r}(\mathbf{x})}{\mu(\mathbf{x})}\right]+\beta\log Z_{r}. (6) As in DPO, we assume that the log probability of human listeners preferring sample x_{1..N}^{0} over y_{1..N}^{0} is modeled by a Bradley-Terry model [ 17 ] : \log P(x_{1..N}^{0}\succ y_{1..N}^{0})=\log\sigma\left(r(x_{1..N}^{0})-r(y_{1..N}^{0})\right). (7) Given a dataset \mathcal{D} of preference pairs, we can estimate the reward model parameters with maximum likelihood estimation: \max_{r}\mathbb{E}_{x_{1..N}^{0},y_{1..N}^{0}\sim\mathcal{D}}\left[\log P(x_{1..N}^{0}\succ y_{1..N}^{0})\right]. (8) As a result of Eq. ( 6 ), optimizing the reward model r with maximum likelihood can be conducted without an explicit reward model by optimizing: \mathbb{E}_{x_{1..N}^{0},y_{1..N}^{0}\sim\mathcal{D}}\left[\mathcal{J}(x_{1..N}^{0},y_{1..N}^{0})\right], (9) where \displaystyle\mathcal{J}(x_{1..N}^{0},y_{1..N}^{0}):= (10) \displaystyle\log\sigma\Big(TN\beta\cdot\mathbb{E}_{\begin{subarray}{c}\mathcal{U}(t),\mathcal{U}(n)\\ \pi(x_{n}^{t},x_{n}^{t-1}|x_{\leq n}^{0})\\ \pi(y_{n}^{t},y_{n}^{t-1}|y_{\leq n}^{0})\end{subarray}}\left[\ell_{n}^{t}(\mathbf{x})-\ell_{n}^{t}(\mathbf{y})\right]\Big), with \ell_{n}^{t}(\mathbf{x}) defined as: \ell_{n}^{t}(\mathbf{x}):=\log\frac{\pi(x_{n}^{t-1}|x_{n}^{t},x_{<n}^{0})}{\mu(x_{n}^{t-1}|x_{n}^{t},x_{<n}^{0})}. (11) As proposed in Diffusion-DPO [ 30 ] , we assume that the distribution \pi(x_{n}^{t},x_{n}^{t-1}|x_{\leq n}^{0}) can be approximated with q(x_{n}^{t}|x_{n}^{0})\cdot q(x_{n}^{t-1}|x_{n}^{t},x_{n}^{0}) . We also move the expectation over \mathcal{U}(t) , q(x_{n}^{t}|x_{n}^{0}) , and q(y_{n}^{t}|y_{n}^{0}) from the inside of \log\sigma to the outside with Jensen’s inequality to obtain the approximate lower bound \mathcal{L} for \mathcal{J} . The constant factor TN is absorbed into \beta in Eq. ( 12 ). \mathcal{U}(t),\mathcal{U}(n) are uniform distributions over all possible values. \displaystyle\mathcal{J}(x_{1..N}^{0},y_{1..N}^{0})\geq\mathcal{L}(x_{1..N}^{0},y_{1..N}^{0}):= (12) \displaystyle\mathbb{E}_{\begin{subarray}{c}\mathcal{U}(t)\\ q(x_{n}^{t}|x_{n}^{0})\\ q(y_{n}^{t}|y_{n}^{0})\end{subarray}}\Big[\log\sigma\Big(\beta\mathbb{E}_{\begin{subarray}{c}\mathcal{U}(n)\\ q(x_{n}^{t-1}|x_{n}^{t},x_{n}^{0})\\ q(y_{n}^{t-1}|y_{n}^{t},y_{n}^{0})\end{subarray}}\left[\ell_{n}^{t}(\mathbf{x})-\ell_{n}^{t}(\mathbf{y})\right]\Big)\Big]. Notice that the expectation \mathbb{E}_{q(x_{n}^{t-1}|x_{n}^{t},x_{n}^{0})}[\ell_{n}^{t}(\mathbf{x})] can be written as the difference of KL divergences: \displaystyle D_{\mathrm{KL}}\left(q(x_{n}^{t-1}|x_{n}^{t},x_{n}^{0})\middle\|\mu(x_{n}^{t-1}|x_{n}^{t},x_{<n}^{0})\right) (13) \displaystyle-D_{\mathrm{KL}}\left(q(x_{n}^{t-1}|x_{n}^{t},x_{n}^{0})\middle\|\pi(x_{n}^{t-1}|x_{n}^{t},x_{<n}^{0})\right). Without loss of generality, suppose that the ARDM is trained with the denoising objective. Then Eq. ( 13 ) is equivalent to: \omega_{t}\left(-\|v_{\theta}(x_{n}^{t},x_{<n}^{0})-x_{n}^{0}\|_{2}^{2}+\|v_{\mathrm{ref}}(x_{n}^{t},x_{<n}^{0})-x_{n}^{0}\|_{2}^{2}\right), (14) where \omega_{t} is a time-dependent weight. Finally, we arrive at the ARDM-DPO training objective: \displaystyle\mathcal{L}(x_{1..N}^{0},y_{1..N}^{0}):=\mathbb{E}_{\begin{subarray}{c}\mathcal{U}(t)\\ q(x_{n}^{t}|x_{n}^{0})\\ q(y_{n}^{t}|y_{n}^{0})\end{subarray}}\Big[\log\sigma\Big(\beta\omega_{t}\mathbb{E}_{\mathcal{U}(n)} (15) \displaystyle\Big[ \displaystyle-\left\|v_{\theta}(x_{n}^{t},x_{<n}^{0})-x_{n}^{0}\right\|_{2}^{2}+\left\|v_{\mathrm{ref}}(x_{n}^{t},x_{<n}^{0})-x_{n}^{0}\right\|_{2}^{2} \displaystyle+\left\|v_{\theta}(y_{n}^{t},y_{<n}^{0})-y_{n}^{0}\right\|_{2}^{2}-\left\|v_{\mathrm{ref}}(y_{n}^{t},y_{<n}^{0})-y_{n}^{0}\right\|_{2}^{2}\Big]\Big)\Big]. The DiTAR model used in our experiments is