Aligning Diffusion Language Models via Unpaired Preference Optimization
Vaibhav Jindal 1 * † Hejian Sang 1 * Chun-Mao Lai 3 Yanning Chen 1 Zhipeng Wang 1 1 LinkedIn Corporation, CA, USA 3 University of California San Diego, CA, USA · 2025
Read Aligning Diffusion Language Models via Unpaired Preference Optimization by Vaibhav Jindal 1 * † Hejian Sang 1 * Chun-Mao Lai 3 Yanning Chen 1 Zhipeng Wang 1 1 LinkedIn Corporation, CA, USA 3 University of California San Diego, CA, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23658.txt.
Introduction
Vaibhav Jindal and four co-authors align diffusion language models with human preferences without paired comparison data. Diffusion models generate text by gradual denoising rather than word by word, which makes their sequence likelihoods intractable and preference tuning awkward. Paired preference data, the usual fuel for alignment, is also costly to gather.
Their method, ELBO-KTO, swaps the true likelihood for a tractable lower-bound stand-in and plugs it into an unpaired preference objective drawn from prospect theory. The authors analyze the bias and variance this swap introduces and add variance-reduction tricks to steady training. Applied to the LLaDA-8B-Instruct model, the method won about sixty-six and sixty-two percent of judged comparisons against the base model on two datasets. On downstream tests like GSM8K and MMLU it matched or beat the base model, showing unpaired tuning is a real alternative.
Catalog record
| CREATOR | Vaibhav Jindal 1 * † Hejian Sang 1 * Chun-Mao Lai 3 Yanning Chen 1 Zhipeng Wang 1 1 LinkedIn Corporation, CA, USA 3 University of California San Diego, CA, USA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23658
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aligning Diffusion Language Models via Unpaired Preference Optimization Vaibhav Jindal 1 * † Hejian Sang 1 * Chun-Mao Lai 3 Yanning Chen 1 Zhipeng Wang 1 1 LinkedIn Corporation, CA, USA 3 University of California San Diego, CA, USA Abstract Diffusion language models (dLLMs) are an emerging alternative to autoregressive (AR) generators, but aligning them to human preferences is challenging because sequence log-likelihoods are intractable and pairwise preference data are costly to collect. We introduce ELBO-KTO, which combines an ELBO surrogate for diffusion log-likelihoods with a prospect-theoretic, unpaired preference objective (Kahneman–Tversky Optimization, KTO). We analyze the bias and variance induced by the ELBO substitution and employ variance-reduction practices that stabilize gradients during training. Applied to LLaDA-8B-Instruct, ELBO-KTO yields 65.9% and 62.3% adjusted win rates on kto-mix-14k and UltraFeedback-Binary, respectively, versus the base model under an automatic LLM judge. Across downstream tasks, including GSM8K, MMLU, and additional reasoning/knowledge benchmarks, ELBO-KTO trained on UltraFeedback-Binary performs on par with or better than the base model under identical decoding. This establishes unpaired preference optimization as a viable alternative to pairwise alignment in diffusion LLMs. 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding author: Vaibhav Jindal < [email protected] >. 1 INTRODUCTION Aligning large language models (LLMs) with human preferences is central to building helpful systems. In autoregressive LLMs, methods such as Reinforcement learning from human feedback (RLHF) ( Christiano et al. 2017 ; Ouyang et al. 2022 ; Stiennon et al. 2020 ; Schulman et al. 2017 ) and Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) and its variants ( Meng et al. 2024 ; Yuan et al. 2023 ; Azar et al. 2024 ) are standard and rely on curated paired comparisons for each prompt. Diffusion-style large language models (dLLMs) extend text generation beyond left-to-right decoding by iteratively refining a sequence in parallel ( Ye et al. 2025 ; Ou et al. 2024 ; Lou et al. 2024 ; Nie et al. 2025 ; Khanna et al. 2025 ) which makes sentence-level likelihoods intractable and complicates preference optimization. Recent approaches to dLLMs have replaced log-likelihoods with ELBO-based surrogates and incorporated variance reduction techniques to stabilize training, but remain constrained by the assumption of paired preference data ( Zhu et al. 2025 ) . This assumption limits scalability and applicability in practical settings where feedback is abundant but inherently unpaired—as in binary “good/bad” user ratings or safety filter signals. In this work, we take a fundamentally different direction by proposing ELBO-KTO as a framework to train dLLMs directly from unpaired preference signals. Our approach integrates ELBO estimates into KTO’s value computation via an ELBO margin (policy vs. reference) and a baseline for variance control. This formulation unlocks learning from a broader class of real-world “good/bad” feedback signals while preserving training stability through lightweight variance reduction techniques. The main contributions are summarized as follows: • We present ELBO-KTO, a principled preference optimization framework that enables alignment of diffusion language models (dLLMs) from unpaired feedback. • We provide a theoretical analysis of the bias and variance tradeoff introduced by ELBO substitution, and prove that our estimator enjoys bounded bias with controlled variance. • We empirically validate our proposed method ELBO-KTO on kto-mix-14k ( TRL contributors 2024 ) and UltraFeedback-Binary ( Cui et al. 2023 ) , where it achieves 65.9% and 62.3% adjusted win rates over the base model, demonstrating consistent improvements. Extensive analysis confirms that our method provides a simple yet effective way to align dLLMs with stronger models. On downstream reasoning/knowledge tasks such as GSM8K and MMLU, ELBO-KTO trained on UltraFeedback-Binary performs on par with or better than the LLaDA-8B-Instruct. 2 PRELIMINARIES 2.1 Diffusion Large Language Models and LLaDA LLaDA ( Nie et al. 2025 ) is an 8B-parameter masked diffusion model (MDM) pretrained on 2.3T tokens (LLaDA-8B-Base) and fine-tuned on 4.5M instruction pairs (LLaDA-8B-Instruct). LLaDA achieves performance competitive with leading autoregressive LLMs, demonstrating strong scalability, in-context learning, and instruction-following capabilities. In this work, we adopt LLaDA-8B-Instruct as our base model for studying alignment in diffusion LLMs using the KTO algorithm. 2.2 VRPO: Variance Reduced Preference Optimization for dLLMs While Direct Preference Optimization (DPO) operates on sentence-level log-likelihoods, such quantities are intractable in masked diffusion LMs. Variance-Reduced Preference Optimization (VRPO) addresses this by replacing log-likelihoods with ELBO surrogates that decompose over diffusion steps ( Zhu et al. 2025 ) . ELBO. Given a prompt x , and a corresponding full response y , we draw a noised intermediate y_{t} from the forward process q(y_{t}\mid t,y,x) at a randomly sampled diffusion timestep t\sim\mathcal{U}[0,1] and evaluate a per-step loss \ell_{\pi} . Averaging over t and y_{t} yields the ELBO \mathcal{B}_{\pi}(y\mid x)=\mathbb{E}_{t\sim\mathcal{U}[0,1]}\ \mathbb{E}_{y_{t}\sim q(\cdot\mid t,y,x)}\!\big[\ell_{\pi}(y_{t},t,y\mid x)\big], (1) where \ell_{\pi} is the per-step loss of the mask prediction model as defined in Appendix A. \mathcal{B}_{\pi} is a lower-bound of \log\pi_{\theta}(y\mid x) ( Lou et al. 2024 ; Ou et al. 2024 ) . Monte Carlo Estimator. Sampling n_{t} time steps, \{t_{j}\}_{j=1}^{n_{t}}\sim\mathcal{U}[0,1] , and n_{y_{t}} draws per timestep, \{y^{(i)}_{t_{j}}\}_{i=1}^{n_{y_{t}}}\sim q(y_{t_{j}}\mid t_{j},y,x) , \widehat{\mathcal{B}}_{\pi}(y\mid x)=\frac{1}{n_{t}}\sum_{j=1}^{n_{t}}\frac{1}{n_{y_{t}}}\sum_{i=1}^{n_{y_{t}}}\ell_{\pi}\!\big(y^{(i)}_{t_{j}},t_{j},y\mid x\big) (2) is the Monte Carlo Estimator of \mathcal{B}_{\pi} . VRPO introduces techniques to reduce the variance of this estimator by (i) increasing the sampling budget, (ii) allocating more samples across diffusion steps by increasing n_{t} , and (iii) using common random numbers shared between policy and reference. These techniques help stabilize training and reduce the variance of loss and gradient. Reference-Adjusted ELBO margin. Given a desirable completion y_{w} and a undesirable completion y_{l} , VRPO compares their policy-vs-reference ELBO differences: \displaystyle\widehat{m}_{\theta}(x;y_{w},y_{l}) \displaystyle=\Big[\widehat{\mathcal{B}}_{\pi_{\theta}}(y_{w}\mid x)-\widehat{\mathcal{B}}_{\pi_{\theta}}(y_{l}\mid x)\Big] (3) \displaystyle-\Big[\widehat{\mathcal{B}}_{\pi_{\mathrm{ref}}}(y_{w}\mid x)-\widehat{\mathcal{B}}_{\pi_{\mathrm{ref}}}(y_{l}\mid x)\Big]. VRPO Loss. Finally, the DPO-style logistic loss over the margin is \mathcal{L}_{\mathrm{VRPO}}(\theta)=\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\bigl[-\log\sigma\bigl(\beta\,\widehat{m}_{\theta}(x;y_{w},y_{l})\bigr)\bigr] (4) where \sigma(\cdot) is the sigmoid and \beta>0 is a temperature. This loss encourages the policy to assign a relatively higher probability mass to y_{w} than to y_{l} under ELBO surrogates, thus teaching the preference to the model. 2.3 Kahneman–Tversky Optimization (KTO) KTO ( Ethayarajh et al. 2024 ) is a preference optimization algorithm inspired by prospect theory in behavioral economics, which models human perception of gains and losses as asymmetric. Unlike DPO, which relies on paired preference data, KTO can learn directly from unpaired binary signals (“desirable” vs. “undesirable”), making it well-suited for settings with limited or imbalanced feedback. Let \pi_{\theta} be the policy, \pi_{\mathrm{ref}} be a frozen reference, and \sigma(\cdot) the sigmoid function. KTO defines the reference–adjusted reward as r_{\theta}(x,y)\;=\;\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}. (5) This reward is mapped through a prospect-theoretic value function with separate loss-aversion weights for desirable and undesirable examples: v(x,y)=\begin{cases}\lambda_{D}\sigma(\beta\,(r_{\theta}(x,y)-z_{0})),&y\sim y_{\text{desirable}}|x,\\[3.0pt] \lambda_{U}\sigma(\beta\,(z_{0}-r_{\theta}(x,y))),&y\sim y_{\text{undesirable}}|x.\end{cases} (6) The baseline z_{0}=\text{KL}(\pi_{\theta}(\cdot|x)\,\|\,\pi_{\mathrm{ref}}(\cdot|x)) acts as a reference point to control loss saturation. Here, \beta>0 controls risk aversion, while \lambda_{D} and \lambda_{U} govern asymmetric loss aversion for desirable and undesirable samples. Assigning \lambda_{y}=\lambda_{D} for desirable and \lambda_{y}=\lambda_{U} for undesirable samples, the KTO objective is \mathcal{L}_{\mathrm{KTO}}(\theta)=\mathbb{E}_{(x,y)\sim\mathcal{D}}[\lambda_{y}-v(x,y)]. (7) Intuitively, desirable examples increase utility when the policy’s log-likelihood exceeds that of reference, while undesirable ones increase utility when it falls below the reference. The term \lambda_{y} ensures non-negativity. In practice, gradients are only taken with respect to \pi_{\theta} , while \pi_{\mathrm{ref}} and z_{0} are treated as stop-gradient terms, and we don’t backpropagte through them. 3 METHOD 3.1 ELBO-KTO for dLLMs KTO scores a response by the log-likelihood ratio \log\frac{\pi_{\theta}(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} , which is intractable for diffusion