Preference-Based Alignment of Discrete Diffusion Models

Umberto Borso, Davide Paglieri, Jude Wells, Tim Rocktäschel · 2025

Read Preference-Based Alignment of Discrete Diffusion Models by Umberto Borso, Davide Paglieri, Jude Wells, Tim Rocktäschel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08295.txt.

Introduction

Umberto Borso, Davide Paglieri, Jude Wells, and Tim Rocktäschel adapt, in 2025 work, a taste-led tuning method to stepwise text builders that work on fixed alphabets. Such builders now craft code and symbol strings, yet tuning them to human taste was hard when no score card is at hand. Pair picks — this string over that — are the fuel here.

Their method bends the build path straight from pair picks, while staying near a trusted base form. A fresh loss does that work for builders cast as chained coin flips in smooth time. On a toy string task the tuned builder matched taste and kept its strings sound. The team sees a path past score-led drills.

Catalog record

CREATORUmberto Borso, Davide Paglieri, Jude Wells, Tim Rocktäschel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08295

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Preference-Based Alignment of Discrete Diffusion Models Umberto Borso, Davide Paglieri, Jude Wells, Tim Rocktäschel Abstract Diffusion models ( Ho et al., 2020 ; Song et al., 2020 ) have achieved state-of-the-art performance across multiple domains ( Austin et al., 2021 ; Watson et al., 2023 ; Anand & Achim, 2022 ) , with recent advancements extending their applicability to discrete data ( Lou et al., 2023 ; Shi et al., 2024 ; Campbell et al., 2022 ; Campbell et al., 2024 ) . However, aligning discrete diffusion models with task-specific preferences remains challenging, particularly in scenarios where explicit reward functions are unavailable. In this work, we introduce D iscrete D iffusion DPO ( D2-DPO ), the first adaptation of Direct Preference Optimization (DPO) ( Rafailov et al., 2024 ) to discrete diffusion models formulated as continuous-time Markov chains. Our approach derives a novel loss function that directly fine-tunes the generative process using preference data while preserving fidelity to a reference distribution. We validate D2-DPO on a structured binary sequence generation task, demonstrating that the method effectively aligns model outputs with preferences while maintaining structural validity. Our results highlight that D2-DPO enables controlled fine-tuning without requiring explicit reward models, making it a practical alternative to reinforcement learning-based approaches. Future research will explore extending D2-DPO to more complex generative tasks, including language modeling and protein sequence generation, as well as investigating alternative noise schedules, such as uniform noising, to enhance flexibility across different applications. 1 Introduction Diffusion models have emerged as powerful generative models, achieving state-of-the-art results in a variety of domains, including image generation ( Ho et al., 2020 ; Song et al., 2020 ) and molecular design ( Watson et al., 2023 ; Anand & Achim, 2022 ) . While originally formulated in continuous spaces, recent advancements have extended diffusion models to discrete domains ( Austin et al., 2021 ; Campbell et al., 2022 ) , including language modelling ( Lou et al., 2023 ; Shi et al., 2024 ; Sahoo et al., 2024 ; Ou et al., 2024 ) , symbolic music composition ( Campbell et al., 2022 ) and biological sequence generation ( Campbell et al., 2024 ) . Discrete diffusion models have demonstrated remarkable effectiveness in tasks where autoregressive approaches struggle, particularly in capturing long-range dependencies and modelling global consistency. However, in many applications, generating plausible sequences alone is insufficient. One often seeks to optimize generation with respect to specific task objectives, such as increasing factual accuracy in text generation, generating more harmonious music compositions, or designing protein sequences with improved stability. To address this challenge, recent works have explored fine-tuning pre-trained discrete diffusion models to optimize task-specific reward functions ( Wang et al., 2024 ) . However, explicitly defining a reward function is often infeasible when generation quality depends on subjective or hard-to-quantify criteria. In such cases, experts’ feedback can provide valuable guidance: they can qualitatively compare generated candidates and express preferences based on fundamental knowledge of the domain. Direct Preference Optimization (DPO) has recently emerged as a powerful method for fine-tuning generative models based on preference data, eliminating the need for explicit reward modelling. It has been successfully applied in natural language processing to align model responses with human feedback ( Rafailov et al., 2024 ) , in text-to-image generation to improve adherence to human aesthetic preferences ( Wallace et al., 2024 ) , and in protein design to enhance the stability of generated sequences ( Widatalla et al., 2024 ) . Despite its success in autoregressive and continuous generative models, DPO has not been explored for discrete diffusion models, which differ fundamentally in their formulation and training dynamics. In this work, we introduce D iscrete D iffusion DPO ( D2-DPO ), the first adaptation of DPO to discrete diffusion models. Unlike continuous diffusion models, which leverage score-matching, discrete diffusion models are formulated as Continuous-Time Markov Chains (CTMCs), requiring a different optimization framework. We derive a novel loss function that directly fine-tunes discrete diffusion models using pairwise preference data while preserving fidelity to a reference distribution. Our key contributions are as follows. Firstly, we introduce D2-DPO , a DPO-based optimization framework tailored for CTMCs, enabling preference alignment in discrete diffusion models without requiring an explicit reward function. Secondly, we show that under a masking-state noising process, our preference-based objective simplifies to an intuitive closed-form expression, providing theoretical insights into its effectiveness. Thirdly, we empirically validate D2-DPO on a structured sequence generation task, demonstrating that it successfully aligns discrete diffusion models with preferences while maintaining distributional coherence. 2 Background and Notation 2.1 Discrete Diffusion Models Continuous-Time Markov Chain (CTMC). A CTMC describes a sequence of discrete states \{x_{t}\} evolving over continuous time t\in[0,1] . The process begins at t=0 with an initial state x_{0}\sim p_{0} , and transitions between states occur stochastically governed by a rate matrix R_{t}\in\mathbb{R}^{\mathcal{X}\times\mathcal{X}} . The probability of transitioning from state x_{t} to x_{t+dt} over an infinitesimal time interval dt is given by: \displaystyle p_{t+dt|t}\left(x_{t+dt}|x_{t}\right)=\delta(x_{t},x_{t+dt})+R_{t}(x_{t},x_{t+dt})dt, (1) where \delta is the Kronecker delta function, which equals 1 when x_{t+dt}=x_{t} and 0 otherwise. The off-diagonal elements of the rate matrix, R_{t}(j,k)\geq 0 for j\neq k , specify the rate at which probability mass transitions from state j to state k at time t . The diagonal elements R_{t}(j,j)=-\sum_{k\neq j}R_{t}(j,k) represent the total rate at which probability mass moves out of state j and are thus negative. Noising Process. The noising process q_{t|1}(x_{t}|x_{1}) progressively perturbs the data distribution p_{1}(x)=p_{\text{data}}(x) gradually transforming it into the noise prior p_{0}(x)=p_{\text{noise}}(x) as t\to 0 . A widely used approach is the masking-state noise process ( Shi et al., 2024 ; Sahoo et al., 2024 ; Ou et al., 2024 ) which gradually maps all states x\in\mathcal{X} to a masked state M as t\to 0 . Under this scheme, the noise prior is p_{\text{noise}}^{\text{mask}}(x)=\delta\{M,x\} , and the state space is augmented to \mathcal{X}\cup\{M\} . The corresponding transition kernel for this process is given by: \displaystyle q_{t|1}^{\text{mask}}\left(x_{t}|x_{1}\right)=t\delta(x_{1},x_{t})+(1-t)\delta(M,x_{t}). (2) Generative Modelling. To generate samples from p_{\text{data}}(x) , we begin by drawing the initial noisy state from the noise prior, x_{0}\sim p_{\text{noise}}(x) , and then simulate the trajectory \{x_{t}\}_{t=0}^{t=1} by iteratively applying the transition kernel p_{t+dt|t}(x_{t+dt}|x_{t}) . This process allows the system to evolve towards the target distribution, ensuring that the final state at t=1 is effectively a sample from the clean data distribution, i.e., x_{1}\sim p_{\text{data}}(x) . Reconstructing the transition kernel in equation 1 requires knowledge of the rate matrix R_{t}(x_{t},x_{t+dt}) . Campbell et al. (2024) demonstrate that this matrix can be expressed as an expectation over a simpler conditional rate matrix. Specifically, we can write: \displaystyle R_{t}(x_{t},x_{t+dt})=\mathbb{E}_{p_{1|t}(x_{1}|x_{t})}\left[R^{q}_{t}(x_{t},x_{t+dt}|x_{1})\right], (3) where p_{1|t}(x_{1}|x_{t}) represents the denoising distribution, which we approximate using a neural network p_{1|t}^{\theta}(x_{1}|x_{t}) . We define the rate matrix R_{t}^{\theta}(x_{t},x_{t+dt}) by substituting p_{1|t}^{\theta}(x_{1}|x_{t}) into the expectation. The conditional rate matrix R^{q}_{t}(x_{t},x_{t+dt}|x_{1}) depends on the chosen noise schedule and is defined as: \displaystyle R^{q}_{t}(x_{t},x_{t+dt}|x_{1})=\frac{\operatorname{ReLU}\left(\partial_{t}q_{t|1}(x_{t+dt}|x_{1})-\partial_{t}q_{t|1}(x_{t}|x_{1})\right)}{S\cdot q_{t|1}(x_{t}|x_{1})}. (4) 2.2 Direct Preference Optimization Bradley-Terry (BT) Model. We assume access to a dataset of pairwise preferences \mathcal{P} over clean data samples x_{1} . Each preference is represented as a tuple (x_{1}^{w},x_{1}^{l},c) , where c\in\mathcal{C} represents a conditioning variable, x_{1}^{w} is the preferred sample, and x_{1}^{l} is the less preferred sample. The ranking between samples is assumed to

Read the complete text