Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization

Yunjae Won, Hyunji Lee, Hyeonbin Hwang, Minjoon Seo · 2025

Read Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization by Yunjae Won, Hyunji Lee, Hyeonbin Hwang, Minjoon Seo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23761.txt.

Introduction

Won and colleagues offer in 2025 a Bayesian reading of direct preference optimization, the popular method for aligning language models with human preferences. They define a differential information distribution: the samples that carry the evidence needed to move a reference policy to a target one. From that view, the log-ratio reward at the heart of the method is uniquely justified. Common training behaviors fall out of a power-law shape in that distribution.

The entropy of the learned information predicts downstream strengths. High entropy favored open-ended instruction following, while low entropy favored knowledge-heavy question answering. The authors present the theory as both foundation and practical guide.

Catalog record

CREATORYunjae Won, Hyunji Lee, Hyeonbin Hwang, Minjoon Seo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE106 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23761

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization Yunjae Won, Hyunji Lee, Hyeonbin Hwang, Minjoon Seo Abstract Direct Preference Optimization (DPO) has been widely used for aligning language models with human preferences in a supervised manner. However, several key questions remain unresolved: the rationale behind its log-ratio reward, how the statistical structure of preference datasets shapes its training dynamics, and how those dynamics impact downstream capabilities. We approach these questions from a Bayesian perspective, interpreting the goal of preference optimization as learning the differential information required to update a reference policy into a target policy. To formalize this view, we introduce the Differential Information Distribution (DID), defined as the distribution over samples that carry the Bayesian evidence required to update policies. We introduce three complementary insights by viewing preference optimization through the DID. First, we find that DPO’s log-ratio reward is uniquely justified when preferences encode the Differential Information needed to update a reference policy into the target policy. Second, we discuss how commonly observed training dynamics in DPO, including changes in log-likelihood and policy exploration, stem from a power-law DID relationship. Finally, we analyze how training dynamics influence downstream performance using the entropy of DID, a principled measure of uncertainty in the learned information. We observe that learning high-entropy DID improves open-ended instruction-following, while low-entropy DID benefits knowledge-intensive QA. Taken together, our results show that DPO’s reward design, training dynamics, and downstream capabilities all emerge as natural consequences of learning Differential Information, offering both a principled theoretical foundation and practical guidance for preference-based alignment. 1 Introduction Aligning language models to human preferences is essential for both safety and usefulness ( Ouyang et al., 2022 ; Bai et al., 2022 ) . Among various alignment methods, Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) has gained popularity for its strong empirical performance, training stability, and computational efficiency ( Xiao et al., 2024b ; Liu et al., 2025 ) . Despite its widespread use, several fundamental questions remain: what justifies the log-ratio reward beyond its derivation from a KL-regularized RL objective, what statistical structure in preference datasets underlie DPO’s training dynamics, and how these dynamics impact downstream capabilities. To address these gaps, we propose a Bayesian perspective that interprets the goal of preference optimization as learning the information needed to update a reference policy into a target policy. We call this information “Differential Information” as it represents the difference in the information encoded by the two policies. To formalize this, we introduce the Differential Information Distribution (DID), defined as the distribution of samples containing the Bayesian evidence 1 1 1 We consider Bayesian evidence that is conditionally independent of the prior given a sample, ensuring that this evidence reflects a sample’s intrinsic content rather than the distribution it was drawn from. This aligns with Shannon information being additive for independent events. See Section 2 and Appendix C for details. for this policy update. The DID can be expressed as the normalized ratio of the two policy distributions (Theorem 2.2 ). By analyzing the DID in preference optimization, we show that DPO’s key components–reward parameterization, training dynamics, and learned capabilities–emerge naturally from this Bayesian perspective. In Section 3 , we present a Bayesian interpretation of the optimality of DPO’s log-ratio reward parameterization. We first show how a preference data generation process naturally yields a preference distribution that encodes the Differential Information required to update a reference policy into a target policy. We then prove that the log-ratio reward is the unique Bradley-Terry reward that learns this target policy, revealing that DPO’s design follows directly from Bayesian principles. We validate these findings through controlled Energy-Based Model experiments, where we simulate a preference dataset encoding Differential Information and demonstrate that only the log-ratio reward converges to the target policy while other objectives ( e.g. , SimPO ( Meng et al., 2024 ) ) fail to do so. In Section 4 , we analyze the training dynamics of DPO based on a power-law relationship in the DID imposed by DPO. This DID relationship links the DPO-converged policy to the preference sampling distribution. Because the normalized ratio form of the DID is algebraically tied to the KL-divergence, it explains the consistent changes in the log-likelihood observed during DPO training. The DID power-law further clarifies how policy exploration is jointly influenced by the KL-penalty term and the sharpness of preference data, where sharpness reflects the sparsity (reciprocal of sampling temperature) of distributions over chosen and rejected responses. Using Energy-Based Models, we confirm these predictions and show that the identified properties of DPO persist under gradient-based stochastic optimization. In Section 5 , we investigate the empirical link between training dynamics and downstream capabilities by analyzing the Shannon entropy of the DID. Recent work finds that preventing log-likelihood displacement (LLD) improves factual accuracy ( e.g. , MMLU ( Hendrycks et al., 2021 ) ) at the cost of open-ended tasks ( e.g. , Wild-Bench ( Lin et al., 2024a ) ) ( Shi et al., 2024 ; Chen et al., 2024 ; Xiao et al., 2024a ) . We hypothesize that this trade-off reflects differences in the DID entropy. To test this, we compare standard DPO against a variant that prevents LLD. We train Mistral7B-v0.3 ( Jiang et al., 2024 ) and Qwen3-4B ( Team, 2025 ) on Magpie-Pro and Magpie-G27 datasets ( Xu et al., 2024b ) . Across all settings, preventing LLD consistently learns a low-entropy DID that improves factual QA, while allowing LLD learns a high-entropy DID that enhances open-ended generation. These results suggest that DID entropy could serve as a useful factor in characterizing whether a model’s learned capabilities align with factual precision or open-ended generation. Taken together, our Bayesian perspective unifies reward design, training dynamics, and learned capabilities of DPO. By explicitly linking preference data to the Differential Information they convey, our approach provides both theoretical grounding and practical guidance for designing and understanding preference-based alignment. 2 Preliminaries Let \mathcal{Y} denote the sample space of all possible sentences. A policy ( i.e. , language model) \pi defines a probability distribution over \mathcal{Y} , and we assume full support, i.e. , \pi(y)>0 for all y\in\mathcal{Y} . Let \pi^{*} be the target policy we wish to learn and \pi_{\mathrm{ref}} be a fixed reference policy. Preferences are ordered pairs (y_{w},y_{\ell}) , where y_{w} is preferred to y_{\ell} , written as y_{w}\succ y_{\ell} . The Bradley-Terry (BT) model ( Bradley & Terry, 1952 ; Luce et al., 1959 ) assigns the probability of such a preference under an implicit distribution p^{*} as p^{*}(y_{w}\succ y_{\ell})\coloneqq\frac{p^{*}(y_{w})}{p^{*}(y_{w})+p^{*}(y_{\ell})}. If a latent reward r:\mathcal{Y}\to\mathbb{R} induces a Boltzmann distribution P(Y=y\mid r)\propto\exp(r(y)) , then the BT preference probability can be expressed via the logistic sigmoid \sigma(x)=1/(1+\exp(-x)) as p(y_{w}\succ y_{\ell}\mid r)\coloneqq\sigma\big(r(y_{w})-r(y_{\ell})\big). DPO objective and log-ratio reward. Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) parameterizes a BT reward derived from the KL-regularized RL objective, the log-ratio between the learned policy and the reference: r_{\mathrm{DPO}}(y)\coloneqq\beta\log(\pi(y)/\pi_{\mathrm{ref}}(y)) , where \beta>0 corresponds to the KL-penalty strength. Under this parameterization, the preference probability becomes p(y_{w}\succ y_{\ell}\mid r_{\mathrm{DPO}})\coloneqq\sigma\!\left(\beta\log\tfrac{\pi(y_{w})}{\pi_{\mathrm{ref}}(y_{w})}-\beta\log\tfrac{\pi(y_{\ell})}{\pi_{\mathrm{ref}}(y_{\ell})}\right). Preference generation assumption. We assume preferences (y_{w},y_{\ell}) are sampled independently from two distributions \pi_{w} and \pi_{\ell} respectively. Given an unordered pair of distinct responses (y_{1},y_{2}) , the ground-truth

Read the complete text