From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
Tarun Raheja, Nilay Pochhi, [0.5em] Independent Researchers[0.3em] {tarunraheja1234, pochhi.nilay}@gmail.com, [0.5em] Equal contribution · 2026
Read From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models by Tarun Raheja, Nilay Pochhi, [0.5em] Independent Researchers[0.3em] {tarunraheja1234, pochhi.nilay}@gmail.com, [0.5em] Equal contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06108.txt.
Introduction
Many ways to teach language models human preferences look unrelated. Tarun Raheja and 1 co-author unify them in a 2026 survey. Reinforcement learning from human feedback set the pattern, then direct methods multiplied, leaving practitioners unsure which to pick.
The survey shows the diversity reduces to choices along three axes: the preference model behind the objective, how deviation from a reference model is controlled, and whether learning is online or offline. Theorems cover coverage gaps, scaling laws for reward overoptimization, and when direct methods fail. Known failure modes like length hacking and mode collapse follow predictably from design choices. A decision guide built from over fifty papers helps practitioners choose.
Catalog record
| CREATOR | Tarun Raheja, Nilay Pochhi, [0.5em] Independent Researchers[0.3em] {tarunraheja1234, pochhi.nilay}@gmail.com, [0.5em] Equal contribution |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-06108
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models Tarun Raheja, Nilay Pochhi, [0.5em] Independent Researchers[0.3em] {tarunraheja1234, pochhi.nilay}@gmail.com, [0.5em] Equal contribution Abstract Aligning large language models (LLMs) with human preferences has become essential for safe and beneficial AI deployment. While Reinforcement Learning from Human Feedback (RLHF) established the dominant paradigm, a proliferation of alternatives—Direct Preference Optimization (DPO), Identity Preference Optimization (IPO), Kahneman-Tversky Optimization (KTO), Simple Preference Optimization (SimPO), and many others—has left practitioners without clear guidance on method selection. This survey provides a theoretical unification of preference learning methods, revealing that the apparent diversity reduces to principled choices along three orthogonal axes: (I) Preference Model (what likelihood model underlies the objective), (II) Regularization Mechanism (how deviation from reference policies is controlled), and (III) Data Distribution (online vs. offline learning and coverage requirements). We formalize each axis with precise definitions and theorems, establishing key results including the coverage separation between online and offline methods, scaling laws for reward overoptimization, and conditions under which direct alignment methods fail. Our analysis reveals that failure modes—length hacking, mode collapse, likelihood displacement—arise from specific, predictable combinations of design choices. We synthesize empirical findings across 50+ papers and provide a practitioner’s decision guide for method selection. The framework transforms preference learning from an empirical art into a theoretically grounded discipline. Keywords: Preference learning, RLHF, direct preference optimization, Bradley-Terry model, language model alignment 1 Introduction The Orthodoxy. Since Christiano et al. [ 2 ] introduced learning from human preferences and Ouyang et al. [ 3 ] demonstrated its transformative potential with InstructGPT, Reinforcement Learning from Human Feedback (RLHF) has become the canonical approach for aligning large language models with human values. The standard RLHF pipeline consists of three stages: (1) supervised fine-tuning (SFT) on high-quality demonstrations, (2) reward model training on pairwise human preferences, and (3) policy optimization via Proximal Policy Optimization (PPO) [ 4 ] to maximize the learned reward while staying close to the SFT policy. This paradigm underlies the alignment of GPT-4 [ 5 ] , Claude, and other frontier models. The Challenge. Yet RLHF’s dominance belies significant practical difficulties. PPO is notoriously unstable, requiring careful hyperparameter tuning and multiple model copies [ 12 , 13 ] . The reward model introduces a “proxy” that can be exploited, leading to Goodhart’s law violations where optimizing the proxy degrades true performance [ 11 ] . These challenges motivated a wave of alternatives: Direct Preference Optimization (DPO) [ 6 ] eliminates the reward model entirely; Identity Preference Optimization (IPO) [ 7 ] addresses DPO’s overfitting to deterministic preferences; Kahneman-Tversky Optimization (KTO) [ 8 ] requires only binary feedback; Simple Preference Optimization (SimPO) [ 9 ] removes the reference model; and Odds Ratio Preference Optimization (ORPO) [ 10 ] unifies SFT with preference learning. The resulting landscape is bewildering. When should one use DPO versus PPO? Why does SimPO outperform DPO despite apparent simplicity? Why do all methods suffer from length hacking? Practitioners face an embarrassment of riches without theoretical guidance. The Resolution. This survey provides that guidance through a unified theoretical framework. We show that the zoo of preference learning methods reduces to principled choices along three orthogonal axes : • Pillar I: Preference Model — What probabilistic model relates rewards to preferences? (Bradley-Terry, Plackett-Luce, prospect-theoretic, game-theoretic) • Pillar II: Regularization — How is policy deviation from a reference controlled? (Explicit KL, implicit KL, reference-free, f-divergences) • Pillar III: Data Distribution — What assumptions on data coverage enable learning? (Online, offline, hybrid) This taxonomy is not merely organizational—it is predictive . We show that failure modes arise from specific combinations: length hacking from reward model bias interacting with insufficient regularization; mode collapse from overly strong KL penalties; likelihood displacement from offline data limitations. Contributions. This survey makes four contributions: 1. A unified theoretical framework showing DPO, IPO, KTO, SimPO, ORPO, and GRPO as special cases of a general \Psi PO objective ( Section 3 ). 2. Formal theorems establishing the coverage separation between online/offline methods, conditions for preference collapse, and overoptimization scaling laws ( Sections 5 and 6 ). 3. A systematic taxonomy of failure modes connecting theoretical properties to empirical pathologies ( Section 7 ). 4. A practitioner’s decision guide providing actionable recommendations based on constraints ( Section 9 ). 2 Preliminaries: Choice Theory Meets Language Models We establish the mathematical foundations connecting classical choice theory to modern LLM alignment. 2.1 Notation and Setup Let \mathcal{X} denote the space of prompts and \mathcal{Y} the space of responses. A policy \pi:\mathcal{X}\to\Delta(\mathcal{Y}) maps prompts to distributions over responses. We write \pi(y|x) for the probability of response y given prompt x . A reward function r:\mathcal{X}\times\mathcal{Y}\to\mathbb{R} assigns scalar values to prompt-response pairs. 2.2 The Bradley-Terry Model The Bradley-Terry model [ 1 ] provides the foundational preference model for RLHF. Definition 2.1 (Bradley-Terry Preference Model) . Given a reward function r , the probability that response y_{1} is preferred to y_{2} given prompt x is: p^{*}(y_{1}\succ y_{2}|x)=\sigma(r(x,y_{1})-r(x,y_{2})) (1) where \sigma(z)=1/(1+e^{-z}) is the sigmoid function. The Bradley-Terry model makes two key assumptions: (1) preferences depend only on reward differences , making the reward scale-free; (2) preferences are transitive —if y_{1}\succ y_{2} and y_{2}\succ y_{3} , then y_{1}\succ y_{3} with high probability. Both assumptions are violated in practice, motivating extensions we discuss in Section 4 . 2.3 The Canonical RLHF Objective The standard RLHF objective balances reward maximization against deviation from a reference policy \pi_{\mathrm{ref}} : Definition 2.2 (KL-Regularized Reward Maximization) . The RLHF objective is: \max_{\pi}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi(\cdot|x)}\left[r(x,y)-\beta\mathrm{KL}(\pi(\cdot|x)\|\pi_{\mathrm{ref}}(\cdot|x))\right] (2) where \beta>0 controls regularization strength and \mathcal{D} is the prompt distribution. The KL penalty serves multiple purposes: preventing reward hacking by keeping \pi near the capable \pi_{\mathrm{ref}} ; maintaining response diversity; and ensuring the optimization problem is well-posed. Theorem 2.3 (Optimal Policy Form [ 6 , 15 ] ) . The optimal policy for Equation 2 has the closed form: \pi^{*}(y|x)=\frac{1}{Z(x)}\pi_{\mathrm{ref}}(y|x)\exp\left(\frac{r(x,y)}{\beta}\right) (3) where Z(x)=\sum_{y}\pi_{\mathrm{ref}}(y|x)\exp(r(x,y)/\beta) is the partition function. This result is the foundation of direct preference optimization methods: by inverting Equation 3 , rewards can be expressed in terms of policies, eliminating the need for explicit reward modeling. 3 A Unified Framework: \Psi PO We present the \Psi PO framework of Azar et al. [ 7 ] , which unifies preference learning methods and reveals their structural relationships. 3.1 The Two Approximations in RLHF Standard RLHF makes two approximations that introduce potential errors: 1. Pointwise Reward Approximation : Pairwise preferences are converted to pointwise rewards via the Bradley-Terry model, discarding relational information. 2. Reward Model Generalization : A reward model trained on finite data must generalize to out-of-distribution responses sampled by the policy. DPO addresses the second approximation by eliminating the reward model but retains the first. The \Psi PO framework addresses both. 3.2 The General \Psi PO Objective Definition 3.1 ( \Psi PO Objective [ 7 ] ) . For a convex function \Psi:\mathbb{R}\to\mathbb{R} , the \Psi PO objective is: \mathcal{L}_{\Psi}(\pi)=\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\Psi\left(\beta\log\frac{\pi(y_{w}|x)}{\pi_{\mathrm{ref}}(y_{w}|x)}-\beta\log\frac{\pi(y_{l}|x)}{\pi_{\mathrm{ref}}(y_{l}|x)}\right)\right] (4) where y_{w} denotes the preferred (“winning”) response and y_{l} the dispreferred (“losing”) response. Different choices of \Psi recover existing methods: Theorem 3.2 (Instantiations of \Psi PO) . • \Psi(z)=-\log\sigma(z) : DPO [ 6 ] • \Psi(z)=(z-1)^{2} : IPO [ 7 ] • \Psi(z)=\max(0,1-z) : SLiC-HF [ 14 ] This unification reveals that the methods