Doubly Robust Alignment for Large Language Models
Erhan Xu †, Kai Ye †, Hongyi Zhou †, Luhan Zhu, Francesco Quinzan ‡, Chengchun Shi ‡ · 2025
Read Doubly Robust Alignment for Large Language Models by Erhan Xu †, Kai Ye †, Hongyi Zhou †, Luhan Zhu, Francesco Quinzan ‡, Chengchun Shi ‡ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01183.txt.
Introduction
Erhan Xu and co-authors harden language model alignment against wrong assumptions in a 2025 paper. Training from human feedback leans on a preference model, a reference policy, and a reward guess. When any of them is off, tuning can go astray. Their doubly robust method borrows a classic statistics safety net: it stays correct if either the preference model or the reference policy is right, with no need for both.
Theory and practice both favor the new algorithm over leading rivals, they report. Gains hold up across misspecified settings where older methods wobble. Code is public. The result promises steadier fine-tuning when human taste is hard to pin down.
Catalog record
| CREATOR | Erhan Xu †, Kai Ye †, Hongyi Zhou †, Luhan Zhu, Francesco Quinzan ‡, Chengchun Shi ‡ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 114 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01183
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Doubly Robust Alignment for Large Language Models Erhan Xu †, Kai Ye †, Hongyi Zhou †, Luhan Zhu, Francesco Quinzan ‡, Chengchun Shi ‡ Abstract This paper studies reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. While RLHF has demonstrated promising results, many algorithms are highly sensitive to misspecifications in the underlying preference model (e.g., the Bradley-Terry model), the reference policy, or the reward function, resulting in undesirable fine-tuning. To address model misspecification, we propose a doubly robust preference optimization algorithm that remains consistent when either the preference model or the reference policy is correctly specified (without requiring both). Our proposal demonstrates superior and more robust performance than state-of-the-art algorithms, both in theory and in practice. The code is available at https://github.com/DRPO4LLM/DRPO4LLM 1 Introduction Recent advances in large language models (LLMs) have revolutionized various natural language processing tasks, ranging from text generation to human-AI conversation and more complex reasoning tasks [ Brown et al., 2020 , Touvron et al., 2023 , Guo et al., 2025 ] . These models are typically trained in two stages. In the pre-training stage, LLMs learn general linguistic patterns and commonsense knowledge from vast, unlabeled text data through autoregressive next-token prediction. However, pretrained models face a critical objective mismatch: while they are optimized for token prediction, real-world deployment requires alignment with complex human values such as helpfulness, honesty and harmlessness [ Askell et al., 2021 ] . This mismatch calls for an additional post-training stage, aiming at better aligning these pre-trained models with human preference. The paper studies reinforcement learning from human feedback (RLHF), a post-training paradigm that adapts pre-trained models through reinforcement learning [ Sutton et al., 2018 , RL,] . The RLHF literature has rapidly expanded in recent years, where existing algorithms can be broadly categorized as reward-based or preference-based (Section 2 for a review). While demonstrating remarkable success in domains including robotics control, video games, and LLMs fine-tuning [ Christiano et al., 2017 , Ziegler et al., 2019 , Bai et al., 2022 , Bakker et al., 2022 , Ouyang et al., 2022 , see e.g.,] , they often suffer from various model misspecifications (see also Table 1 for a summary): 1. Preference model misspecification . Most reward-based algorithms rely on the Bradley-Terry [ Bradley and Terry, 1952 , BT,] preference model (see Equation 1 ). However, this model entails various unrealistic assumptions on human preference, including transitivity, context-independence and perfect relationality, which are likely violated based on empirical evidence [ May, 1954 , Tversky, 1969 , Gardner, 1970 , Agranov and Ortoleva, 2015 , Michaud et al., 2020 , Milano et al., 2021 , Lindner and El-Assady, 2022 ] . While some preference-based algorithms impose more general preference model (GPM) assumptions [ Zhang et al., 2024d , see e.g.,] , their effectiveness still depends on correct model specification. 2. Reward model misspecification . Under the BT model assumption, classical reward-based algorithms first estimate the reward function from human preference data and then apply RL algorithms such as the proximal policy optimization [ Schulman et al., 2017 , PPO,] to derive the optimal policy. However, policy learning through RL is highly sensitive to the estimated reward. Misspecifying the reward can lead to reward hacking [ Skalse et al., 2022 , Laidlaw et al., 2024 ] and misguide policy learning [ Kaufmann et al., 2023 , Zheng et al., 2023 , Chen et al., 2024 ] . 3. Reference policy misspecification . To alleviate misspecification of the reward, recent algorithms based on direct preference optimization [ Rafailov et al., 2023 , DPO,] propose to express the reward in closed form using the reference policy for policy learning. However, these algorithms are sensitive to the specification of reference policy [ Liu et al., 2024b , Gorbatovski et al., 2024 , Xu et al., 2024 ] . Drawing from doubly robust estimation methods in econometrics and RL (see Section 2 for a literature review), we introduce a novel RLHF algorithm that is robust to model misspecification and statistically efficient; see Figure 1 for a visualization of our algorithm. Our major contributions are summarized as follows: • We propose a robust and efficient estimator for preference evaluation, i.e., evaluating the probability of a target policy being preferred over the reference policy. The proposed preference estimator achieves two desirable properties: (i) double robustness (Corollary 5.6 ) – it converges to the true preference probability when either the preference model or the reference policy is correctly specified, and (ii) semi-parametric efficiency (Corollary 5.7 ) – it attains the smallest mean squared error (MSE) among all regular and asymptotically linear estimators [ Newey, 1990 , Tsiatis, 2006 ] . • Leveraging this preference estimator, we further develop a preference optimization algorithm for LLM fine-tuning. The proposed algorithm maintains double robustness (Corollary ( 5.9 )) and remains consistent even when the BT model assumption is violated (Theorem 5.8 ). Meanwhile, when the BT model assumption holds, its suboptimality gap is less sensitive to the reward model and reference policy compared to PPO- or DPO-based algorithms, and is likely smaller than that of both types of algorithms (Theorem 5.10 ). Figure 1: A visualization of our proposed preference optimization algorithm. \widehat{\pi}_{\textrm{ref}} denotes the specified reference policy whereas \widehat{g} denotes the specified preference model. Our proposal is doubly robust in that it requires correct specification of either the reference policy, or the preference model. 2 Related Works Our work is closely related to reward- and preference-based RLHF algorithms, as well as doubly robust (DR) methods. We discuss these related works below. Reward-based RLHF. Reward-based algorithms assume the existence of a latent utility or reward function that determines human preferences, estimate the reward function from the data and apply RL for policy learning. Recent research has focused on addressing practical challenges such as reward hacking and model-collapse. These issues arise due to PPO’s sensitivity to reward specification, gradient clipping thresholds, and the tuning parameter controlling Kullback–Leibler (KL)-divergence regularization [ Engstrom et al., 2019 , Zheng et al., 2023 , Xiao et al., 2024 ] . Existing approaches to these challenges fall into three categories: (i) The first category focuses on improving the reward learning algorithm to obtain more accurate reward functions [ Li et al., 2023b , Chan et al., 2024 , Gao et al., 2024 , Liu et al., 2024a , Fu et al., 2025 , Xiao et al., 2025 , Ye et al., 2025 ] . (ii) The second category develops better policy learning algorithms using the estimated reward function [ Wu et al., 2024a , Zhang et al., 2024a , Shao et al., 2024 , Hu, 2025 , Liu et al., 2025b , Yu et al., 2025 , Yuan et al., 2025 ] . (iii) The third category is DPO-based, which bypasses reward learning entirely and directly optimizes policies under the BT model assumption [ Zhao et al., 2022 , Wang et al., 2023a , Ethayarajh et al., 2024 , Song et al., 2024 , Tang et al., 2024 ] . Unlike many of these algorithms, our proposal does not rely on the BT model assumption, and is more robust to the misspecification of reward or reference policy when the BT model holds. Preference-based