On Symmetric Losses for Robust Policy Optimization with Noisy Preferences
Soichiro Nishimori, Yu-jie Zhang, Thanawat Lodkaew, Masashi Sugiyama · 2025
Read On Symmetric Losses for Robust Policy Optimization with Noisy Preferences by Soichiro Nishimori, Yu-jie Zhang, Thanawat Lodkaew, Masashi Sugiyama (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24709.txt.
Introduction
Human ratings that train AI systems are noisy, and a 2025 paper by Soichiro Nishimori and co-authors builds a training method that shrugs off that noise. Reward models learn from pairs of answers ranked by people, yet raters make mistakes and carry biases that warp the lesson.
The team reframes reward learning as a sorting problem, which lets them borrow symmetric loss functions, a family known to resist label noise. Their method, SymPO, trains the policy with those losses. Proofs show the learned reward keeps the true ranking order even under noisy labels, and that order is all a policy needs to improve. Tests on made-up and real tasks back the claim.
Catalog record
| CREATOR | Soichiro Nishimori, Yu-jie Zhang, Thanawat Lodkaew, Masashi Sugiyama |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24709
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On Symmetric Losses for Robust Policy Optimization with Noisy Preferences Soichiro Nishimori, Yu-jie Zhang, Thanawat Lodkaew, Masashi Sugiyama Abstract Optimizing policies based on human preferences is the key to aligning language models with human intent. This work focuses on reward modeling, a core component in reinforcement learning from human feedback (RLHF), and offline preference optimization, such as direct preference optimization. Conventional approaches typically assume accurate annotations. However, real-world preference data often contains noise due to human errors or biases. We propose a principled framework for robust policy optimization under noisy preferences based on the view of reward modeling as a classification problem. This viewpoint allows us to leverage symmetric losses, well known for their robustness to the label noise in classification, for reward modeling, which leads to our Symmetric Preference Optimization (SymPO) method, a novel offline preference optimization algorithm. Theoretically, we prove that symmetric losses enable successful policy optimization even with noisy labels, as the resulting reward is rank-preserving—a property we identify as sufficient for policy improvement. Empirical evaluations on a synthetic dataset and real-world language model alignment tasks demonstrate the efficacy of SymPO. The code is available at https://github.com/nissymori/SymPO . 1 Introduction Policy optimization with human preferences aims to train a policy that aligns with human desires, given pairs of actions (a_{1},a_{2}) and annotations indicating which action is preferred ( a_{1}\succ a_{2} or a_{1}\prec a_{2} ) ( Ouyang et al., 2022 ; Stiennon et al., 2020 ; Rafailov et al., 2024 ) . This paradigm has become increasingly prominent in language model alignment, where the goal is to develop models that behave according to human values, preferences, and instructions. A central component of this process is reward modeling, which involves learning an underlying reward function from preference data. Reward modeling is the foundation for two major approaches to policy optimization using preference data: Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ) and offline preference optimization ( Tang et al., 2024 ) . In RLHF, a reward model is first trained and then used to fine-tune the policy via on-policy RL methods. In contrast, offline preference optimization directly learns the policy from the collected preference data based on a reward modeling objective, as exemplified by Direct Preference Optimization (DPO) ( Rafailov et al., 2024 ) . Most existing methods assume that preference labels are accurate. However, real-world preference data often suffer from noise due to annotation errors or systematic biases ( Gao et al., 2024 ) . This issue has garnered particular attention in offline preference optimization ( Gao et al., 2024 ; Chowdhury et al., 2024 ; Liang et al., 2024 ; Wu et al., 2024 ; Fisch et al., 2024 ) . These methods either require the prior knowledge of the noise rate ( Chowdhury et al., 2024 ) or entail additional hyperparameters ( Liang et al., 2024 ; Wu et al., 2024 ) to be tuned. Furthermore, they assume symmetric noise ( Van Rooyen et al., 2015 ) , where the preferences will flip with equal probability for a_{1}\succ a_{2} and a_{1}\prec a_{2} . However, preference noise is often asymmetric in practice. For instance, an annotator may systematically favor responses from GPT-4 ( Achiam et al., 2023 ) over those from GPT-3 ( Brown et al., 2020 ) regardless of quality, introducing one-sided bias. In this paper, we propose a general framework for learning reward models from noisy preferences, grounded in the perspective of binary classification. Viewing reward modeling as a risk minimization in binary classification provides both methodological and theoretical benefits. Figure 1: The symmetric losses are plotted with a solid line, while the 0-1 loss and convex losses with a dashed line. Methodologically, this classification viewpoint provides a principled way to handle asymmetric noise and enables the development of a robust objective function for reward modeling. Specifically, it makes an inherent structure in preference data explicit: swapping the positions of input pairs flips the label. Leveraging this, we show that asymmetric and symmetric noise are equivalent in reward modeling (Sec. 3.1 ). For robust reward modeling, we then employ symmetric losses , \ell:\mathbb{R}\rightarrow\mathbb{R} , that satisfy so called the symmetric condition: \ell(z)+\ell(-z)=K for some constant K as shown in Fig. 1 (Sec. 3.2 ). Symmetric losses are proven to be robust against the symmetric label noise in binary classification settings ( Van Rooyen et al., 2015 ; Charoenphakdee et al., 2019 ) , and we extend this result to show their robustness to the general asymmetric noise in reward modeling. This gives rise to our method: Sym metric P reference O ptimization ( SymPO ), a novel offline preference optimization algorithm based on the symmetric loss. In our theoretical analysis, we uncover a connection between a property of loss functions in binary classification and the success of policy optimization. We first pinpoint a sufficient condition of the reward for policy improvement in policy optimization: rank preservation meaning that actions’ ordering matches the true underlying reward function (Sec. 4.1 ). Next, we prove that minimizing the risk with classification-calibrated losses ( Bartlett et al., 2006 ) leads to rank preservation, showing that a broad class of loss functions—including symmetric ones—are appropriate for policy optimization. By combining the robustness of symmetric losses with this policy improvement guarantee, we provide theoretical justification for our method, SymPO (Sec. 4.2 ). Finally, we validate our approach through two experiments: a synthetic setup based on MNIST and language model alignment tasks. In the MNIST setting, we evaluate the robustness of symmetric losses for reward modeling. The language model alignment experiment shows that SymPO facilitates robust policy improvement with noisy preference data. Related Work. Research on noise-robust preference optimization has been extensive. Chowdhury et al. (2024) proposed Robust DPO (rDPO), which adapts noisy label learning techniques ( Natarajan et al., 2013 ) used in statistical machine learning to construct an unbiased loss from noisy preference data, given prior knowledge of the noise rate. Wu et al. (2024) categorized noise in preference data as pointwise and pairwise and proposed Distributionally Robust DPO (DrDPO), which applies distributionally robust optimization ( Duchi & Namkoong, 2021 ) to address these types of noise. Liang et al. (2024) introduced Robust Preference Optimization (ROPO), which combines noisy sample filtering, rejection sampling, and sigmoid-based regularization processes. Unlike these works, we address general asymmetric noise and prove its equivalence to symmetric noise in risk minimization. Furthermore, our method, SymPO, achieves robustness with theoretical guarantees for policy improvement while maintaining methodological simplicity. While prior work ( Tang et al., 2024 ) also framed reward modeling as a binary classification task, it focused on convex losses, such as logistic, exponential, and squared losses (see Fig. 1 ). In contrast, we explore non-convex symmetric losses for their robustness to noisy labels. Please refer to App. A for a complete list of related works. Contributions. The main contributions of this paper are as follows: 1) We leverage the binary classification perspective of reward modeling to prove the equivalence between asymmetric and symmetric noise, and we propose a new offline preference optimization method, SymPO, that exploits the robustness of symmetric losses. 2) We bridge the gap between classification theory and policy optimization by proving that classification-calibrated losses yield policy improvement, thereby