Preference learning made easy: Everything should be understood through win rate

Lily H. Zhang, Rajesh Ranganath · 2025

Read Preference learning made easy: Everything should be understood through win rate by Lily H. Zhang, Rajesh Ranganath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10505.txt.

Introduction

Lily Zhang and Rajesh Ranganath published a paper in 2025 arguing that win rate is the one honest scoreboard for teaching models human preferences. Preference learning aligns generators with comparison data, yet the field lacks the conceptual tidiness of older areas. The authors start from how pairwise preference data are sampled and prove a sharp claim: only win-rate-style evaluation respects both the preferences and their frequency in the data.

Sorting popular methods into win-rate optimizers and the rest, they show the optimizers share two theoretical benefits that common alternatives like direct preference optimization lack. In practice, though, the optimizers are hard to optimize, and optimization success predicts results better than objective choice. Their advice: either pull the alternatives toward win rate or fix the optimizers' training.

Catalog record

CREATORLily H. Zhang, Rajesh Ranganath
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10505

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Preference learning made easy: Everything should be understood through win rate Lily H. Zhang, Rajesh Ranganath Abstract Preference learning, or the task of aligning generative models to preference comparison data, has yet to reach the conceptual maturity of classification, density estimation, etc. To close this gap, this work presents a framework to understand preference learning starting from the sampling distribution of pairwise preference data. First, we prove that the only evaluation of a generative model that respects both preferences and prevalences in the data distribution is a form of win rate, justifying win rate as the focal point to understand preference learning. We then analyze preference learning methods as win rate optimization (WRO) or non-WRO. We present novel instances of WRO beyond existing examples (RLHF, NLHF) and identify two key theoretical benefits of all such methods. We prove that common non-WRO methods like DPO and SFT on preferred samples lack these properties and suggest ways to mitigate such theoretical limitations. We also show that WRO underperforms in practice due optimization difficulties and that optimization success predicts performance better than choices which affect the objective’s solution. Our analysis highlights best practices for existing methods and provides recommendations for future research, guided by the principle that one should either align non-WRO methods more closely with WRO or improve the optimization of WRO objectives. 1 Introduction Learning from preference data, often referred to as human feedback, has emerged as a key step in training large language models, particularly given the success of reinforcement learning from human feedback (RLHF) ( Christiano et al. 2017 ) on state-of-the-art and high-profile language models such as GPT-4 ( OpenAI 2024 ) . The goal of learning from preference data is to finetune powerful base language models to output generations more in line with human preferences ( Stiennon et al. 2020 ; Ouyang et al. 2022 ) , motivated by the fact that pretraining on internet-scale data has enabled large language models to exhibit fluent generations of text ( Minaee et al. 2024 ) but not necessarily responses aligned with what humans prefer to see. In recent years, the landscape of algorithms and evaluations for preference learning has grown significantly ( Kaufmann et al. 2023 ; Jiang et al. 2024 ) , resulting in a complex and often fragmented field that can pose challenges for practitioners and researchers deciding where to focus their efforts. Whereas well-established machine learning tasks such as classification are grounded in principles such as maximum likelihood, preference learning lacks such a unifying conceptual foundation. To advance the field, we thus first ask: what underlying principles underpin preference learning? We address this question by developing a framework starting from the sampling distribution implied by pairwise preference data. We first tackle the question of evaluating a model under the preference learning paradigm. We show that the only evaluation of a generative model rooted in the preference data sampling distribution itself is a form of win rate we call h -win rate; any other notions of good either do not evaluate the model or respect the preference data, or are based on assumptions outside of either ( Section 3 ). From this insight, we introduce a win rate-centric framework for understanding the landscape of preference learning methods. Given h -win rate is the only relevant evaluation without additional assumptions, we relate common preference learning algorithms to directly optimizing for win rate, dividing the preference learning space into win rate optimization (WRO) and non-WRO objectives. In Section 4 , we generalize the space of WRO beyond existing methods in the literature and identify two theoretical benefits of such methods: (1) win rate-correspondence, i.e., optimizing the objective corresponds to optimizing for h -win rate; and (2) win rate-consistency, i.e., the solution can achieve the maximum win rate possible over a competitor as regularization strength goes to zero. In Section 5 , we show that direct preference optimization (DPO) and other direct alignment algorithms ( Rafailov et al. 2024a ) fail win rate-correspondence, and that supervised finetuning (SFT) on preferred samples additionally fails win rate-consistency. In Section 6 , we show that despite their theoretical benefits, WRO methods underperform relative to expectations due to difficulties in optimization, which has a much more significant effect on performance than other design choices that distinguish WRO methods from each other. We conclude with takeaways for current practice and future research in preference learning ( Section 7 ). Our contributions can be summarized as follows: 1. We prove that the only evaluation of a generative model grounded in the preference data distribution is win rate. This result justifies using win rate as the focal point to understand the landscape of preference learning. 2. We present a win rate-centric framework to understand preference learning. Under this framework, we: (a) introduce win rate-correspondence and consistency and show that WRO methods satisfy both while non-WRO do not; (b) highlight the central role of optimization success for the performance; (c) motivate best practices for existing methods (e.g., win rate checkpointing with direct alignment algorithms, multiple seeds with WRO algorithms, inducing diversity for SFT, combining different approaches); and (d) provide recommendations for future research—namely, to focus on better surrogates for or improved optimization of WRO. 2 Related Work Our work is most closely related to previous work in win rate evaluation and optimization as well as analysis of RLHF and preference learning objectives. Win rate evaluation and optimization. Win rate is already a central evaluation in preference learning ( Li et al. 2023b ; Zheng et al. 2024 ) ; however, our work goes further and proves that it is the only evaluation grounded in the sampling distribution itself, thus motivating its use as the central object to understand the rest of the preference learning landscape. Several works have proposed methods that perform some form of win rate optimization ( Munos et al. 2023 ; Swamy et al. 2024 ; Rosset et al. 2024 ) , including as a minimax optimization with a dynamic (rather than fixed) opponent model. Our work provides a more general framework to characterize the landscape of win rate optimization (WRO) methods beyond the specific instances that currently exist in the literature. Moreover, our analysis points to the theoretical merit of these approaches, while our experiments demonstrate the implementation bottlenecks of them. Analyzing RLHF, DPO, and other preference learning methods. Our work is related to work that seeks to better understand RLHF, DPO, and other existing methods in preference learning (e.g., best-of-n). Examples include benchmarking generalization and diversity ( Kirk et al. 2024 ) , comparing on- vs off-policy approaches ( Tajwar et al. 2024 ) , investigating length bias ( Singhal et al. 2024 ) , and disentangling design choices empirically ( Ivison et al. 2024 ) . For RLHF in particular, existing works consider the complexity of proximal policy optimization ( Ahmadian et al. 2024 ) , vanishing gradients ( Razin et al. 2024b ) , reward model overoptimization ( Zhu et al. 2024 ) , or limitations of the Bradley-Terry assumption to relate preferences to rewards ( Wang et al. 2024b ; Azar et al. 2023 ; Munos et al. 2023 ) . For DPO, there exists not only large space of

Read the complete text