Generalizing while preserving monotonicity in comparison-based preference learning models
Julien FageotTournesol, Peva BlanchardKleis TechnologyGilles BareillesCTU in Prague, TournesolLê-Nguyên HoangCalicarpa, Tournesol · 2025
Read Generalizing while preserving monotonicity in comparison-based preference learning models by Julien FageotTournesol, Peva BlanchardKleis TechnologyGilles BareillesCTU in Prague, TournesolLê-Nguyên HoangCalicarpa, Tournesol (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08616.txt.
Introduction
Julien Fageot and three co-authors fix a fairness flaw in models that learn from comparisons, in a 2025 paper. When you say you prefer one option over another, the model's score for the winner should rise and the loser's should fall. That property, monotonicity, sounds obvious, yet many deployed preference learners, language models included, lack it.
Until now only one model family was proven monotone, and it cannot extend to options never compared. The authors add diffusion-based priors to that family and state conditions on option embeddings that keep monotonicity while allowing generalization. Experiments confirm the guarantee is rare in the wild and show their models gain accuracy most when data are scarce. The work matters wherever rankings feed decisions about people.
Catalog record
| CREATOR | Julien FageotTournesol, Peva BlanchardKleis TechnologyGilles BareillesCTU in Prague, TournesolLê-Nguyên HoangCalicarpa, Tournesol |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08616
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generalizing while preserving monotonicity in comparison-based preference learning models Julien FageotTournesol, Peva BlanchardKleis TechnologyGilles BareillesCTU in Prague, TournesolLê-Nguyên HoangCalicarpa, Tournesol Abstract If you tell a learning model that you prefer an alternative a a over another alternative b b , then you probably expect the model to be monotone , that is, the valuation of a a increases, and that of b b decreases. Yet, perhaps surprisingly, many widely deployed comparison-based preference learning models, including large language models, fail to have this guarantee. Until now, the only comparison-based preference learning algorithms that were proved to be monotone are the Generalized Bradley-Terry models DBLP:conf/aaai/FageotFHV24 . Yet, these models are unable to generalize to uncompared data. In this paper, we advance the understanding of the set of models with generalization ability that are monotone . Namely, we propose a new class of Linear Generalized Bradley-Terry models with Diffusion Priors, and identify sufficient conditions on alternatives’ embeddings that guarantee monotonicity. Our experiments show that this monotonicity is far from being a general guarantee, and that our new class of generalizing models improves accuracy, especially when the dataset is limited. 1 Introduction Preference learning, sometimes known as alignment , has become central to machine learning. In particular, in recent years, there has been a growing interest to leverage comparative judgments to fine-tune AI models, using frameworks like Reinforcement Learning with Human Feedback (RLHF) DBLP:conf/nips/ChristianoLBMLA17 or Direct Preference Optimization (DPO) rafailovDirectPreferenceOptimization2024 in the context of language models, or linear models in the context of applications ranging from trolley dilemmas to food donation awadMoralMachineExperiment2018 ; DBLP:journals/pacmhci/LeeKKKYCSNLPP19 .These models are now deployed at scale. Yet bizarre aspects of these preference learning algorithms are regularly observed. In particular, strikingly, DBLP:journals/corr/abs-2402-13228 ; DBLP:journals/corr/abs-2410-08847 ; pairwise-monotonicity showed that, when they are updated based on a comparative judgment saying that an alternative a is preferable to b , the probability of generating item a can in fact decrease. In fact, most deployed learning algorithms fail to guarantee monotonicity . A preferred alternative can be downgraded after it was said to be better than another alternative. Perhaps surprisingly, the root cause of this lack of monotonicity is not the nonlinearity of the models. In fact, we can expose this “bug" with a very basic example. Consider a linear two-dimensional model with a parameter \beta\in\mathbb{R}^{2} to be learned: the score of alternative a and b are \beta^{\top}x_{a} and \beta^{\top}x_{b} , where x_{a} and x_{b} are (two-dimensional) vector embeddings. We are given a comparison that favors alternative a over b , whose embeddings are x_{a}=(1,0) and x_{b}=(2,0) . Since x_{a1}<x_{b1} , this comparison will push \beta_{1} towards lower values. But since the score of a according to the linear model is \beta^{T}x_{a}=\beta_{1} , this means that the score of a will also decrease. Thus, including a comparison that favors a over b has decreased the score of a . This becomes all the more troubling when there exists an alternative c with embedding x_{c}=(0,1) , whose score remains unchanged. This example questions whether preference learning algorithms can be trusted. A user who witnesses a surprising evolution of alternatives’ scores as illustrated above might, understandably, prefer not to use such an algorithm. Worse yet, if they are nevertheless forced to use the algorithm, they could want to remove the data they previously provided, because the eventual learned model was deteriorated by their truthful data reporting. More generally, this may discourage users to report their preferences, and rather provide tactical preferences in the hope to steer the model towards their goal. This is reminiscent of tactical or “useful” strategies in voting systems. Now, some models, like the (Generalized) Bradley-Terry model, were proved to be monotone DBLP:conf/aaai/FageotFHV24 . But to the best of our knowledge, none of them enables score generalization to non-evaluated alternatives. Hence the following research problem. Can a generalizing comparison-based preference learning algorithm guarantee monotonicity? Contributions. Our first contribution is to identify a large class of preference learning algorithms that leverage both comparisons between alternative pairs, and descriptive information (embeddings) on individual alternatives, which we call Linear Generalized Bradley-Terry with Diffusion Prior (Definition 3 ). This class extends the (Generalized) Bradley-Terry models DBLP:conf/aaai/FageotFHV24 by including a linear mapping of the embeddings—and priors on alternative similarities, thereby allowing preference generalization to yet uncompared alternatives. As a second contribution, we provide conditions on the embeddings that guarantee that the learning algorithm behaves monotonically when new comparisons are provided. As discussed above, this property is highly desirable and yet hard to guarantee in practice. In particular, we propose a class of diffusion embeddings that guarantee monotony, and for which membership is easy-to-check. Interestingly, diffusion embeddings yield a very appealing interpretation as heat diffusion dynamics where comparisons play the role of heat pumps. A direct consequence is that categorical information (one-hot encoding embedding) yields a monotone learning algorithm. In particular, this class enables us to provide a positive answer to our research question. Finally, we evaluate the statistical performance of our learning algorithms through numerical experiments on synthetic datasets. Our evaluations show that a linear model with good embeddings and diffusion priors outperforms the classical GBT model DBLP:conf/aaai/FageotFHV24 , in particular when the amount of data is limited. Related works. Learning preferences from comparisons has a long history, dating back to Thurstone thurstone27 , Zermelo zermelo1929berechnung , and Bradley and Terry bradley1952rank . To handle inconsistent judgments, such algorithms define a probabilistic model of how latent scoring of alternatives are transformed into noisy comparisons. Their approach was adapted by luce1959individual and plackett1975analysis to model the selection of one preferred alternative out of several proposed ones. While the Bradley-Terry model models binary-valued comparisons, various authors have proposed extensions to real-valued comparisons, e.g., ranging the interval [-1,1] . Historically, this started with the modeling of draws in games like chess davidson1970extending . More recently, DBLP:journals/corr/abs-1911-11658 proposed the platform Climpact where users are given pairs of activities, and are asked to evaluate the comparative pollutions of two activities. They then develop a model based on a quadratic error to turn the comparisons into evaluations of the amounts of pollution of the individual activities. Their model was then generalized by DBLP:conf/aaai/FageotFHV24 in a framework they call Generalized Bradley-Terry (GBT) to turn real-valued comparisons into scores. All these models however consider that each alternative’s score is an independent latent variable to be learned. Thus, they fail to generalize to non-evaluated alternatives. Independent of user-provided comparisons, alternatives usually come with descriptive information. DBLP:journals/nca/MenkeM08 , csiszar2012algorithms and zhao2016deep proposed to model an alternative’s score as a parametrized function of a vector embedding of the description of the alternative. Recently, this trick has been widely used in the context of language models vaswani2017attention ; brown2020language , especially through algorithms like Reinforcement Learning with Human Feedback (RLHF) DBLP:conf/nips/ChristianoLBMLA17 ; DBLP:conf/nips/StiennonO0ZLVRA20 , Direct Preference Optimization (DPO) dpo , or Generalized Preference Optimization (GPO) DBLP:conf/icml/TangGZCMRRVPP24 , to name a few. In this paper, we restrict ourselves to linear models , where an alternative’s score is assumed to be a linear function of their embedding. Such linear models of preferences trained from comparative judgments have previously been studied and used, e.g. by DBLP:conf/ijcai/GuoTKOCCEDI18 ; DBLP:conf/aaai/NoothigattuGADR18 ; DBLP:journals/pacmhci/LeeKKKYCSNLPP19 . The study of the mathematical guarantees of preference learning algorithms has only