Jackpot! Alignment as a Maximal Lottery

Roberto-Rafael Maura-Rivero, Marc Lanctot, Francesco Visin, Kate Larson · 2025

Read Jackpot! Alignment as a Maximal Lottery by Roberto-Rafael Maura-Rivero, Marc Lanctot, Francesco Visin, Kate Larson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19266.txt.

Introduction

Roberto-Rafael Maura-Rivero, Marc Lanctot, Francesco Visin, and Kate Larson tackle a flaw in how chat models learn from human choices, in a 2025 paper. The standard method can ignore what most people prefer. The team propose a fix drawn from voting theory, called maximal lotteries, which picks outcomes by fair chance over stated choices.

They show that a newer family of training methods, built on game-style learning, already comes close to the lottery result. Such methods thus gain the lottery's good traits, like respect for the majority view and fair handling of loops in people's choices. Tests back the claim on hard cases where the standard method fails. The work points to training that tracks human values more closely.

Catalog record

CREATORRoberto-Rafael Maura-Rivero, Marc Lanctot, Francesco Visin, Kate Larson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-19266

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Jackpot! Alignment as a Maximal Lottery Roberto-Rafael Maura-Rivero, Marc Lanctot, Francesco Visin, Kate Larson Abstract Reinforcement Learning from Human Feedback (RLHF), the standard for aligning Large Language Models (LLMs) with human values, is known to fail to satisfy properties that are intuitively desirable, such as respecting the preferences of the majority ( Ge et al., 2024 ) . To overcome these issues, we propose the use of a probabilistic Social Choice rule called maximal lotteries as a replacement for RLHF. We show that a family of alignment techniques, namely Nash Learning from Human Feedback (NLHF) ( Munos et al., 2023 ) and variants, approximate maximal lottery outcomes and thus inherit its beneficial properties. We confirm experimentally that our proposed methodology handles situations that arise when working with preferences more robustly than standard RLHF, including supporting the preferences of the majority, providing principled ways of handling non-transitivities in the preference data, and robustness to irrelevant alternatives. This results in systems that better incorporate human values and respect human intentions. \printAffiliationsOnly 1 Introduction Figure 1 : Although B is the option preferred by the majority , LLMs aligned with RLHF fail to capture that, returning R . Thus, RLHF violates major democratic properties such as majority rule, while methods that emulate Maximal Lotteries satisfy them. Reinforcement Learning from Human Feedback (RLHF) has emerged as the de-facto standard to align Large Language Models (LLMs) with human values and preferences. Using ideas from revealed preference theory in economics, current RLHF methods adapt the LLM’s distribution of generated text or tokens so as to maximize a reward model learned from the ratings of human evaluators. Despite its widespread use in fine tuning LLMs ( Touvron et al., 2023 ; OpenAI, 2023 ; Anthropic, 2023 ; Google, 2023 ) , it has been recognized that current approaches suffer from fundamental limitations in the human feedback, the reward model, and training the policy ( Casper et al., 2023 ) . These challenges include tradeoffs between the richness and efficiency of feedback types, with binary preferences between pairs of examples being more prominent ( Christiano et al., 2017 ) , the assumption that a single reward function can represent a diverse population, which leads to current approaches modelling differences among evaluators as noise instead of important sources of disagreement ( Baumler et al., 2023 ) or ambiguity ( Schaekermann et al., 2018 ) , and reward models failing to generalize even with perfect training data ( Skalse & Abate, 2023 ) . A number of recent papers have begun to explore alignment problems through the lens of Social Choice ( Ge et al., 2024 ; Dai & Fleisig, 2024 ; Mishra, 2023 ; Siththaranjan et al., 2024 ) , which provides principled methods for aggregating preferences, particularly for diverse populations, as well as tools and insights to understand the benefits and challenges that arise from that ( Brandt et al., 2016 ) . In a recent position paper, Conitzer et al. argued that methods from Social Choice Theory provide alternative approaches to current RLHF methodologies ( Conitzer et al., 2024 ) . In this paper we make the argument that a Probabilistic Social Choice function, maximal lotteries ( Fishburn, 1984a ) , is particularly well suited for RLHF and alignment problems. In particular, • We propose an alternative alignment method to RLHF based on Maximal Lotteries, a stochastic voting rule from Social Choice Theory. • We formally prove that game-theoretic approaches to preference modeling in RLHF, specifically Nash Learning from Human Feedback ( Munos et al., 2023 ) and its variants ( Calandriello et al., 2024 ; Swamy et al., 2024 ) , emulate maximal lotteries. • Through controlled experiments, we show that our approach produces LLM outputs that better reflect aggregate human preferences compared to standard RLHF, supporting the preferences of the majority, providing principled ways of handling non-transitivities in the preferences, and robustness to of irrelevant alternatives. 2 Background 2.1 Reinforcement Learning from Human Feedback RLHF involves training a reward model and then using this model to guide a policy (the LLM) through reinforcement learning. A reward model r_{\theta}(x,y) is trained to predict a score indicating how good the response y is to the prompt x . This model is learned from a dataset of pairwise comparisons, where human annotators indicate their preferences. The training objective involves maximizing the likelihood of correctly predicting the preferred option using a binary cross-entropy loss, \mathcal{L}(\theta)=-E_{(x,y^{+},y^{-})}[\log(\sigma(r_{\theta}(x,y^{+})-r_{\theta}(x,y^{-}))) . Here, (x,y^{+},y^{-}) represents a data point, with x being the prompt, y^{+} the preferred completion (the “winner”), and y^{-} the less preferred completion (the “loser”) , r_{\theta} is the reward model parameterized by \theta and \sigma is the sigmoid function. This loss function is based on the Bradley-Terry model ( Rafailov et al., 2023 ) which is the foundation of the classical Elo rating system ( Elo, 1978 ) . While this model is widely-used for RLHF, it has several well-documented problems that could affect preference learning ( Shah & Wainwright, 2017 ; Balduzzi et al., 2019 ; Bertand et al., 2023 ; Lanctot et al., 2023 ; Munos et al., 2023 ) . The LLM, acting as the policy \pi_{\phi} parameterized by \phi , is then trained using reinforcement learning algorithms like Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) . A simplified objective (ignoring regularization) can be written as \max_{\phi}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\phi}(x)}[r_{\theta}(x,y)] , where \mathcal{D} is the distribution of prompts. This loss encourages the LLM to generate completions that receive high reward. 2.2 Social Choice Theory The central problem addressed by Social Choice Theory is how to aggregate the preferences of a population so as to reach some optimal collective decision. Assume there is a population \mathcal{P} of individuals, each of whom have preferences over some set of options \mathcal{Y} . Given a pair of different alternatives a,b\in\mathcal{Y} , an individual i\in\mathcal{P} is able to report that either they prefer a to b ( a\succ_{i}b ) or b to a ( b\succ_{i}a ). 1 1 1 For ease of exposition we will assume strict preferences in the rest of the paper, but results can be extended to weak preferences. A Social Choice function f is a map that assigns to each preference profile \{\succ_{i}\}_{i\in\mathcal{P}} a winning alternative in \mathcal{Y} , i.e. f(\{\succ_{i}\}_{i\in\mathcal{P}})\in\mathcal{Y} . A Probabilistic Social Choice function \rho is a similar concept that returns a distribution over the set of alternatives, \rho(\{\succ_{i}\}_{i\in\mathcal{P}})\in\Delta(\mathcal{Y}) . Much of Social Choice Theory is axiomatic in nature ( Brandt et al., 2016 ) , in that the field tries to understand what properties Social Choice functions can and should exhibit. For example, a Condorcet winner defines a fairly intuitive concept: an alternative a is a Condorcet winner if a preferred by more individuals than b in every head-to-head pairing for every b\in\mathcal{Y} (for a more formal definition of Condorcet winners, see Definition 2 in Section 3.2 ). Social Choice functions that are guaranteed to return a Condorcet winner when it exists are called Condorcet-consistent rules. Not all Social Choice functions are Condorcet consistent, like the well known class of scoring rules which include plurality and Borda. These rules translate individual’s preference rankings over m alternatives to a score vector \mathbf{w}=(w_{1},\ldots,w_{m}) where w_{1}\geq w_{2}\geq\ldots w_{m} and

Read the complete text