Beyond the Binary: Capturing Diverse Preferences With Reward Regularization

Vishakh PadmakumarNew York University [email protected], Chuanyang JinJohns Hopkins University [email protected], Hannah Rose KirkUniversity of Oxford [email protected] He HeNew York University [email protected] · 2024

Read Beyond the Binary: Capturing Diverse Preferences With Reward Regularization by Vishakh PadmakumarNew York University [email protected], Chuanyang JinJohns Hopkins University [email protected], Hannah Rose KirkUniversity of Oxford [email protected] He HeNew York University [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03822.txt.

Introduction

Vishakh Padmakumar, Chuanyang Jin, Hannah Rose Kirk, and He He wrote this machine learning paper, published in 2024. It argues that tuning chatbots on simple pick-one-of-two feedback misses what a broad crowd of users really wants.

Reward models are usually trained on pairs where a judge picks the better reply. The authors name two ways this fails. Some prompts have many right answers, so users split. Other replies are near-copies of each other, so the pick is a coin toss. In both cases, reward models track group taste weakly. Their fix estimates where users would disagree, using machine-made extra judgments, and trains the model to keep a margin there. The adjusted models matched group preferences better. The paper is a step toward chatbots that serve diverse users fairly.

Catalog record

CREATORVishakh PadmakumarNew York University [email protected], Chuanyang JinJohns Hopkins University [email protected], Hannah Rose KirkUniversity of Oxford [email protected] He HeNew York University [email protected]
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03822

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond the Binary: Capturing Diverse Preferences With Reward Regularization Vishakh PadmakumarNew York University [email protected], Chuanyang JinJohns Hopkins University [email protected], Hannah Rose KirkUniversity of Oxford [email protected] He HeNew York University [email protected] Abstract Large language models (LLMs) are increasingly deployed via public-facing interfaces to interact with millions of users, each with diverse preferences. Despite this, preference tuning of LLMs predominantly relies on reward models trained using binary judgments where annotators select the preferred choice out of pairs of model outputs. In this work, we argue that this reliance on binary choices does not capture the broader, aggregate preferences of the target user in real-world tasks. We propose a taxonomy that identifies two dimensions of subjectivity where different users disagree on the preferred output—namely, the Plurality of Responses to Prompts , where prompts allow for multiple correct answers, and the Indistinguishability of Responses , where candidate outputs are paraphrases of each other. We show that reward models correlate weakly with user preferences in these cases. As a first step to address this issue, we introduce a simple yet effective method that augments existing binary preference datasets with synthetic preference judgments to estimate potential user disagreement. Incorporating these via a margin term as a form of regularization during model training yields predictions that better align with the aggregate user preferences. 1 Introduction A ubiquitous step in the training of contemporary large language models (LLMs) is aligning their output with human preferences [ 1 , 2 , 3 , 4 , 5 ] . This process involves collecting human preference judgments over model outputs, which serve as the reward signal for either reinforcement learning with human feedback [ 1 , 6 ] or various direct alignment algorithms [ 7 , 8 ] . Conventionally, a single human annotator provides a binary judgment for a pair of model outputs [ 9 , 10 ] . This “preferentist” approach to alignment has been criticized [ 11 , 12 , 13 ] , pitted against alternatives such as deliberative procedures that establish normative standards among diverse stakeholders [ 14 , 15 ] or principle-based frameworks that seek higher-order rules or “constitutions” to guide AI behavior [ 16 , 17 , 18 ] . However, preference judgments remain central to LLM post-training due to their cognitive and cost efficiency —it is far less demanding for annotators to choose between two outputs than to articulate an ideal demonstration—and surprising efficacy in steering LLMs towards being more friendly, helpful, and adept at interpreting user intent [ 1 , 2 , 19 , 20 ] . So, even if preference fine-tuning is not the optimal solution, it is likely to remain a key component in LLM development for the foreseeable future, especially in industry settings. However, despite its ubiquity and early success, this dominant methodology for preference-tuning LLMs is now at odds with the reality of their deployment in society. Accessible via public interfaces, LLMs have amassed hundreds of millions of users [ 21 ] , who have diverse preferences, needs, and linguistic backgrounds. The single binary choice between outputs becomes an unreliable signal for this vast user base, particularly in subjective domains like safety [ 22 , 23 ] or toxicity [ 24 ] , and when judging the overall quality of the output [ 6 ] in absence of detailed guidelines for what counts as high-quality language [ 25 , 26 ] . In this paper, we address this inherent gap in development-deployment preference formulation. We argue that if preference tuning continues to be a crucial component in modern LLM training, steps must be taken to calibrate the preference signal so that it better translates from a single-annotator development setting to a multi-user deployment environment. We present our empirical argument in two phases: The “break it” phase demonstrates the need for more calibrated reward estimation through a detailed error analysis of existing reward models. We hypothesize that over-reliance on single-annotator judgments leads to noisy and poorly calibrated preference signals, especially on subjective examples where different annotators would disagree on the preferred choice. We present a qualitative taxonomy of reward model failures in such cases ( Section 3.1 ) and validate that reward model predictions on these examples correlate more poorly with human judgments ( Section 3.2 . The “fix it” phase tests a simple yet effective modification to the reward model training objective. Based on recent work [ 3 ] , we introduce a margin term in the objective function of reward modeling that is scaled to reflect the degree of disagreement of preference judgments in a hypothetical population of users. Our experiments show that these regularized model predictions better align with human preferences ( Section 3.3 ). Our contributions add to a body of work seeking alignment of LLMs that can better account for natural distributions of human preferences across global populations [ 27 , 28 , 29 ] . As a resolution to preference disagreements, some prior work has targeted personalization [ 30 , 31 , 32 ] , but our approach is more similar to those seeking more principled approaches to aggregate preference estimation [ 33 ] , for example, those drawing insight from social choice theory [ 34 , 35 , 36 ] , seeking distributional learning [ 37 , 38 ] or uncertainty modeling [ 39 ] . Our margin-based method can be seen as adopting a utilitarian approach to preference maximization with uniform weightings, but the framework can also accommodate other normative considerations of distributive justice in AI alignment. Figure 1 : Two dimensions of subjectivity along which reward model predictions correlate poorly with the diverse preferences of a population of users—when the prompt allows for multiple correct answers and when the two responses are indistinguishable from one another ( Section 3 ). Regularizing the training objective with a margin term ( Section 3.3 ) leads to predictions that better correlate with these preferences. Our method represents a simple impactful adjustment that is compatible with existing dominant paradigms of preference fine-tuning, though it is not a complete solution. By further refining our model and data methodologies for preference alignment, we can work towards AI systems that better accommodate the diversity of subjective human experiences. 2 Problem Formulation The canonical method to train a reward model, r_{\theta} , involves collecting a human preference judgment, j , that indicates which of two candidate outputs y\in\{y_{0},y_{1}\} is preferred for a given input prompt x , resulting in dataset \mathrm{D} . The model parameters are optimized by minimizing the loss function: \textrm{loss}(r_{\theta})=-\mathrm{E}_{(x,y_{0},y_{1},j)\sim\mathrm{D}}[\log(\sigma(r_{\theta}(x,y_{j})-r_{\theta}(x,y_{1-{j}})))], (1) where \sigma denotes the sigmoid function and j\in\{0,1\} . The reward model parameters are trained to maximize the difference in score assigned to y_{j} and y_{1-j} based on the preference judgment, j . This is effective when j is unanimously the ‘better’ output, as j would likely be consistent with the preferences of any future user interacting with the model. However, we argue that in practical applications of LLMs, the models are frequently required to make decisions on examples where there is legitimate disagreement among users. Consider a scenario where a set of n users each provide a judgment, j_{1\dots n} , on the same example. The intended behavior of the model is to predict

Read the complete text