Learning a Canonical Basis of Human Preferences from Binary Ratings
Kailas Vodrahalli, Wei Wei, James Zou · 2025
Read Learning a Canonical Basis of Human Preferences from Binary Ratings by Kailas Vodrahalli, Wei Wei, James Zou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-24150.txt.
Introduction
Kailas Vodrahalli, Wei Wei, and James Zou find that human preferences collapse onto a tiny set of core values, in a paper from 2025. The preferences come from the rating data used to align AI models. Alignment pipelines collect binary choices between model answers. The authors ask what those piles of choices actually encode.
From nearly five thousand distinct preference descriptions, just twenty-one categories captured over 89 percent of the variation between people. The set behaves like a basis in linear algebra: a short coordinate system for taste. It held up on synthetic and real data, and within single topics. Fine-tuning on slices defined by these categories steered models as predicted. The authors offer the basis as a lens for auditing what alignment teaches.
Catalog record
| CREATOR | Kailas Vodrahalli, Wei Wei, James Zou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-24150
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning a Canonical Basis of Human Preferences from Binary Ratings Kailas Vodrahalli, Wei Wei, James Zou Abstract Recent advances in generative AI have been driven by alignment techniques such as reinforcement learning from human feedback (RLHF). RLHF and related techniques typically involve constructing a dataset of binary or ranked choice human preferences and subsequently fine-tuning models to align with these preferences. This paper shifts the focus to understanding the preferences encoded in such datasets and identifying common human preferences. We find that a small subset of 21 preference categories (selected from a set of nearly 5,000 distinct preferences) captures >89% of preference variation across individuals. This small set of preferences is analogous to a canonical basis of human preferences, similar to established findings that characterize human variation in psychology or facial recognition studies. Through both synthetic and empirical evaluations, we confirm that our low-rank, canonical set of human preferences generalizes across the entire dataset and within specific topics. We further demonstrate our preference basis’ utility in model evaluation, where our preference categories offer deeper insights into model alignment, and in model training, where we show that fine-tuning on preference-defined subsets successfully aligns the model accordingly. 1 Introduction One of the major breakthroughs that has enabled generative AI in recent years is the use of alignment techniques such as human feedback reinforcement learning (RLHF) [ 7 ] . RLHF and related techniques are broadly applicable and have seen use in alignment for general utility and helpfulness [ 4 , 5 , 22 , 3 , 10 ] , decreasing toxicity [ 1 , 17 ] , or a myriad of other combinations of specific preferences [ 30 , 32 ] . The general paradigm of RLHF techniques involves several steps. The first step is to construct a dataset of human preferences, typically in the form of binary choices between two possible outcomes or, more generally, as a ranking across many such choices. For language models, for example, this may come in the form of two possible answers to a user’s question, with an annotation for which answer the user prefers. Subsequent steps involve augmenting this first, often relatively small dataset, and finally fine-tuning the base model to better align its responses with the preferences encoded in the aforementioned human preference dataset. In this paper, however, we focus our attention to the human preference dataset. Rather than treat the dataset as simply a means for aligning the generative model, we seek to understand what preferences are actually encoded in that dataset , and, more broadly, what preferences humans typically have . We find that a relatively small subset of preferences encodes much of the variation between people. This result is not surprising, as these types of results have previously been observed. The seminal work of Turk and Pentland on eigenfaces found that a small set of canonical human faces can capture most physical variation across people [ 28 ] . It has also long been an object of study in psychology to group people by a small set of personality traits. There are many examples of such groupings ranging from the popular Myers-Briggs test [ 19 ] , to the more academically relevant Big Five [ 8 ] , or to the clinically relevant Minnesota Multiphasic Personality Inventory [ 11 ] . Our main contributions are as follows. (1) We develop a method for characterizing the preferences contained within human preference datasets. While we focus on data that contains binary preferences, the method generalizes to datasets with ranked preferences as well. (2) As a byproduct of this method, we also generate a richly annotated preference dataset that contains a hierarchical categorization of preferences as well as topic categorizations; we release this dataset as part of our results. (3) We discover a low-rank canonical set of human preferences. Despite significant variation in human preferences across topics, this set generalizes and works at both a dataset level and a topic level. (4) We validate the discovered preference sets using both synthetic and empirical methods. (5) And finally, we demonstrate the utility of our preference decomposition in evaluating and fine-tuning models to better align with individual users. Our dataset and associated code is made available at https://github.com/kailas-v/basis-of-human-preferences . (a) Each binary human choice is converted into a set of preferences and topic annotation. (b) Preferences and topics are aggregated and then independently refined, resulting in a small set of preferences covering most of the original dataset. Figure 1 : Our pipeline converts a binary rating into a set of common human preferences. (A) This process is run in parallel for each binary choice. (B) This results in close to 5,000 preferences and over 3,000 topics. These preferences and topics are aggregated and then refined, resulting in just 21 preferences and 21 topics covering >89% of the original dataset. 1.1 Related Works In the context of aligning large language models, human preferences are often encoded as binary or ranked choice data. The reason for this is that methods like RLHF and other competing methods typically involve training a reward model or optimizing directly using binary human preference comparisons [ 7 , 22 , 3 ] . These datasets can be categorized into two groups: (1) those that explicitly encode manually-selected preferences through either explicit annotations for the given preference(s) or through curated data collection [ 4 , 5 , 30 ] , and (2) those that focus on more generic helpfulness and utility data and not a specific set of preferences [ 33 , 9 , 16 ] . While some prior work catalogues the conversation topics present in these datasets [ 33 , 6 ] or characterizes some of the preferences implicitly learned by the reward model [ 25 , 29 ] , there has not been an explicit characterization of all the preferences encoded in generic preference data. Additionally, while the aforementioned datasets and methods are largely focused on improving model performance for a generic user, a few benchmarks and datasets instead focus on individual users, and so, encode a user-level set of preferences rather than high-level, aggregate user preferences [ 24 , 26 , 20 ] . Our work is distinct in several ways. First, we seek to build on existing pairwise preference datasets by characterizing the preferences encoded in them. Secondly, by doing so, we aim to decompose the broad concepts of “preference” or “helpfulness” into something more interpretable. This characterization allows for not only a more transparent understanding of the specific preferences encoded in the data (e.g., “concise”, “humor”, or “follows instructions”), but it is also useful for characterizing models and further aligning them along specific directions as dictated by individual users or task requirements. 2 Discovering a Representative Subset for Human Preferences To discover a canonical basis of human preferences, we leverage an existing dataset of binary human preferences. The choice to use binary preference data is not a limitation—ranked choice data can easily be adapted for our method by either converting it into a set of binary choice data or by slightly adjusting the model prompts we use. The binary choice dataset, which consists of human annotations for preferred responses, is used to uncover the implicit preference categories (e.g.,