MixDPO: Modeling Preference Strength for Pluralistic Alignment

Saki Imai, Pedram Heydari, Anthony Sicilia, Asteria Kaeberlein, Katherine Atwell, Malihe Alikhani · 2026

Read MixDPO: Modeling Preference Strength for Pluralistic Alignment by Saki Imai, Pedram Heydari, Anthony Sicilia, Asteria Kaeberlein, Katherine Atwell, Malihe Alikhani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06180.txt.

Introduction

Not all likes are felt with the same force. A mild like and a strong love look the same in most training data. Saki Imai and 5 co-authors fix that flaw in a 2026 paper on how AI learns from human choices. Their goal is a model that serves groups with tastes of their own.

Their method, MixDPO, grows out of a known choice-based tool. It learns how strongly each like is felt, case by case, much as choice theory in past work on trade did. Tests used three choice data sets and two open models. Scores rose as a whole, and group tastes stayed in place. Gains were best where felt force spread out most. The team says plain talk of force in the math helps AI keep real human range.

Catalog record

CREATORSaki Imai, Pedram Heydari, Anthony Sicilia, Asteria Kaeberlein, Katherine Atwell, Malihe Alikhani
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-06180

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MixDPO: Modeling Preference Strength for Pluralistic Alignment Saki Imai, Pedram Heydari, Anthony Sicilia, Asteria Kaeberlein, Katherine Atwell, Malihe Alikhani Abstract Preference based alignment objectives implicitly assume that all human preferences are expressed with equal strength. In practice, however, preference strength varies across individuals and contexts—a phenomenon established in behavioral economics and discrete choice theory. This mismatch limits the ability of existing objectives to faithfully capture heterogeneous human judgments. Inspired by this literature, we introduce Mix ed Logit D irect P reference O ptimization (MixDPO), a generalization of Direct Preference Optimization that models variation in preference strength. MixDPO enables alignment objectives to capture heterogeneity in how strongly preferences are expressed across training examples. We evaluate MixDPO on three preference datasets using two open-weight language models. Across datasets, MixDPO improves aggregate alignment performance (+11.2 points on Pythia-2.8B) while preserving subgroup level preferences, with the largest gains appearing in settings with higher inferred preference heterogeneity. MixDPO makes preference heterogeneity explicit through learned strength distributions. We release our code for reproducibility. Figure 1: Motivating example of latent preference strength heterogeneity. Although annotators provide only binary comparisons, variation in repeated preference judgments—holding preference direction fixed—can arise from latent differences in how sharply individuals distinguish between alternatives. Mixed logit models capture this variation through a random sensitivity parameter. 1 Introduction Preference based alignment methods 26 ; 25 train language models from human comparisons, implicitly assuming that all preferences exert equal influence during training. However, human preferences are inherently heterogeneous. Fig. 1 illustrates a scenario in preference based alignment. Given a prompt “Should companies be allowed to use AI to screen job applicants?” , annotators compare two responses emphasizing different considerations such as efficiency (A) versus fairness (B). Although annotators may agree on which response they prefer, they often differ in how strongly that preference is expressed. Importantly, preference strength is not directly observed in alignment datasets. Instead, we model it as a latent random variable whose distribution is inferred from variation in pairwise comparisons, following the mixed logit tradition. We argue that alignment under heterogeneous preferences requires modeling preference strength itself as a distribution. The existence of heterogeneous preference strength is a finding in behavioral economics and decision theory that long predates AI alignment 21 ; 8 . Mixed logit models were developed to capture this phenomenon by modeling preferences as draws from a distribution rather than assuming a fixed sensitivity 19 . Pluralistic datasets used for language model alignment inherit this heterogeneity through controversial or value guided prompts 17 ; 36 . Despite this, existing preference based alignment objectives assume a global preference strength, effectively treating all preference comparisons as equally important. A fixed sensitivity parameter couples frequency with influence: preference comparisons that occur more frequently dominate the training gradient. As a result, frequently occurring but weakly expressed preferences can outweigh infrequent but strongly expressed ones, causing aggregate improvements to mask regressions on underrepresented groups. We introduce Mix ed Logit D irect P reference O ptimization (MixDPO), which relaxes this assumption by modeling preference strength as a variable drawn from a learned distribution. MixDPO automatically downweights ambiguous or weakly expressed preferences while preserving strong signals, without requiring subgroup labels or heuristic reweighting. This formulation makes preference heterogeneity explicit and measurable via learned distributional parameters. We evaluate MixDPO on three preference datasets (Anthropic HH, PRISM, and Community Alignment) and investigate two questions. First, we examine whether aggregate evaluation metrics mask subgroup level alignment differences , by downweighting underrepresented subpopulations. Second, we assess whether MixDPO objectives preserve subgroup preferences without sacrificing aggregate performance . On PRISM, which is explicitly designed to capture value diversity, MixDPO achieves higher subgroup averaged preference margins across dimensions such as age, gender, education, and employment status, and conversation type, while remaining competitive under aggregate metrics. Contributions. We show that aggregate alignment metrics can mask subgroup level differences by disproportionately weighting frequent preference comparisons (§ 5.1 ). We introduce MixDPO , which allows preference comparisons to exert varying strengths of influence during optimization by modeling preference strength as a distribution(§ 3 ). We demonstrate that this distributional modeling leads to better preservation of subgroup preferences while maintaining competitive aggregate alignment performance (§ 5.2 ). Finally, we show that MixDPO captures dataset level preference heterogeneity and naturally collapses to fixed strength behavior when preferences are homogeneous (§ 5.3 ). 2 Background 2.1 Preference Optimization as Probabilistic Choice Preference based alignment methods can be viewed as learning a probabilistic choice model over pairs of model outputs. Given a prompt x and two candidate responses (y_{w},y_{l}) , a preference annotation indicates that y_{w} is preferred to y_{l} . Methods such as DPO 26 , KTO 11 , IPO 2 and SimPO 24 model this preference as arising from a reward function, with the probability of preferring y_{w} over y_{l} increasing with the reward difference. Under a standard logistic choice model, this probability takes the form P(y_{w}\succ y_{l}\mid x)=\sigma\big(\beta\,\Delta r\big) where \Delta r is the reward difference between the two responses and \beta>0 controls the sensitivity of the choice probability to this difference. Larger values of \beta yield more deterministic preferences while smaller values correspond to weaker preferences. In this sense, \beta controls the sensitivity of observed preferences to differences in reward. In existing alignment methods, \beta is typically treated as a fixed scalar shared across all training examples. This implies that all preference annotations are assumed to reflect the same level of confidence, consistency, or decisiveness, regardless of prompt, annotator, or context. 2.2 Preference Heterogeneity and Pluralism Empirically, human preferences are not expressed with uniform strength 21 . Preferences are known to be heterogeneous, context dependent, and often stochastic rather than fixed 29 . From the perspective of social choice theory, aggregating diverse and conflicting preferences has been recognized as a challenge, with no single aggregation rule capable of fully representing pluralistic values 1 . Recent work in machine learning has framed these concerns under the notion of pluralistic alignment , which emphasizes the need for systems to reflect diverse human values rather than optimizing for a single averaged objective 30 . Existing approaches have explored preference inputs that augment binary comparisons with multi-attribute annotations 18 , dispersion based modeling 7 , and ideal preference points 6 . While these methods address diversity in preference direction, they generally assume that preferences are expressed with uniform strength or reliability. 2.3 Mixed Logit Models for Preference Heterogeneity Mixed logit models (also known as the random-coefficients logit) represent heterogeneity in choice behavior by assuming that each decision instance is generated by a sample from a distribution over preference strength 21 . Instead of assuming a fixed sensitivity coefficient, mixed logit treats such coefficients as random variables drawn from a population level distribution. 19 show that mixed logit can approximate any random utility model arbitrarily well under mild regularity conditions. We provide additional background on mixed logit models and their use for modeling heterogeneous preferences in Appendix A . In the context of preference optimization, this corresponds to treating the preference sensitivity \beta as a random variable drawn from a distribution p(\beta) . The mean of this distribution captures the typical strength of preferences in the dataset, while its variance reflects the degree of heterogeneity in how strongly preferences are

Read the complete text