Misaligned by Reward: Socially Undesirable Preferences in LLMs
Gayane Ghazaryan & Esra Dönmez · 2026
Read Misaligned by Reward: Socially Undesirable Preferences in LLMs by Gayane Ghazaryan & Esra Dönmez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05003.txt.
Introduction
Gayane Ghazaryan and Esra Dönmez examine whether the reward models used to align language models actually prefer socially desirable answers. Reward models stand in for human preferences during training, yet most testing covers only general instruction following. Social failures can therefore stay hidden behind good headline scores.
The authors convert social evaluation datasets on bias, safety, morality, and ethics into paired choices and test five public reward models plus two instruction-tuned stand-ins. No model led overall, and all often preferred socially undesirable options, producing biased output patterns. Stronger bias avoidance also dulled sensitivity to context, revealing a trade-off at the heart of alignment.
Catalog record
| CREATOR | Gayane Ghazaryan & Esra Dönmez |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05003
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Misaligned by Reward: Socially Undesirable Preferences in LLMs Gayane Ghazaryan & Esra Dönmez Abstract Reward models are a key component of large language model alignment, serving as proxies for human preferences during training. However, existing evaluations focus primarily on broad instruction-following benchmarks, providing limited insight into whether these models capture socially desirable preferences . As a result, important failures in social alignment can remain hidden. We extend reward-model benchmarking to four socially consequential domains: bias , safety , morality , and ethical reasoning . We introduce a framework that converts social evaluation datasets into pairwise preference data, leveraging gold labels where available and directional bias indicators otherwise. This enables us to test whether reward models prefer socially undesirable responses, and whether their preferences produce systematically biased distributions over selected outputs. Across five publicly available reward models and two instruction-tuned models used as reward proxies, we find substantial variation across domains, with no single model performing best overall. The models fall well short of strong social intelligence: they often prefer socially undesirable options, and their preferences produce systematically biased distributions. Moreover, stronger bias avoidance can reduce sensitivity to context, revealing a key alignment trade-off between avoiding biased outcomes and preserving contextual faithfulness. These findings show that standard reward benchmarks are insufficient for assessing social alignment and highlight the need for evaluations that directly measure the social preferences encoded in reward models. 1 Introduction As widely used general-purpose systems, large language models (LLMs) must be aligned with socially desirable preferences, including the avoidance of harmful, biased, or unethical outputs. Yet, a growing body of work shows that even state-of-the-art LLMs can still produce biased, unfair, toxic, or subtly offensive content, raising concerns about sociotechnical alignment and the downstream effects of these systems on individuals and communities ( 50 ; 12 ; 11 ) . In practice, alignment is commonly implemented through preference-based optimization approaches that steer models toward desirable behavior, most notably Reinforcement Learning from Human Feedback (RLHF) ( 27 ) . In RLHF, reward models (RMs) play a central role by assigning scalar scores to model outputs based on human preference data and thereby guiding optimization during fine-tuning ( 36 ) . Prior work has introduced reward-benchmarking to systematically evaluate reward models across alignment domains, such as broad safety and reasoning ( 25 ; 28 ) , yet these benchmarks provide little coverage of social domains, including bias, fairness, and moral judgment. Consequently, whether reward models reliably encode socially desirable preferences remains largely unknown , leaving potential failures in social alignment poorly understood. We address this gap by (i) introducing a framework for converting social evaluation datasets into pairwise preference data, leveraging gold labels where available and directional bias indicators otherwise, and (ii) extending reward benchmarking to social alignment with four new evaluation domains: bias , safety , morality , and ethical reasoning . The resulting framework constructs pairwise preference datasets from existing social evaluation benchmarks in a format compatible with existing pipelines ( 25 ) . Using this setup, we evaluate five publicly available reward models and two instruction-tuned models used as reward proxies, asking RQ1: whether they prefer socially undesirable responses and RQ2: whether their preferences induce systematically biased output distributions . We find substantial variation across domains, with no single model performing best overall (§ 5.6 ). The models fall well short of strong social intelligence: they often prefer socially undesirable options, and their preferences produce systematically biased distributions. Moreover, stronger bias avoidance can reduce sensitivity to context, revealing a key alignment trade-off between avoiding biased outcomes and preserving contextual faithfulness. 2 Related Work Social Harms. Work on evaluating LLMs has identified several recurring social harms. First, models often reproduce or prefer stereotypical associations about social groups, especially along dimensions such as gender, race, religion, age, and profession ( 31 ; 29 ) . Second, they can generate toxic, hateful, or offensive language , including more implicit forms of hate speech ( 14 ; 18 ; 35 ) . Third, they can exhibit unfair or uneven treatment across demographic groups in downstream tasks, for example by favoring dominant-group interpretations under ambiguity or producing different answers for otherwise comparable cases ( 38 ) . More broadly, recent survey work organizes these failures as forms of social bias and fairness violations, including representational and allocational harms ( 13 ) . Together, these findings show that LLM failures in social alignment span stereotyping, toxicity, unfair group-dependent behavior, and inconsistent moral judgment, but they have been studied primarily at the level of model outputs rather than the reward signals used to align them . Alignment and Reward Models. Work on alignment aims to steer LLMs toward responses that better match human preferences and social expectations, such as being helpful, safe, and avoiding harmful outputs ( 36 ; 3 ) . In modern systems, this is typically done during post-training using preference-based methods. In Reinforcement Learning from Human Feedback (RLHF), human preference data is used to train a reward model that scores candidate responses, and the policy is then optimized toward higher-reward outputs ( 36 ) . Related approaches such as Constitutional AI and DPO also use preference signals to shape model behavior, either through explicit reward models or implicit preference optimization ( 4 ; 40 ) . Prior work shows that these methods can improve instruction following and safety, but also involve trade-offs across helpfulness, harmlessness, and contextual sensitivity ( 36 ; 4 ; 27 ; 5 ) . Since they guide policy learning by defining preferred responses, reward models are a central component of alignment pipelines and a plausible source of downstream social failures . Reward Model Evaluation. Despite their central role in alignment, reward models have historically received much less direct attention than the policies they supervise. Recent work has begun to address this gap with dedicated benchmarks such as RewardBench , RewardBench 2 , and M-RewardBench , which evaluate reward models on chat, reasoning, safety, multilingual, and more challenging downstream-relevant preference tasks ( 26 ; 28 ; 16 ) . Other work studies reward models more directly, including their robustness, calibration, and sensitivity to data collection and optimization choices ( 48 ; 42 ; 20 ) . A particularly relevant recent direction examines bias and fairness in reward modeling itself, arguing that harmful downstream behavior may arise not only from pretrained models but also from the reward signals used during alignment ( 37 ; 43 ; 17 ; 24 ) . However, existing reward-model benchmarks mostly target chat quality, reasoning, and broad safety, while social-harm benchmarks are seldom built for pairwise preference evaluation . This makes social benchmarking difficult, since tasks must be converted into preference comparisons without losing labels, directional judgments, or context. As a result, it is still unclear whether current reward models reliably capture socially desirable preferences in areas like bias, morality, and ethical reasoning. 3 Methods To evaluate reward models on social domains, we first require a unified preference-based framework that (1) converts dataset instances into prompt–candidate outputs and (2) defines a corresponding measure of social alignment . Correspondingly, preference pairs are derived from the original annotations or templates of each dataset. For datasets with normative labels, the preferred continuation is the socially aligned option