Reflective Verbal Reward Design for Pluralistic Alignment The behavioral studies in this paper were approved by the University of Waterloo’s Human Research Ethics Board (File 46074).
Carter BlairKate LarsonEdith Law · 2025
Read Reflective Verbal Reward Design for Pluralistic Alignment The behavioral studies in this paper were approved by the University of Waterloo’s Human Research Ethics Board (File 46074). by Carter BlairKate LarsonEdith Law (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17834.txt.
Introduction
Carter Blair, Kate Larson, and Edith Law present a new way to align AI agents with personal values in a paper from two thousand twenty-five. Standard training merges everyone's feedback into one reward model. That can drown out minority views, since people hold different and sometimes clashing values.
Their method guides each user through a reflective dialogue about agent behavior. The user critiques examples and builds up their own preferences in words. That dialogue then serves as context for a model that scores new behavior for that user alone. In studies with thirty people, the approach beat simpler verbal reward models by about ten percent in accuracy and used fewer examples than standard supervised learning.
Catalog record
| CREATOR | Carter BlairKate LarsonEdith Law |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-17834
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reflective Verbal Reward Design for Pluralistic Alignment The behavioral studies in this paper were approved by the University of Waterloo’s Human Research Ethics Board (File 46074). Carter BlairKate LarsonEdith Law Abstract AI agents are commonly aligned with “human values” through reinforcement learning from human feedback (RLHF), where a single reward model is learned from aggregated human feedback and used to align an agent’s behavior. However, human values are not homogeneous–different people hold distinct and sometimes conflicting values. Aggregating feedback into a single reward model risks disproportionately suppressing minority preferences. To address this, we present a novel reward modeling approach for learning individualized reward models. Our approach uses a language model to guide users through reflective dialogues where they critique agent behavior and construct their preferences. This personalized dialogue history, containing the user’s reflections and critiqued examples, is then used as context for another language model that serves as an individualized reward function (what we call a “verbal reward model”) for evaluating new trajectories. In studies with 30 participants, our method achieved a 9-12% improvement in accuracy over non-reflective verbal reward models while being more sample efficient than traditional supervised learning methods. 1 Introduction Effectively aligning AI systems with human values requires addressing the inherent diversity and context-dependence of those values Schwartz 1992 ; Friedman et al. 2013 ; Le Dantec et al. 2009 . However, prevalent alignment techniques, such as RLHF Bai et al. 2022 , typically train a single reward model from preferences aggregated across multiple users. By default, these singular models implicitly make trade-offs among competing values and can disproportionately suppress minority viewpoints Siththaranjan et al. 2023 ; Chakraborty et al. 2024 . In response to this shortcoming, recent work has explored approaches that better preserve the diversity of human values Siththaranjan et al. 2023 ; Chakraborty et al. 2024 ; Poddar et al. 2024 . While the specifics differ, these approaches broadly increase the granularity of the reward signal to account for differences in opinion. For example, some methods learn multiple distinct reward models to capture different preference clusters Chakraborty et al. 2024 . Others learn reward models that output distributions rather than scalar values to represent uncertainty and variation in preferences Siththaranjan et al. 2023 , and others learn individualized reward models Poddar et al. 2024 . These higher-granularity approaches offer benefits for collective and individual alignment. In collective settings where we must aggregate preferences across many stakeholders, having granular representations of preferences enables making transparent choices about balancing competing values by explicitly aggregating rewards. Rather than implicitly averaging preferences, we can optimize for desired properties like egalitarian welfare. Meanwhile, for personal AI assistants, techniques that learn high granularity preference representations can give rewards that are more representative of the given user compared to the population average. However, a challenge exists: While a single reward model can leverage data from all users to learn a scalar reward, higher-granularity approaches must learn more complex patterns (e.g., preference distributions Siththaranjan et al. 2023 or personalized reward functions Poddar et al. 2024 ), which are more challenging to estimate due to their increased dimensionality and sparser per-pattern data coverage. A further complication arises from the process of preference construction in complex and novel domains, where humans actively construct their preferences rather than simply reveal them Lichtenstein and Slovic 2006 . While current reward modeling approaches typically rely on passive data collection and straightforward labeling, this may be insufficient: the process of constructing preferences benefits from deliberate reflection, which helps individuals transform their latent values into concrete preferences Fischhoff 1991 . This insight is supported by findings from psychology, consumer research, and deliberative polling, which demonstrate that prompting people to actively contemplate their underlying values and reasoning processes leads to more well-defined and considered preferences Hauser et al. 2014 ; Ver Donck et al. 2020 ; Fishkin and Luskin 2005 . To address these two challenges, we introduce Interactive-Reflective Dialogue Alignment (IRDA), a system that uses large language models (LLMs) to learn personalized reward functions through interactive dialogue. IRDA combines three components: (1) reflective verbal preference elicitation that guides users in articulating their values, (2) active learning to strategically select examples for human critique, and (3) LLM-driven verbal reward modeling where the LLM directly serves as the reward function by leveraging its in-context learning ability to generalize from sparse user feedback. IRDA’s architecture confronts the preference construction problem by replacing passive labeling with LLM-guided dialogues that provoke deliberate, context-sensitive reflection (System 2 cognition). At the same time, its data-efficient learning strategy (active few-shot learning) mitigates the difficulty of learning personalized reward functions. We evaluated IRDA with two user studies involving 30 participants in total. The first study (21 participants) focused on building a reward model for each user’s personal definition of “respectful behavior,” while the second (9 participants) explored ethical decision-making in autonomous vehicles. Across both studies, participants differed widely in their value judgments, and our system was able to capture these individual definitions of value-aligned behavior more accurately than baseline approaches. Our contributions are as follows: • A novel pipeline for aligning AI agents to individual values, informed by AI, HCI, and social science. • A comprehensive evaluation in two distinct domains, demonstrating that our system captures individual preferences more accurately than baselines. • Empirical characterization of how individuals diverge in their conceptions of value-aligned AI behavior. • Insights for future research on interactive systems that help end users construct, refine, and operationalize their latent values. 2 Related Work Pluralistic Alignment. Recent work has highlighted the importance of moving beyond monolithic reward models and toward approaches that capture heterogeneous or uncertain human preferences in AI alignment. For instance, distributional preference learning (DPL) estimates an entire distribution over possible reward values, thereby accommodating hidden context and diverse annotator criteria Siththaranjan et al. 2023 . Similarly, methods like MaxMin-RLHF learn a mixture of reward models and optimize an egalitarian objective to avoid disproportionately favoring majority viewpoints Chakraborty et al. 2024 . Others have proposed user-specific latent variables that personalize reward models without requiring extensive per-user labels Poddar et al. 2024 or have leveraged meta-learning to reduce feedback requirements Hejna and Sadigh 2022 . However, these approaches assume users have direct access to their preferences in novel contexts despite evidence from preference elicitation and psychology literature suggesting otherwise. Our work complements these advances by actively eliciting fine-grained, user-specific preferences and helping users turn their latent values into concrete preferences through guided reflection. Verbal Reward Design. A separate line of research explores using large language models (LLMs) to act as or generate reward functions. Some methods prompt LLMs to propose reward code, which is then used to train RL policies via standard optimization Ma et al. 2024 ; Xie et al. 2024 ; Verma et al. 2024 ; Behari et al. 2024 . Other methods directly treat an LLM as a proxy reward function by prompting it with desired behavior descriptions Kwon et al. 2022 . These LLM-based approaches have made reward specification more accessible, particularly in domains where handcrafting objectives is difficult. However, they treat preference specification as a one-way street, where users tell the LLM what they want. Our system makes it a two-way dialogue, using LLMs both to help users