No Preference Left Behind: Group Distributional Preference Optimization

Binwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, Junjie Hu · 2024

Read No Preference Left Behind: Group Distributional Preference Optimization by Binwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, Junjie Hu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20299.txt.

Description

Abstract Preferences within a group of people are not uniform but follow a distribution. While existing alignment methods like Direct Preference Optimization (DPO) attempt to steer models to reflect human preferences, they struggle to capture the distributional pluralistic preferences within a group. These methods often skew toward dominant preferences, overlooking the diversity of opinions, especially when conflicting preferences arise.

Catalog record

CREATORBinwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, Junjie Hu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20299

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

No Preference Left Behind: Group Distributional Preference Optimization Binwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, Junjie Hu Abstract Preferences within a group of people are not uniform but follow a distribution. While existing alignment methods like Direct Preference Optimization (DPO) attempt to steer models to reflect human preferences, they struggle to capture the distributional pluralistic preferences within a group. These methods often skew toward dominant preferences, overlooking the diversity of opinions, especially when conflicting preferences arise. To address this issue, we propose Group Distributional Preference Optimization (GDPO), a novel framework that aligns language models with the distribution of preferences within a group by incorporating the concept of beliefs that shape individual preferences. GDPO calibrates a language model using statistical estimation of the group’s belief distribution and aligns the model with belief-conditioned preferences, offering a more inclusive alignment framework than traditional methods. In experiments using both synthetic controllable opinion generation and real-world movie review datasets, we show that DPO fails to align with the targeted belief distributions, while GDPO consistently reduces this alignment gap during training. Moreover, our evaluation metrics demonstrate that GDPO outperforms existing approaches in aligning with group distributional preferences, marking a significant advance in pluralistic alignment. Our data and code are released at https://github.com/BigBinnie/GDPO . 1 Introduction Recent studies ( Durmus et al., 2023 ; Jarrett et al., 2023 ; Chuang et al., 2024b ) have explored the potential of large language models (LLMs) to reflect people’s opinions and preferences across a wide range of topics. This line of research opens up various possibilities, such as modeling the opinions of a population on specific topics ( Santurkar et al., 2023 ) , estimating human responses to surveys or new policies ( Argyle et al., 2023 ; Aher et al., 2023 ) , and even simulating hypothetical social interactions between individuals ( Park et al., 2022 ; Park et al., 2023 ; Chuang et al., 2024a ) . As a result, aligning LLMs to accurately reflect diverse preferences within a group is crucial. However, existing alignment approaches like Reinforcement Learning from Human Feedback (RLHF; Ouyang et al., 2022 ) and Direct Preference Optimization (DPO; Rafailov et al., 2024 ) still struggle to capture the diversity of opinions that exist within a group ( Durmus et al., 2023 ; Zhao et al., 2023 ) . For instance, if we ask a group of people the following question: “Is the availability of overseas products good for our country?” , we are likely to get diverse responses from individuals living in the same county, in the same town, or even in the same family ( Kirk et al., 2024 ) . This highlights that human preferences, even within a single group, are not singular but in fact distributional , or, pluralistic . This creates the challenge of conflicting preferences , where some annotators prefer one response while others prefer a different one. As a result, existing alignment algorithms (e.g., DPO) may be adversely affected by pairwise noise ( Wu et al., 2024 ) , as they assume the existence of a shared preference among people. This leads to a focus on seeking commonsense rather than representing diverse preferences during the alignment process. Consequently, the alignment results become biased toward dominant preferences, overlooking minority preferences within the group. With this challenge in mind, we focus on a research question: How can a probabilistic LLM align with distributional preferences within a group? To answer this question, we propose Group Distribution Preference Optimization (GDPO) , a novel framework designed to pluralistically align LLMs with the distributional preferences of a group. In particular, we introduce the concept of belief from epistemology, which represents the degree to which individuals agree with a particular stance, to the preference alignment process. Preferences are inherently shaped by people’s beliefs ( Sharma et al., 2023 ) . Conflicting preferences exist when individuals hold different beliefs about the same topic. Motivated by this epistemological phenomenon, our GDPO framework optimizes two main objectives to ensure that conflicting preferences no longer interfere with one another. First, to encourage the model to generate diverse beliefs that reflect the group distribution, we calibrate the model’s belief predictions using a statistical estimate of the belief distribution within a group. Second, we introduce a belief-conditioned preference alignment objective to resolve preference conflicts by constructing preference data pairs conditioned on their corresponding preferred beliefs. Similar to the DPO training, we begin GDPO training from a checkpoint derived from supervised fine-tuning (SFT). Figure 1: Demonstration of GDPO. Training Dataset : We create the belief-conditioned preference datasets for training, where people’s beliefs on a topic are diverse according to a specified distribution, and their preferences are conditioned on those beliefs. Training Objective : Instead of optimizing all preferences simultaneously, GDPO first calibrates the belief distribution, followed by belief-conditioned preference alignment. Inference Time : When a new query is received, the model predicts a belief and generates responses based on it. To validate GDPO’s performances on group distributional preference alignment, we apply GDPO to two tasks: 1) controllable opinion generation on a synthetic dataset; 2) controllable review generation on a real-world dataset. For the controllable opinion generation, the model first generates an opinion as its belief, then produces a response expressing that opinion (as the training dataset in Figure 1 ). We construct the synthetic dataset to simulate one-turn dialogues that reflect diverse opinions from various countries, using GlobalOpinionQA ( Durmus et al., 2023 ) , a multi-choice question-answer dataset focused on global issues. For the controllable review generation, we use movie reviews from Amazon users to build the preference dataset. Here, the model is required to predict a rating score as the belief first, then generate a review that justifies that score. Our experiments, conducted using GPT-2 Large ( Radford et al., 2019 ) and Pythia-2.8B ( Biderman et al., 2023 ) , reveal that DPO faces challenges in aligning with the target distribution, particularly when dealing with minority beliefs. Specifically, we observe that the gap between the predicted and actual belief distributions widens significantly during DPO training. In contrast, GDPO effectively mitigates this issue, enabling the model to progressively approximate the target distribution throughout training. Moreover, we introduce two sets of evaluation metrics to assess distribution calibration and conditional generation performance. Our results demonstrate that GDPO outperforms existing preference alignment methods, excelling in both distribution alignment and conditional generation tasks. 2 Related Work Preference Alignment. LLMs require additional training with human feedback to better align their outputs with human preferences, such as Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ) and Direct Preference Optimization (DPO) ( Rafailov et al., 2024 ) . Both methods train on datasets of pairwise human preferences ( Bai et al., 2022 ) , aiming to maximize rewards for human-preferred responses while minimizing rewards for dispreferred responses. Although both methods have shown significant success in aligning LLMs to human preferences, they operate under the assumption that preferences are universal, treating conflicting preferences as noise rather than meaningful variations to model ( Cui et al., 2023 ) . Recent efforts have focused on making preference alignment methods more robust to conflicting preferences. For instance, there is work on

Read the complete text