Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas

Nishant Balepur, Vishakh Padmakumar, Fumeng Yang, Shi Feng, Rachel Rudinger, Jordan Boyd-Graber · 2025

Read Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas by Nishant Balepur, Vishakh Padmakumar, Fumeng Yang, Shi Feng, Rachel Rudinger, Jordan Boyd-Graber (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11549.txt.

Introduction

Nishant Balepur and five co-authors ask whose tastes preference tuning should serve. Language models learn from pairs of responses, but the data rarely say why a user liked one reply over another. The team uses abductive reasoning to infer user personas from preference data. One step infers personas for users who picked chosen or rejected outputs, and a second step trains models to tailor replies to those personas.

Models inferred personas that plausibly explained both kinds of choices. Training with the inferred personas boosted personalization and carried over to user-written personas. Personas behind rejected replies made harder tests, and tailoring helped users with uncommon tastes most.

Catalog record

CREATORNishant Balepur, Vishakh Padmakumar, Fumeng Yang, Shi Feng, Rachel Rudinger, Jordan Boyd-Graber
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-11549

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas Nishant Balepur, Vishakh Padmakumar, Fumeng Yang, Shi Feng, Rachel Rudinger, Jordan Boyd-Graber Abstract LLMs are aligned to follow input instructions by learning which of two responses users prefer for a prompt. However, such preference data do not convey why users prefer responses that are chosen or rejected, so LLMs trained on these datasets cannot tailor responses to varied user needs. To surface these parameters of personalization, we apply abductive reasoning to preference data, inferring needs and interests of users, i.e., personas, that may prefer either response. We test this idea in two steps: Persona Inference (PI) —abductively inferring personas of users who prefer chosen or rejected outputs—and Persona Tailoring (PT) —training models to tailor outputs to personas from PI . We show: 1) LLMs infer personas accurately explaining why different users may prefer both chosen or rejected outputs; 2) Training on preference data augmented with PI personas via PT boosts personalization and generalizes to supporting user-written personas; and 3) Rejected response personas form harder personalization evaluations, showing PT better aids users with uncommon preferences versus typical alignment methods. We argue for an abductive view of preferences for personalization, asking not only which response is better but when, why, and for whom. 1 1 1 Code and datasets are available at: https://github.com/Pinafore/alignment-personalization 1 Every Preference Happens for a Reason Current methods for aligning large language models (LLMs) predominantly use preference data Ji et al. (2023) , created by finding: for a given prompt, which of two outputs do users prefer? LLMs are then trained on this data via preference tuning methods like direct preference optimization ( Rafailov et al., 2024 , DPO) , learning to give outputs like the majority chosen response and unlike the rejected one. This improves LLM outputs in dialogue Köpf et al. (2024) , question answering ( Fan et al., 2019 , qa ) , and summarization Völske et al. (2017) . While preference datasets are valuable, they assume chosen responses are universally better, failing to consider why users prefer responses Joshi et al. (2025) . In reality, some users genuinely prefer rejected outputs, even if their reasons are less common. In Figure 2 (left), most users prefer the chosen output for taking brownies to a cake sale. While users valuing simplicity may prefer this direct response, practical users may favor the rejected one, as it also gives packaging logistics (Figure 2 , middle). Since users prefer responses for varied reasons Kirk et al. (2024) , models trained on preference data should personalize outputs to meet these specific, individual needs Salemi et al. (2024) . A common, interpretable method for personalization involves users specifying their needs and interests as an extra input (e.g., “I like short answers”)—forming a system prompt persona to tailor model responses Zhang et al. (2024c) . However, current preference dataset formats lack personas explaining why users prefer responses, overlooking signals for training personalized models Lee et al. (2024) . Figure 1: Training methods with typical preference datasets like DPO cannot fully cater to a user’s specified personas . To overcome this, we train models on preference data augmented with LLM-inferred personas, which we call persona tailoring . Figure 2: Overview of this paper. Preference data has a prompt, chosen response, and rejected response (left). Most users prefer the chosen response, but there are valid reasons and personas of users that may prefer either response, which we uncover via abductive reasoning in Persona Inference ( PI , middle). We then study Persona Tailoring ( PT ) to use the personas for data augmentation in few-shot prompts, fine-tuning, and direct preference optimization—enhancing personalization (right). The only way for models trained on existing preference dataset formats to use a persona is by adding it to the inference prompt, hoping the model treats it as an extra instruction to follow Deb et al. (2022) . However, standard preference tuning methods like DPO cannot naturally adapt to personas in this way. In Fig 1 , DPO tells a sober user to hire a bartender (left), while a user desiring a small list gets 10 ideas (right); such responses are unhelpful, inappropriate, and do not tailor to the users’ specified personas. As models’ poor personalization stems from a lack of training on personas, we propose abductive reasoning Peirce (1974) to augment preference training data with LLM-inferred personas. Abduction infers hidden contexts that explain a given outcome Zhao et al. (2024) . Similarly, we adopt this reasoning to infer (i.e., adduce) hidden needs, interests, and traits of users, i.e., personas Tseng et al. (2024) , that explain why chosen and rejected outputs may be preferred. If LLMs surface valid personas of users who may prefer chosen or rejected responses, we can attach them to preference datasets and train models that tailor to user-specified needs, enhancing model personalization (Figure 1 , blue ). We segment this idea into a two-task sequence: 1) Persona Inference ( PI , section 2 ): adducing personas for preference data responses (Figure 2 , middle); and 2) Persona Tailoring ( PT , section 4 ): producing outputs tailored to personas from PI (Figure 2 , right). We first test LLMs in PI on dialogue Bai et al. (2022a) , qa Ji et al. (2024) , and education Balepur et al. (2024c) preference data. LLM personas accurately convey different users who could prefer chosen or rejected outputs; LLaMA-405B has 91% accuracy, judged by GPT-4o with 90% human agreement ( section 3.1 ). Further, chosen and rejected response personas are often judged as tied in quality ( section 3.2 ), and humans rate rejected ones as plausible but applying to fewer users ( section 3.3 ). Thus, users may prefer rejected outputs for uncommon but valid reasons. Personas are also a useful content analysis tool to find differences in chosen and rejected responses; in BeaverTails Ji et al. (2024) , chosen response personas describe users who are “meticulous”, while rejected ones describe “direct” users, showing these labelers may prefer verbosity Zheng et al. (2024a) . As PI yields accurate personas, we augment preference data with PI personas using LLaMA-405B. We then train LLaMA-8B on this new data for the reverse task of PT : using prompts and inferred personas as inputs to give tailored responses (Figure 2 , right). We test three strategies: prompting ( Brown et al., 2020 , \textsc{PT}_{\textsc{fs}} ) , fine-tuning ( Chung et al., 2024 , \textsc{PT}_{\textsc{sft}} ) , and DPO ( Rafailov et al., 2024 , \textsc{PT}_{\textsc{dpo}} ) . Each generation strategy largely boosts personalization when using PT ( section 5.1 ), with \textsc{PT}_{\textsc{dpo}} being the strongest ( section 5.2 ). Further, DPO fine-tuned on preference datasets without personas cannot always tailor to personas during inference; notably, \textsc{PT}_{\textsc{dpo}} is judged as much stronger than DPO on uncommon but still valid needs in the personas from rejected responses ( 66% average improvement in personalization ), showing rejected responses can form valuable, harder evaluations for personalization ( section 5.3 ). Finally, eight users author 144 diverse personas and rate \textsc{PT}_{\textsc{dpo}} and DPO responses for these personas ( section 5.4 ). The same users find our \textsc{PT}_{\textsc{dpo}} method personalizes more effectively to

Read the complete text