FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in LLMs Elicits Effective Personalization to Real Users

Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, Chelsea Finn · 2025

Read FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in LLMs Elicits Effective Personalization to Real Users by Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, Chelsea Finn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19312.txt.

Introduction

Anikait Singh and co-authors present FSPO, a method to personalize language models to real users from a few examples, in a 2025 paper. Personalization matters for assistants and content curation. Their algorithm reframes reward modeling as a meta-learning problem.

Under FSPO, a model learns to infer a user's personal reward function quickly from a few labeled preferences. A rationalization step, which restates the user description, improves reward modeling and instruction following. Since real preference data are hard to collect at scale, they build over a million synthetic preferences with public models. The method is designed to transfer from synthetic to real users.

Catalog record

CREATORAnikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, Chelsea Finn
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-19312

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in LLMs Elicits Effective Personalization to Real Users Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, Chelsea Finn Abstract Effective personalization of LLMs is critical for a broad range of user-interfacing applications such as virtual assistants and content curation. Inspired by the strong in-context learning capabilities of LLMs, we propose Few-Shot Preference Optimization (FSPO), which reframes reward modeling as a meta-learning problem. Under this framework, an LLM learns to quickly adapt to a user via a few labeled preferences from that user, constructing a personalized reward function for them. Additionally, since real-world preference data is scarce and challenging to collect at scale, we propose careful design choices to construct synthetic preference datasets for personalization, generating over 1M synthetic personalized preferences using publicly available LLMs. In particular, to successfully transfer from synthetic data to real users, we find it crucial for the data to exhibit both high diversity and coherent, self-consistent structure. We evaluate FSPO on personalized open-ended generation for up to 1,500 synthetic users across across three domains: movie reviews, pedagogical adaptation based on educational background, and general question answering, along with a controlled human study. Overall, FSPO achieves an 87% Alpaca Eval winrate on average in generating responses that are personalized to synthetic users and a 72% winrate with real human users in open-ended question answering. \reportnumber “Every story I create, creates me. I write to create myself.” —Octavia E. Butler 1 Introduction As language models increasingly interact with a diverse user base, it becomes important for models to generate responses that align with individual user preferences. People exhibit a wide range of preferences and beliefs shaped by their cultural background, personal experience, and individual values. These diverse preferences may be reflected through human-annotated preference datasets; yet, current preferences optimization techniques like reinforcement learning from human feedback (RLHF) largely focus on optimizing a single model based on preferences aggregated over the entire population. This approach may neglect minority viewpoints, embed systematic biases into the model, and ultimately lead to worse performance compared to personalized models. Can we create language models that can adaptively align with personal preferences of the users and not the aggregated preferences of all users? Addressing this challenge requires a shift from modeling a singular aggregate reward function to modeling a distribution of reward functions ( Sorensen et al. 2024 ; Jang et al. 2023 ) that capture the diversity of human preferences. By doing so, we can enable personalization in language models, allowing them to generate a wide range of responses tailored to individual subpopulations. This approach not only enhances user satisfaction but also promotes inclusivity by acknowledging and respecting the varied perspectives that exist within any user base. However, how can this be effectively done for open-ended question answering and transfer to real users? Figure 1 : Overview of FSPO. N previously collected preferences are fed into the LLM along with the current query, allowing the LLM to personalize its response to the query using the past preferences. In this paper, we introduce Few-Shot Preference Optimization (FSPO), a novel framework designed to model diverse subpopulations in preference datasets to elicit personalization in language models for open-ended question answering. At a high level, FSPO leverages in-context learning to adapt to new subpopulations. This adaptability is crucial for practical applications, where user preferences can be dynamic and multifaceted. Inspired by past work on black-box meta-learning for language modeling ( Chen et al. 2022 ; Min et al. 2022 ; Yu et al. 2024 ) , we fine-tune the model with a meta-learning objective, using preference-learning objectives such as IPO ( Gheshlaghi Azar et al. 2023 ) . We additionally propose user description chain-of-thought (COT), allowing the model to leverage additional inference-compute for better reward modeling and the model’s instruction following capabilities for better response generation. However, to learn a model that effectively personalizes to real people, we need to collect a diverse preference dataset spanning diverse users. One natural approach to do this is to curate data from humans, but this curation is difficult and time-consuming. In contrast, in this work, we propose instantiating this dataset synthetically, and present careful design decisions to generate a dataset that is diverse and structured, following task construction considerations from the meta-learning literature ( Hsu et al. 2019 ; Yin et al. 2019 ) . To evaluate the efficacy of our approach, we construct a set of three semi-realistic domains to study personalization: (1) Reviews , studying the generation ability of models for reviews of movies, TV shows, and books that are consistent with a user’s writing style, (2) Explain Like I’m X (ELIX) : studying the generation ability of models for responses that are consistent with a user’s education level, and (3) Roleplay : studying the generation ability of models for responses that are consistent with a user’s description, with effective transferability to a real human-study. Here we find that FSPO outperforms an unpersonalized model on average by 87%. We additionally perform a controlled human study showcasing a winrate of 72% of FSPO over unpersonalized models. By addressing limitations of existing reward modeling techniques, our work paves the way for more inclusive and personalized LLMs. We believe that FSPO represents a significant step toward models that better serve the needs of all users, respecting the rich diversity of human preferences. 2 Related Work Personalized learning of preferences. Prior research has explored personalization through various methods. One approach is distributional alignment, which focuses on matching model outputs to broad target distributions rather than tailoring them to individual user preferences. For example, some prior work have concentrated on aligning model-generated distributions with desired statistical properties ( Siththaranjan et al. 2024 ; Meister et al. 2024 ; Melnyk et al. 2024 ) , yet they do not explicitly optimize for individual preference adaptation. Another strategy involves explicitly modeling a distribution of rewards ( Lee et al. 2024 ; Poddar et al. 2024 ) . However, these methods suffer from sample inefficiency during both training and inference ( Rafailov et al. 2023 ; Gheshlaghi Azar et al. 2023 ) . Additionally, these approaches have limited evaluations: Lee et al. 2024 focuses solely on reward modeling, while Poddar et al. 2024 tests with a very limited number of artificial users (e.g helpfulness user and honest user). Other works have investigated personalization in multiple-choice questions, such as GPO ( Zhao et al. 2024 ) . Although effective in structured survey settings, these methods have not been validated for open-ended personalization tasks. Similarly, Shaikh et al. 2024 explores personalization via explicit human corrections, but relying on such corrections is expensive and often impractical to scale. Finally, several datasets exist for personalization, such as Prism ( Kirk et al. 2024 ) and Persona Bench ( Castricato et al. 2024 ) . Neither of these datasets demonstrate that policies trained on these benchmarks lead to effective personalization. Unlike these prior works which study personalization based off of human values and controversial questions, we instead study more general questions that a user may ask. Algorithms for preference learning. LLMs are typically fine-tuned via supervised next-token prediction on high-quality responses and later refined with human preference

Read the complete text