Optimizing User Profiles via Contextual Bandits for Retrieval-Augmented LLM Personalization

Linfeng Du, Ye Yuan, Zichen Zhao, Fuyuan Lyu, Emiliano Penaloza, Xiuying Chen, Zipeng Sun, Jikun Kang, Laurent Charlin, Xue Liu, Haolun Wu · 2026

Read Optimizing User Profiles via Contextual Bandits for Retrieval-Augmented LLM Personalization by Linfeng Du, Ye Yuan, Zichen Zhao, Fuyuan Lyu, Emiliano Penaloza, Xiuying Chen, Zipeng Sun, Jikun Kang, Laurent Charlin, Xue Liu, Haolun Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12078.txt.

Introduction

Linfeng Du, Ye Yuan, Zichen Zhao, and a broad team tune user profiles for personal replies from large models. Retrieval can adapt a model to a user with no retraining. Past tools pick past records by sense match alone.

The team use contextual bandits to pick profile records. A bandit learns which picks pay off from past results. That can beat a fixed match rule as user needs shift. A better profile pick means replies that fit the user with less noise. Bandit picks learn from each reply the user rates.

Catalog record

CREATORLinfeng Du, Ye Yuan, Zichen Zhao, Fuyuan Lyu, Emiliano Penaloza, Xiuying Chen, Zipeng Sun, Jikun Kang, Laurent Charlin, Xue Liu, Haolun Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-12078

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimizing User Profiles via Contextual Bandits for Retrieval-Augmented LLM Personalization Linfeng Du, Ye Yuan, Zichen Zhao, Fuyuan Lyu, Emiliano Penaloza, Xiuying Chen, Zipeng Sun, Jikun Kang, Laurent Charlin, Xue Liu, Haolun Wu Abstract Large Language Models (LLMs) excel at general-purpose tasks, yet adapting their responses to individual users remains challenging. Retrieval augmentation provides a lightweight alternative to fine-tuning by conditioning LLMs on user history records, and existing approaches typically select these records based on semantic relevance. We argue that relevance serves as an unreliable proxy for utility: a record may be semantically similar to a query yet fail to improve generation quality or even degrade it due to redundancy or conflicting information. To bridge this gap, we propose PURPLE , a contextual bandit framework that o P timizes U se R P rofiles for L lm p E rsonalization. In contrast to a greedy selection of the most relevant records, PURPLE treats profile construction as a set generation process and utilizes a Plackett-Luce ranking model to capture complex inter-record dependencies. By training with dense feedback provided by the likelihood of the reference response, our method aligns retrieval directly with generation quality. Extensive experiments on nine personalization tasks demonstrate that PURPLE consistently outperforms strong heuristic and retrieval-augmented baselines in both effectiveness and efficiency, establishing a principled and scalable solution for optimizing user profiles 1 1 1 Our code is available here . . § § footnotetext: Corresponding author. Figure 1: An illustration of the discrepancy between relevance and utility for a user’s past interaction records given a query. The thought bubbles illustrate the potential movie genres that satisfy the underlying user intent for each query. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable success in various natural language processing tasks. As these models are increasingly applied to personalized applications, tailoring responses to individuals based on their own preferences has become a crucial challenge. Existing approaches for personalizing LLMs, such as Parameter-Efficient Fine-Tuning (PEFT) ( 10 ) and Reinforcement Learning from Human Feedback (RLHF) ( 28 ) , generally require modifying model parameters. These approaches incur high computational costs, demand frequent updates, and are impractical for real-time personalization at scale, especially when the LLM is not fully open-sourced. Moreover, continually fine-tuning models for different individuals would complicate safety evaluation and deployment, since each personalized variant would require separate testing. In this paper, we study a lightweight approach to LLM personalization via retrieval augmentation ( 46 ) , in which user profiles are constructed by retrieving and injecting past user records into the prompt. Prior work shows that such profiles can effectively steer LLM outputs toward individual preferences ( 36 ; 12 ) . Compared to parameter-updating methods, retrieval-augmented personalization is lightweight, transparent, and readily deployable, as users can directly inspect and modify the records guiding generation. However, naively including the full user history introduces redundancy and noise, and may exceed the model’s context window, while aggressive pruning risks discarding critical personalization signals. This tension raises a central challenge: which user records should be selected to construct a user profile that maximizes personalization utility , i.e., downstream task performance when conditioned on by the LLM? Existing strategies for building user profiles often rely on heuristics, such as selecting records with the highest relevance , defined as lexical or semantic similarity to the query ( 13 ) . However, we argue that relevance is not a reliable proxy for utility of personalization, which measures the extent to which a record helps the LLM generate an appropriate personalized response. Relevance-based heuristics conflate surface-level contextual similarity with the underlying preference signals that drive personalization, prioritizing records that look similar to the query rather than those that support the user’s current intent. To illustrate this dichotomy, consider Figure 1 , where the user seeks “relaxing movies for Friday night” . Standard relevance heuristics prioritize record h_{1} due to the high lexical overlap of “Friday night” , ignoring the latent preference for “suspense” which contradicts the current need for relaxation. Conversely, record h_{2} offers high utility by capturing the semantic intent to “unwind” , despite lacking surface-level keyword matches. This highlights that relevance scores prioritize how similar a record looks to the query, whereas utility depends on whether it supports the underlying user intent. Beyond evaluating individual records, constructing an effective user profile requires set-aware reasoning rather than independent selection ( 1 ; 29 ) . Even if the utility of each record could be estimated accurately in isolation, greedy top- k selection remains suboptimal because record utilities do not compose additively. Instead, the value of a record is contingent on the context provided by others, as irrelevant or conflicting information within the retrieved set can significantly degrade downstream generation ( 22 ; 47 ) . Revisiting Figure 1 , while h_{2} and h_{3} individually appear useful by reflecting a shared preference for lighthearted content, their combination is suboptimal. Jointly prompting with {h_{2},h_{3}} introduces conflicting constraints by combining “comedy” preferences with “young kids” requirements, thereby diluting generation quality relative to using h_{2} alone. This confirms that personalization utility is non-monotonic. Simply accumulating useful records may introduce interference and actively degrade performance. The limitations highlighted above call for a re-ranking mechanism that is directly optimized for downstream generation performance and is explicitly aware of interactions among records. Existing approaches fall short of these requirements: heuristic retrieval relies on static proxies like similarity, while recent listwise rerankers, though capable of modeling dependencies, remain constrained by relevance-oriented supervision. To bridge this gap, we propose PURPLE , a framework that o P timizes U se R P rofiles for L lm p E rsonalization under a contextual bandit setting ( 18 ) . In its formulation, the context consists of both the current query and the user’s past records. The selection policy is guided by a reward function reflecting downstream personalized generation performance. PURPLE outputs a propensity score for each user record, which is passed through a Plackett-Luce ranking model to produce the final user profile. This formulation enables the model to capture interactions between records and adaptively select those that are most beneficial. We train PURPLE end-to-end using the policy gradient method ( 41 ) . Our main contributions are as follows: • We introduce PURPLE, a framework that casts retrieval-augmented LLM personalization as a contextual bandit problem, adaptively optimizing user profiles beyond static heuristics. • We show through extensive experiments on nine personalization tasks, covering classification, regression, and short- and long-text generation, that PURPLE consistently outperforms strong baselines in both effectiveness and efficiency. • We perform comprehensive ablation studies on PURPLE’s key design choices and further validate PURPLE’s effectiveness through extended analysis and human evaluation. 2 Related work LLMs for Personalization. LLMs demonstrate strong performance across domains ( 27 ) , yet their outputs often diverge from user expectations because pre-training captures general rather than individual needs. Reinforcement Learning from Human Feedback (RLHF) ( 28 ) and Parameter-Efficient Fine-Tuning (PEFT) (e.g., LoRA ( 10 ) ) can align models with user preferences, but both require model fine-tuning and are impractical for end users who lack access to resources. A complementary direction personalizes LLMs through user profiles ( 36 ) , built from prior user interactions or external signals. Leveraging

Read the complete text