User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction
Yuren Hao, Shuhaib Mehri, ChengXiang Zhai, Dilek Hakkani-Tür · 2026
Read User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction by Yuren Hao, Shuhaib Mehri, ChengXiang Zhai, Dilek Hakkani-Tür (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20939.txt.
Introduction
Hao and co-authors give chat agents a lasting model of each user's preferences. Most assistants keep no user model, so people must restate their tastes each session. Their VARS framework stores long term and short term vectors for each user in a shared preference space. These vectors bias retrieval from a structured preference memory. Weak feedback scores update the vectors online, with no per user fine tuning. On a multi session benchmark, the agent cuts timeouts and user effort, and its vectors align with real preference overlap.
Catalog record
| CREATOR | Yuren Hao, Shuhaib Mehri, ChengXiang Zhai, Dilek Hakkani-Tür |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75376 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20939
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction Yuren Hao, Shuhaib Mehri, ChengXiang Zhai, Dilek Hakkani-Tür Abstract Large language models are increasingly used as personal assistants, yet most lack a persistent user model, forcing users to repeatedly restate preferences across sessions. We propose Vector-Adapted Retrieval Scoring (VARS), a pipeline-agnostic, frozen-backbone framework that represents each user with long-term and short-term vectors in a shared preference space and uses these vectors to bias retrieval scoring over structured preference memory. The vectors are updated online from weak scalar rewards from users’ feedback, enabling personalization without per-user fine-tuning. We evaluate on MultiSessionCollab , an online multi-session collaboration benchmark with rich user preference profiles, across math and code tasks. Under frozen backbones, the main benefit of user-aware retrieval is improved interaction efficiency rather than large gains in raw task accuracy: our full VARS agent achieves the strongest overall performance, matches a strong Reflection baseline in task success, and reduces timeout rate and user effort. The learned long-term vectors also align with cross-user preference overlap, while short-term vectors capture session-specific adaptation, supporting the interpretability of the dual-vector design. Code, model, and data are available at https://github.com/YurenHao0426/VARS . 1 Introduction LLMs are increasingly used as personal assistants for writing, analysis, and programming ( 3 ) . During these interactions, users exhibit distinct preferences for how they communicate, receive feedback, and structure workflows ( 10 ; 11 ) , yet most systems lack mechanisms to remember and adapt to these preferences across sessions, forcing users to repeatedly restate them and reducing the efficiency of long-term collaboration ( 30 ; 16 ; 18 ) . A growing body of work has explored retaining information across sessions for conversational agents. Most systems store past exchanges in a database and retrieve relevant entries at query time, or produce post-session reflections that are reinserted into future conversations ( 37 ; 20 ; 25 ; 4 ) . Several recent works have also explored using memory to directly improve personalization across sessions ( 18 ; 7 ; 16 ) . The retrieval in these systems typically relies on query similarity alone, without a persistent user state to prioritize which preferences are most relevant to the current context ( 22 ; 31 ) . A central gap remains: existing systems recall past interactions but do not maintain a user representation that improves retrieval of relevant preferences through ongoing interaction. We address this gap with Vector-Adapted Retrieval Scoring ( VARS ), a framework that learns a compact dual-vector user state from weak scalar rewards from users’ feedback and uses it to bias retrieval over structured preference memory. The framework comprises four learned components—a lightweight preference extraction model that converts dialogue into structured memory cards, a preference memory that indexes these cards for retrieval, a dual-vector user state that separates stable cross-session preferences from transient within-session context, and a reward-driven update mechanism that refines the user vectors from weak scalar feedback—layered on top of three frozen backbones (chat LLM, embedding model, reranker). At inference time, the effective user vector biases retrieval scoring so that the most relevant preferences are surfaced for each query. Because adaptation occurs only through compact user-specific vectors, the framework supports continuous personalization without per-user fine-tuning—a form of user-centric memory that organizes stored preferences around the user rather than the task. We evaluate this framework on MultiSessionCollab , an online multi-session benchmark for long-horizon personalization ( 18 ) . This benchmark pairs each system with an LLM-based user simulator whose preference profiles are enforced across sessions, making it possible to measure whether a persistent user representation improves behavior over time. We instantiate the framework on math and code tasks using open-source LLMs and compare against context-only, reflection-based, and retrieval-based baselines. Our results show that the main benefit of VARS is improved interaction efficiency rather than large gains in raw task success. Compared with a strong Reflection baseline, the full VARS system achieves comparable task success while reducing timeout and user effort, indicating that a persistent user representation helps the agent surface relevant preferences with less corrective interaction. More broadly, these findings suggest a distinction between task-centric and user-centric memory objectives: task-centric memory is naturally reflected in whether the task is eventually completed, whereas user-centric memory is also reflected in the cost of collaboration required to reach that outcome. In this setting, organizing memory around the user primarily makes collaboration more efficient, while still yielding modest improvements in task success. The contributions of our work are: • We propose VARS, a frozen-backbone framework that learns a per-user dual vector from weak interaction feedback and adds a learned user-aware bonus to the task-centric reranker, so that retrieval reflects both query relevance and individual user preference. • We evaluate VARS on MultiSessionCollab and show that it improves interaction efficiency—reducing timeout rate and user effort—while matching a strong baseline in task success. • We analyze the learned user vectors and show that the dual-vector design separates stable cross-session preferences from transient within-session context, with long-term vectors aligning with cross-user preference overlap. 2 Related Works LLM personalization. Prior work on LLM personalization includes profile-augmented prompting ( 22 ) , learned user representations ( 11 ) , and retrieval-based profile optimization ( 5 ) . These approaches have established the importance of adapting LLM behavior to individual users, but they typically assume a fixed user profile provided as input rather than one learned and updated from ongoing interaction. In contrast, our method learns a compact user representation online from interaction feedback and uses it to guide retrieval over structured preference memory, without per-user fine-tuning. Long-term memory and multi-session user modeling. A closely related line of work equips LLM agents with long-term memory by storing dialogue history, retrieved notes, or reflection summaries for later use ( 37 ; 20 ; 25 ; 4 ; 31 ; 23 ) . Some systems further separate long-term and short-term state through modular or event-centric architectures ( 16 ; 38 ) . Related benchmarks study long-horizon memory and user modeling across sessions through profile inference, memory probing, or multimodal memory tasks ( 10 ; 11 ; 30 ; 1 ; 24 ) , while 9 highlight the complementary risk of over-personalization. Among these, MultiSessionCollab is closest to our setting because it evaluates downstream collaboration under persistent user preferences rather than question answering over stored history ( 18 ) . Building on this setting, we ask whether a learned persistent user state improves collaboration over time by reducing user effort and corrective interaction, not only by increasing eventual task success. Learning from weak interaction feedback. Our online update mechanism is also related to learning from implicit or bandit-style feedback in ranking and preference elicitation ( 21 ; 8 ; 13 ; 33 ) . 5 optimize profile selection via contextual bandits with a global policy; in contrast, we keep all backbone models frozen and learn a per-user retrieval bias from weak scalar feedback. 3 Method Figure 1: VARS architecture. M_{\text{ext}} extracts preference cards; a dual user vector z_{U,t}^{\mathrm{eff}}=\beta_{L}z_{U}^{(L)}+\beta_{S}z_{U,t}^{(S)} biases reranking; a keyword-based reward \hat{r}_{t} drives REINFORCE updates. Our goal is to learn a persistent representation of the user that improves through interaction and enables the system to selectively surface relevant preferences