Training LLMs with Reinforcement Learning for Intent-Aware Personalized Question Answering
Maryam Amirizaniani, Benjamin Charles Germain Lee, Jevin West, Nicholas Weber · 2026
Read Training LLMs with Reinforcement Learning for Intent-Aware Personalized Question Answering by Maryam Amirizaniani, Benjamin Charles Germain Lee, Jevin West, Nicholas Weber (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12645.txt.
Introduction
Amirizaniani and three colleagues train language models for personalised question answering grounded in user intent. Intent means the implicit why behind a query, beyond its explicit wording, yet existing personalisation leans on multi-turn context or rich user profiles and never models intent explicitly during reasoning. That limits single-turn settings, where a latent goal is guessed from minimal input. Their IAP framework uses reinforcement learning to infer implicit intent from one question and weave it into thinking steps with a tag-based schema, producing intent-grounded personalised answers.
Catalog record
| CREATOR | Maryam Amirizaniani, Benjamin Charles Germain Lee, Jevin West, Nicholas Weber |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40355 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12645
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training LLMs with Reinforcement Learning for Intent-Aware Personalized Question Answering Maryam Amirizaniani, Benjamin Charles Germain Lee, Jevin West, Nicholas Weber Abstract Effective personalized question answering (PQA) in language models requires grounding responses in the user’s underlying intent, where intent refers to the implicit “why” behind a query beyond its explicit wording. However, existing approaches to intent-aware personalization rely on multi-turn conversational context or rich user profiles, and do not explicitly model user intent during reasoning process. This limits their effectiveness in single-turn settings, where the user’s latent goal must be inferred from minimal input and integrated into the thinking and reasoning process. To bridge this gap, we propose IAP ( I ntent A ware P ersonalization), a reinforcement learning framework that trains models to infer implicit user intent directly from a single-turn question and incorporate it into thinking steps through a tag-based schema for generating personalized, intent-grounded answers. By optimizing intent-aware answer trajectories under a personalized reward function, IAP reinforces generation paths that make implicit user intent explicit and produce responses that better align with the user’s underlying goal. Through experiments on the LaMP-QA benchmark across six models, IAP consistently outperforms all baselines, achieving an average macro-score gain of around 7.5% over the strongest competitor, demonstrating that modeling implicit user intent within the training objective is a promising direction for PQA. 1 Introduction Personalized question answering (PQA) aims to generate user-tailored answers and can be further improved by aligning responses with the user’s underlying intent ( 39 ; 44 ) . In this context, intent refers to the underlying goal or purpose behind a user’s query (e.g., seeking emotional support or practical guidance) ( 13 ) . Since intent is often implicit, accurately inferring it is critical for generating answers that are not only correct but also aligned with what the user actually needs ( 29 ; 6 ) . For example, consider the question, “Should I change my job?” Two users can ask the same question with different implicit intents: one may be seeking decision-analytic support (e.g., comparing options and trade-offs), while another may want help interpreting uncertainty and diagnosing their source of dissatisfaction (e.g., burnout, values mismatch, or limited growth). If a system responds to both users with a generic pros and cons list, the answer may still be coherent yet misaligned because it is too abstract, too clinical, or too vague for the user’s intended outcome. To understand user intent, prior work has explored intent modeling from several angles, ranging from classic closed-domain approaches that assign user utterances to predefined intent categories ( 18 ; 12 ; 29 ) to more recent Large Language Model (LLM)-based methods that use instruction tuning to extract intents ( 13 ; 44 ; 3 ; 21 ; 20 ; 9 ) . Despite this progress, three key limitations remain: Figure 1: An example of IAP. (1) current approaches primarily infer intents through prompting, without incorporating intent into the model’s reasoning and training processes before personalized answer generation ( 17 ; 11 ; 3 ; 9 ) ; (2) most depend on multi-turn dialogue history or detailed user profiles, limiting their applicability in single-turn question answering (QA) settings where multiple user intents must be inferred from a single question alone ( 9 ; 18 ) ; and (3) existing methods optimize LLMs for personalized answer generation using only intent-aware signals, while neglecting the role of intent-free responses ( 45 ; 9 ) . Building on these gaps, we investigate the following research questions: RQ1: How can we train a model to infer a user’s latent intents and use it to generate personalized answers in single-turn question answering? RQ2: To what extent is leveraging contrastive intent-aware and intent-free response signals in training LLMs effective for generating user-intent-grounded answers? To answer these research questions, we propose IAP ( I ntent A ware P ersonalization), a Reinforcement Learning (RL) framework that trains LLMs to infer a user’s latent intents from a single-turn input question and incorporate it into their thinking process before generating personalized, intent-grounded answers. IAP instantiates this learning objective using Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) ( 38 ) , which optimizes intent-aware answer trajectories under a personalized reward function that evaluates how well a response aligns with the user’s intent and preferences. The personalized reward further incorporates an auxiliary intent-free response signal that penalizes outputs close to non-contextualized answers, thereby discouraging generic generation and encouraging responses that are more sensitive to user intent and better reflect the user’s preferences. Following ( 21 ; 20 ) , IAP incorporates inferred intent into the model’s thinking and reasoning process through a tag-based format, making the user’s latent goal explicit before answer generation. Specifically, the process is structured into a <think> tag for internal reasoning, an <intent> tag for explicitly capturing the user’s latent intent, and an <answer> tag for generating the final personalized response, as illustrated in Figure 1 . To evaluate IAP, we use the LaMP-QA dataset ( 28 ) , which consists of long-form social questions, and study six different LLMs to assess whether intent-aware reasoning improves PAQ. Experimental results show that IAP consistently outperforms all baselines, achieving an average macro-score gain of approximately 7.5% over the strongest competitor across all evaluated LLMs. The main contributions of this paper are summarized as follows: (1) We introduce IAP, an RL-based framework that trains LLMs to infer implicit user intents from a single-turn question and incorporate them into the reasoning process via a tag-based schema to generate personalized, intent-grounded answers 1 1 1 The code is available at: https://github.com/maryamamiri114/IAP . (2) We leverage a contrastive reward function that explicitly contrasts intent-aware and intent-free responses during training to discourage generic generation and push the model toward more intent-sensitive outputs, and (3) We evaluate IAP on the LaMP-QA across six LLMs, demonstrating consistent improvements of approximately 7.5% on average over the strongest baseline across all models. Figure 2: IAP Framework for intent-aware personalization question answering. 2 Related Work Intent understanding is among the most effective strategies for personalized question answering ( 17 ; 27 ; 26 ; 35 ; 2 ; 5 ) . Existing work, however, largely treats intent as classification over predefined label taxonomies, as reflected in benchmarks like BANK77 ( 12 ) , CLINC150 ( 18 ) , and SGD ( 25 ) . These fixed-category formulations struggle in open-domain settings where intents are ambiguous, compositional, and evolving ( 29 ) As LLMs have become increasingly prevalent, researchers have begun leveraging them to extract user intent in open-domain settings ( 33 ; 7 ; 1 ) , including reformulating intent detection as a language generation problem to enable inference from natural-language descriptions ( 39 ) . Incorporating conversational history and situational cues has further been shown to resolve ambiguity in underspecified queries, underscoring the value of context-aware intent modeling ( 13 ; 46 ; 22 ; 32 ; 10 ; 40 ; 41 ) . Building on this, CA-UIA-QG generates follow-up questions aligned with shifting user intent ( 14 ) , and Chain-of-Intent integrates sequence models with LLM generation to produce intent-guided dialogues ( 37 ) . Beyond text, IntentQA further demonstrates that intent reasoning in video settings requires contextual and