From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning

Ranxu Zhang, Zeyang Li, Jiacheng Huang, Rui Zhang, Xiaozhou Xu, Zhe Sun, Yanyong Zhang, Chao Wang · 2026

Read From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning by Ranxu Zhang, Zeyang Li, Jiacheng Huang, Rui Zhang, Xiaozhou Xu, Zhe Sun, Yanyong Zhang, Chao Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23382.txt.

Introduction

Zhang and colleagues build a personalized agentic reinforcement learning framework for settings where one query deserves different plans and tool choices for different users. Generic rewards miss varied tastes, observed behavior mixes in conformity, and flat memory cannot fetch skills by preference. Their PARPO method splits generic task quality from personal preference rewards and anchors learning per user, joined by a two-stage preference-aware reward model and a skill graph memory. Tests on ETAPP, ETAPP-Hard, and SJAgent beat strong memory and learning baselines.

Catalog record

CREATORRanxu Zhang, Zeyang Li, Jiacheng Huang, Rui Zhang, Xiaozhou Xu, Zhe Sun, Yanyong Zhang, Chao Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103841
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23382

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning Ranxu Zhang, Zeyang Li, Jiacheng Huang, Rui Zhang, Xiaozhou Xu, Zhe Sun, Yanyong Zhang, Chao Wang Abstract Agentic reinforcement learning (Agentic RL) has achieved strong progress in tasks with clear success signals. However, many real-world agent applications require user-conditioned behavior: the same query may call for different planning strategies and tool-use decisions across users. This setting raises key challenges: generic rewards cannot capture heterogeneous user preferences, observed behaviors are entangled with conformity effects, and flat memories cannot support personalized skill retrieval. To this end, we propose a unified personalized Agentic RL framework that embeds personalization into training-time optimization. At its core is Personalized Anchor Reward-Decoupled Policy Optimization ( PARPO ), which decouples generic task-quality rewards from personalized preference rewards and uses user-specific anchors to stabilize learning under heterogeneous reward scales. We further introduce a two-stage preference-disentangled reward model and Preference-Aligned Skill Evolution Graph Memory ( PSGM ) for personalized supervision and preference-aligned skill retrieval. Together, they form a closed loop of preference identification, policy optimization, and structured skill accumulation. Experiments on ETAPP, ETAPP-Hard, and SJAgent show that our framework consistently outperforms strong memory and RL baselines. Code and data are included in the supplementary materials. 1 Introduction Large language model (LLM)-based agentic reinforcement learning (Agentic RL) has emerged as a powerful paradigm for optimizing LLM agents, achieving strong performance in code generation ( 6 ; 11 ; 36 ) , web navigation ( 18 ; 22 ) , tool use ( 23 ; 4 ; 33 ) , and long-horizon planning ( 30 ; 21 ; 32 ) . However, most of this progress has been made in verifiable settings ( 23 ; 9 ; 25 ) , where policy optimization can rely on a unique ground-truth answer. In contrast, in many real-world agent applications—such as e-commerce assistance, travel planning, and daily scheduling ( 26 ; 24 ; 19 ; 34 ; 17 ) —this verifiability breaks down because optimal behavior is user-dependent: the same query may admit multiple plausible trajectories, with the preferred one determined by the user’s preferences, habits, and constraints. Recent work has begun to extend LLM optimization beyond strictly verifiable tasks. Non-verifiable or open-ended optimization methods use LLM-based evaluation ( 37 ; 2 ; 16 ; 43 ) , rubric-based rewards ( 7 ; 15 ) , and other reward construction schemes ( 20 ; 38 ; 29 ; 35 ) to provide supervision when exact answers are unavailable. However, these methods typically optimize generic objectives, such as overall quality, helpfulness, or rubric satisfaction, rather than user-conditioned preferences. In parallel, personalized agent methods incorporate user profiles, prompting strategies, agent designs, and memory retrieval over historical interactions ( 41 ; 12 ; 1 ; 31 ; 17 ; 27 ) . While effective for improving user alignment, they largely personalize behavior at inference time and do not directly optimize policies for user-contingent trajectories. Therefore, existing methods lack a native training-time optimization framework for personalized agent behavior. Personalization fundamentally changes the optimization target of Agentic RL. For the same query, users may prefer different planning strategies and tool-use decisions; for example, “plan a one-day trip in Tokyo” may call for a museum-centered route for one user but an anime-themed route for another. Therefore, the agent must move beyond learning a single average-optimal policy and instead learn user-contingent tool-use trajectories. As illustrated in Figure 1 , this setting introduces three core challenges. (C1) Personalized reward ambiguity : generic rewards mainly capture correctness, task completion, or overall helpfulness, but cannot express how a specific user evaluates the same trajectory, nor can they handle heterogeneous reward scales across users. (C2) Personalized preference disentanglement : observed user behaviors are often shaped by both intrinsic interests and external conformity or contextual effects, making individualized preference signals noisy and difficult to identify accurately. (C3) User-aware memory and skill organization : existing agent memories are often flat and query-centric, and thus cannot explicitly model or retrieve the structured relations among users, intents, skills, tools, scenarios, and trajectories. To bridge this gap, we propose a unified personalized Agentic RL framework that embeds personalization into the training-time optimization loop. At its core is Personalized Anchor Reward-Decoupled Policy Optimization ( PARPO ), which decouples generic task-quality rewards from personalized preference rewards and uses user-specific anchors to stabilize learning under heterogeneous reward scales. PARPO preserves general task competence while enabling user-contingent policy improvement. To provide cleaner personalized supervision, we develop a two-stage preference-disentangled reward model that separates intrinsic interests from conformity and contextual effects. To support personalized rollout context, we introduce Preference-Aligned Skill Evolution Graph Memory ( PSGM ), an evolving heterogeneous graph memory that organizes users, skills, tools, scenarios, and trajectories for preference-aligned skill retrieval. Together, these components form a closed loop of preference identification, personalized policy optimization, and structured skill accumulation. Our contributions are summarized as follows: • We formulate personalized Agentic RL for user-conditioned agent tasks, where optimal behavior depends on individual preferences. • We propose PARPO , an anchor-stabilized and reward-decoupled policy optimization method for learning personalized policies under heterogeneous user reward scales. • We introduce a preference-disentangled reward model and PSGM to provide reliable personalized supervision and preference-aligned skill retrieval. • We evaluate our framework on ETAPP ( 8 ) , ETAPP-Hard, and SJAgent (a real-world industrial agent training scenario from a large Chinese e-commerce platform), showing gains in personalization and procedural quality while maintaining factual and logical quality. Figure 1 : Personalization in Agentic RL changes the notion of optimal behavior: the same query may require different plans for different users. 2 Related Work Agentic RL in Verifiable Settings. RL has proven effective for improving LLM agents on verifiable tasks, including Retool, ToolRL, AutoWebGLM, and Search-R1 ( 4 ; 23 ; 9 ; 10 ) . GRPO-style methods such as GRPO, DAPO, GSPO, GiGPO, and GDPO improve scalability, sequence-level optimization, and multi-reward stability ( 25 ; 39 ; 42 ; 5 ; 14 ) . However, these advances remain centered on verifiable settings with correctness or task-success signals, rather than user-specific alignment ( 9 ; 33 ; 5 ) . Non-verifiable and Open-ended Optimization. RL has also been extended beyond verifiable tasks using LLM-based evaluation, rubric supervision, and learned rewards, e.g., OpenRubrics and Rubrics as Rewards ( 15 ; 7 ) . While these methods support open-ended outputs, they still optimize generic objectives, such as quality, rubric satisfaction, rather than personalized behavior. Personalization, Preference, and Memory. Prior work explores personalization through profiles, memory, and personalized agents, including PersonaAgent, O-Mem, Preference-Aware Memory Update, and Learning Personalized Agents from Human Feedback ( 41 ; 31 ; 28 ; 12 ) . CoPD studies true-interest/conformity entanglement in user behavior ( 44 ) , while memory- and skill-based agents retrieve skills and user context ( 40 ; 33 ; 45 ; 13 ) . Most focus on inference-time personalization or omit personalized policy in user-conditioned environments. Our work instead unifies training-time user-conditioned policy optimization, true-preference reward modeling, and personalized skill retrieval. A more detailed discussion of relevant prior work is provided in Appendix A . 3 Problem Definition As discussed in the Introduction, personalization is important in real-world scenarios such as e-commerce and daily planning, since

Read the complete text