Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction
Haoyu Wang, Yuxin Chen, Liang Luo, Buyun Zhang, Ellie Dingqiao Wen, Pan Li · 2026
Read Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction by Haoyu Wang, Yuxin Chen, Liang Luo, Buyun Zhang, Ellie Dingqiao Wen, Pan Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23550.txt.
Introduction
Wang and five co-authors present ITPO, a method for training chat agents across talks. Multi-turn work like tutoring lacks clear rewards at each step, and user replies vary wildly. ITPO spins per-turn rewards out of sparse final outcomes. An implicit process reward model does that conversion. Turn-level signals prove steadier than token-level ones, and a normalizing step calms training further. With PPO, GRPO, or RLOO, ITPO converges better on tutoring, writing, and medical advice tasks. Its turn scores line up with human judgment.
Catalog record
| CREATOR | Haoyu Wang, Yuxin Chen, Liang Luo, Buyun Zhang, Ellie Dingqiao Wen, Pan Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77541 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.23550
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction Haoyu Wang, Yuxin Chen, Liang Luo, Buyun Zhang, Ellie Dingqiao Wen, Pan Li Abstract Multi-turn human-AI collaboration is fundamental to deploying interactive services such as adaptive tutoring, conversational recommendation, and professional consultation. However, optimizing these interactions via reinforcement learning is hindered by the sparsity of verifiable intermediate rewards and the high stochasticity of user responses. To address these challenges, we introduce Implicit Turn-wise Policy Optimization (ITPO). ITPO leverages an implicit process reward model to derive fine-grained, turn-wise process rewards from sparse outcome signals. Unlike volatile token-level rewards, these turn-level signals exhibit superior robustness and may utilize a normalization mechanism to further enhance training stability. We evaluate ITPO across three representative multi-turn collaborative tasks: math tutoring, document writing, and medical recommendation. Empirical results demonstrate that ITPO, when combined with PPO, GRPO, or RLOO, consistently achieves improved convergence than existing baselines. Elaborate trajectory analysis confirms that ITPO infers turn-wise preferences that are semantically aligned with human judgment. Code is publicly available at https://github.com/Graph-COM/ITPO . 1 Introduction Human-AI collaboration is revolutionizing the societal landscape 72 ; 26 , ranging from enhancing personalization in recommendation systems and education 71 ; 67 to the democratization of specialized expertise in healthcare and law 36 ; 28 . Realizing this potential necessitates a paradigm shift in large language models (LLMs): moving beyond reactive instruction following toward multi-turn interactions, where agents must proactively resolve ambiguity 59 , decompose complex goals 58 ; 18 , and iteratively refine their outputs 5 . However, existing LLMs suffer from performance degradation in long-horizon interactions 24 , making the effective alignment for multi-turn User-LLM collaboration via reinforcement learning (RL) an urgent yet open problem. The core challenge lies in the supervision sparsity. As outcome rewards typically materialize until conversation concludes, relying only on these delayed signals often leads to poor sample efficiency and spurious solutions 47 . Synthesizing fine-grained dense process rewards offers a potential solution, while existing approaches such as value estimation 21 ; 41 or process reward models (PRMs) 32 fall short under the multi-turn collaborative setting. On one hand, training value models to estimate process rewards is empirically difficult 63 ; 13 ; 2 , particularly given the high variance of user dynamics that hinders convergence. On the other hand, current PRM pipelines lack scalability for multi-turn online RL 11 , relying either on labor-intensive human annotation ( e.g., 800,000 labels for math problems 27 ), or Monte Carlo roll-outs with 8-10 times more sample complexity 55 ; 20 . Moreover, unlike structured reasoning or tool-execution tasks with verifiable step-level labels 64 ; 65 ; 52 , defining the quality of an intermediate conversational turn is often ambiguous, requiring costly task-specific rubric design 62 . As to off-the-shelf LLM judges, they introduce prohibitive latency for online RL optimization 62 ; 25 and often suffer from evaluation bias 69 . Figure 1: The overall framework of ITPOfor User-LLM interactions with a real case example in Medical Recommendation. The optimization loop consists: of (1) Multi-turn online roll-out, (2) implicit PRM update with outcome rewards and token-level process reward estimation, (3) Turn-wise process reward aggregation and normalization, (4) Advantage estimation and policy optimization. Recently, implicit process reward models (implicit PRMs) 6 ; 63 ; 37 have emerged as a scalable alternative, deriving dense token-level process rewards in single-turn tasks directly from outcome rewards without manual annotation or auxiliary sampling. However, this token-level granularity often induces high variance and overfitting 39 , while lacking semantic interpretability due to its excessive resolution. In contrast, within multi-turn User-LLM interactions, each turn serves as a natural atomic unit for semantic planning. This level of granularity is inherently interpretable, which may facilitate monitoring by domain experts and practitioners and algorithmically reduce variance by aggregating signals across all tokens within a turn. Motivated by these insights, we propose Implicit Turn-wise Policy Optimization (ITPO), with our closed-loop online optimization framework illustrated in Figure 1 . Crucially, we introduce a normalization mechanism to calibrate the learned process rewards, thereby further enhancing training stability. ITPOintegrates seamlessly with various advantage functions, such as PPO 40 , GRPO 42 , and RLOO 2 ; 22 , to facilitate PPO-style 41 policy updates. We evaluate ITPOon three representative multi-turn collaborative tasks, namely Math Tutoring, Document Writing and Medical Recommendation. Analysis on training dynamics confirms that ITPOlearns stable and interpretable turn-wise preference, closely aligning with human judges. When integrated with PPO 41 , RLOO 2 ; 22 or GRPO 42 advantage estimators, the turn-wise rewards provided by ITPOconsistently outperform sparse outcome-based baselines as well as reward shaping methods such as PRIME 6 and LLMs-as-a-Judge 62 . 2 Related Works 2.1 LLM Alignment Towards Multi-Turn Dynamics While existing post-training based on supervised fine-tuning 29 and RL 41 ; 38 ; 42 prove effective for reactive instruction following 33 , the models struggle in multi-turn interactions 24 . We categorize previous efforts to mitigate this gap according to two distinct application scenarios: the reasoning and the interaction scenarios. Reasoning Dynamics: Goal Decomposition and Tool Use. The multi-step reasoning dynamics involve complex task decomposition, ranging from advanced problem-solving 13 ; 34 , long-horizon gaming 54 ; 57 , to agentic tool use 19 ; 12 ; 17 ; 35 . Trajectory-level standard PPO optimization 41 serves as fundamental baselines 54 ; 13 ; 70 . To address the reward sparsity 47 , 27 ; 52 train process reward models (PRMs) with up to 800k human annotated labels to evaluate intermediate math reasoning steps, yielding significant performance gains, while 55 construct step-level labels via Monte-Carlo sampling. In specific tasks, 9 perform baseline estimation by clustering identical or similar states in finite environments, 64 derive turn-level rewards from verifiable tool-use labels. Crucially, these successes rely on the verifiability of intermediate steps. Figure 2: Dynamics of the relative implicit rewards on a real Math Tutoring case. Turn-wise rewards of Response Turn #1 assigned by ITPOconsistently rank the highest from step 20 , meanwhile reflecting the importance of resolving ambiguity early in the conversation. By contrast, the token-level rewards show high variance, with the signals for identical tokens fluctuating across training steps (red boxes). Interaction Dynamics: Dialogue and Collaboration. The interaction dynamics focus on collaborative multi-turn tasks involving human users or agent partners, spanning social science simulation 62 ; 56 , recommendation systems 53 ; 4 , and collaborative assistance 59 ; 45 . 59 employ vanilla PPO with forward sampling for reward estimation in multi-turn collaborative tasks. Approaches that define fine-grained turn-wise rewards have shown effectiveness. For instance, 53 model categorical user belief states within a POMDP framework for simplified recommendation, while Sotopia-RL 62 leverage task-specific rigid rubrics and LLM-as-a-Judge to assign semantic turn-level rewards. However, these methods depend on restrictive task-specific priors. There is a critical need for scalable and automated algorithms to produce turn-wise rewards. For more related works, see Appendix. A.4 . 3 Methodology We present the methodology by first formalizing the problem of the multi-turn collaborative task and second the preliminary fundamentals of implicit PRM. Subsequently, we introduce ITPO, describing how it derives robust turn-wise rewards to enable scalable online policy optimization. 3.1 Problem Formulation We formulate the multi-turn collaborative task as a Partially Observable Markov Decision Process (POMDP)