Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs
Junbo Li, Peng Zhou, Rui Meng, Meet P. Vadera, Lihong Li, Yang Li, [-3em] The University of Texas at Austin · 2025
Read Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs by Junbo Li, Peng Zhou, Rui Meng, Meet P. Vadera, Lihong Li, Yang Li, [-3em] The University of Texas at Austin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17008.txt.
Introduction
Junbo Li and coauthors present turn-PPO, a variant of Proximal Policy Optimization for training language model agents over many turns. The popular GRPO method wobbles on long-horizon agent tasks, where credit for a win must be split across many steps.
The team first shows plain PPO already beats GRPO in stability for multi-turn work. turn-PPO then recasts the task at turn level, not token level, so advantage scores attach to whole replies. On WebShop and Sokoban, turn-PPO beat the baselines with and without long reasoning traces. The fix is simple: match the training unit to the unit the agent actually chooses.
Catalog record
| CREATOR | Junbo Li, Peng Zhou, Rui Meng, Meet P. Vadera, Lihong Li, Yang Li, [-3em] The University of Texas at Austin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-17008
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs Junbo Li, Peng Zhou, Rui Meng, Meet P. Vadera, Lihong Li, Yang Li, [-3em] The University of Texas at Austin Abstract Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments. However, directly applying the widely used Group Relative Policy Optimization (GRPO) algorithm to multi-turn tasks exposes notable limitations, particularly in scenarios requiring long-horizon reasoning. To address these challenges, we investigate more stable and effective advantage estimation strategies, especially for multi-turn settings. We first explore Proximal Policy Optimization (PPO) as an alternative and find it to be more robust than GRPO. To further enhance PPO in multi-turn scenarios, we introduce turn-PPO, a variant that operates on a turn-level MDP formulation, as opposed to the commonly used token-level MDP. Our results on the WebShop and Sokoban datasets demonstrate the effectiveness of turn-PPO, both with and without long reasoning components. 1 Introduction LLMs have shown strong potential for tool use and environment interaction ( Wang et al. 2025b ; Feng et al. 2025 ; Wei et al. 2025 ) . In these settings, reinforcement learning (RL) is commonly employed because base LLMs already possess strong foundational abilities, and the main objective is to teach them to reason and use tools effectively through interaction with the new environments. Unlike single-turn reasoning tasks, where all response tokens are generated in a single pass and the problem can be framed as a bandit setting, multi-turn tool-use scenarios are better modeled as Markov Decision Processes (MDPs), which involve sequential decision-making and state transitions. The current widely used RL method for this multi-turn setup is a direct adaptation of GRPO ( Wang et al. 2025b ; Wei et al. 2025 ) . This method samples multiple multi-turn trajectories per question, then estimates the advantage for all tokens in a trajectory by normalizing the trajectory-level reward using the group’s mean and standard deviation. However, this adaptation faces several challenges in the multi-turn context that can impair optimization effectiveness: 1) Environment interactions are not fully controllable, leading to higher sampling variance compared to single-turn settings, which makes advantage estimation less stable. 2) Different turns within a trajectory contribute unequally to the final reward, so applying the same advantage uniformly to all tokens across turns can introduce inaccuracies. Several prior works ( Feng et al. 2025 ; Zeng et al. 2025 ) attempt to address this issue by incorporating turn-level advantage estimates. However, these methods are specialized for particular settings and can introduce additional biases, limiting the generalizability. To address these issues, we reintroduce PPO ( Schulman et al. 2017 ) into multi-turn agentic LLM training. Unlike GRPO, which relies on multiple rollouts to estimate advantages and often suffers from instability, PPO leverages a learnable critic model for advantage estimation. Because it is designed for multi-step MDPs, PPO naturally supports effective training through Generalized Advantage Estimation (GAE). In existing multi-turn implementations, however, PPO is typically applied by treating each token as a separate MDP step, a practice inherited from single-turn settings. We show that this formulation is suboptimal for multi-turn tasks because it complicates critic learning due to the mismatch between token-level granularity and the underlying task structure. To improve PPO’s effectiveness, we propose turn-PPO , which redefines the MDP formulation at the turn level. Specifically, we treat the entire input and output of a turn as a single state–action pair rather than operating at the token level. This results in a more coherent representation, enables the critic to learn more accurately, and produces more reliable advantage estimates. Contributions Our main contributions are as follows: • We identify and analyze the instability of sampling-based advantage estimation used in GRPO and its variants in multi-turn settings. • We show that PPO achieves greater training stability and efficiency than GRPO due to its learnable critic and more accurate advantage estimation. • We introduce turn-PPO , a variant of token-PPO, and demonstrate its superior performance across diverse tasks and settings. 1.1 Related work Multi-turn agentic LLMs It is well established that, rather than solving a problem in a single step, LLMs can reason and act step by step with the help of external tools ( Yao et al. 2023 ) and use intermediate results to guide future reasoning steps. When augmented with tool-calling abilities, they can interact with external environments to solve a wide range of tasks, including web navigation ( Wei et al. 2025 ; Yao et al. 2022 ; Gur et al. 2023 ; Putta et al. 2024 ; Jin et al. 2025 ; Qi et al. 2024 ) , GUI automation ( Qin et al. 2025 ; Wang et al. 2025a ) , and embodied AI control ( Li et al. 2024 ; Huang et al. 2022 ) . In this work, we restrict our study to text-only environments, enabling us to concentrate purely on improving reinforcement learning algorithms for training LLMs. RL for LLMs By modeling LLMs as MDPs, we can directly apply policy-based reinforcement learning algorithms to train them. In the single-turn setting, strictly on-policy REINFORCE-style methods have been explored, including Reinforce++ ( Hu et al. 2025 ) , RLOO ( Ahmadian et al. 2024 ) , and ReMax ( Li et al. 2023 ) , as well as off-policy PPO-style methods such as VinePPO ( Kazemnejad et al. 2024 ) , VC-PPO ( Yuan et al. 2025 ) , VAPO ( Yue et al. 2025 ) . Building on PPO, DeepSeek-R1 ( Guo et al. 2025 ) introduces GRPO ( Shao et al. 2024 ) , which replaces the learned critic with sample-based advantage estimation. Several follow-up works, including DAPO ( Yu et al. 2025 ) , Dr. GRPO ( Liu et al. 2025 ) , Reinforce-Rej ( Xiong et al. 2025 ) , and GSPO ( Zheng et al. 2025 ) further refine or extend GRPO to improve stability and performance. Extending from the single-turn to the multi-turn setting, several RL variants have been proposed. ArCHer ( Zhou et al. 2024 ) introduces a hierarchical RL framework that employs an actor–critic algorithm at the turn level and a policy-gradient update at the token level. WebAgent-R1 ( Wei et al. 2025 ) introduces M-GRPO, a direct adaptation of GRPO to multi-turn tasks. However, GRPO is widely reported to be unstable in this setting. To address this, RAGEN ( Wang et al. 2025b ) proposes StarPO-s, which uses proportional trajectory filtering, while GiGPO ( Feng et al. 2025 ) refines advantage estimation by combining state-level advantages obtained by merging identical states with trajectory-level advantages. MT-GRPO ( Zeng et al. 2025 ) further demonstrates the benefit of turn-level credit assignment. Nonetheless, these approaches are not fully general and often require manual tuning of turn-level credit weights. In this work, we reintroduce a learnable critic, as in PPO, but applied at the turn level, providing effective turn-level advantage estimates and mitigating the limitations of GRPO in multi-turn training. 2 Framework In this section, we introduce our RL framework for multi-turn agentic LLM tasks. Section 2.1 presents the general LLM-as-MDP formulation, followed by Section 2.2 , which illustrates the token-MDP formulation and analyzes its limitations. To address these issues, Section 2.3