StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning
Daoyu Wang \frontsup 1, Qingchuan Li \frontsup 1, Mingyue Cheng \frontsup 1 \corrauthor , Jie Ouyang \frontsup 1, Shuo Yu \frontsup 1, Qi Liu \frontsup 1, Enhong Chen \frontsup 1 · 2026
Read StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning by Daoyu Wang \frontsup 1, Qingchuan Li \frontsup 1, Mingyue Cheng \frontsup 1 \corrauthor , Jie Ouyang \frontsup 1, Shuo Yu \frontsup 1, Qi Liu \frontsup 1, Enhong Chen \frontsup 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18401.txt.
Description
Wang and eight colleagues propose CAPO, a critic-guided method for training language-model agents with reinforcement learning. Critic-free methods dominate, but critics suit long-horizon tasks by judging states and assigning credit. Standard critic methods work at token granularity, though environments change only after full actions. CAPO values states at action boundaries, assigns advantages to whole actions, and aggregates token ratios within actions with length calibration, beating representative methods on question-answering, paper search, and text-world tasks.
Catalog record
| CREATOR | Daoyu Wang \frontsup 1, Qingchuan Li \frontsup 1, Mingyue Cheng \frontsup 1 \corrauthor , Jie Ouyang \frontsup 1, Shuo Yu \frontsup 1, Qi Liu \frontsup 1, Enhong Chen \frontsup 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39233 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18401