StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning

Daoyu Wang \frontsup 1, Qingchuan Li \frontsup 1, Mingyue Cheng \frontsup 1 \corrauthor , Jie Ouyang \frontsup 1, Shuo Yu \frontsup 1, Qi Liu \frontsup 1, Enhong Chen \frontsup 1 · 2026

Read StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning by Daoyu Wang \frontsup 1, Qingchuan Li \frontsup 1, Mingyue Cheng \frontsup 1 \corrauthor , Jie Ouyang \frontsup 1, Shuo Yu \frontsup 1, Qi Liu \frontsup 1, Enhong Chen \frontsup 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18401.txt.

Description

Wang and eight colleagues propose CAPO, a critic-guided method for training language-model agents with reinforcement learning. Critic-free methods dominate, but critics suit long-horizon tasks by judging states and assigning credit. Standard critic methods work at token granularity, though environments change only after full actions. CAPO values states at action boundaries, assigns advantages to whole actions, and aggregates token ratios within actions with length calibration, beating representative methods on question-answering, paper search, and text-world tasks.

Catalog record

CREATORDaoyu Wang \frontsup 1, Qingchuan Li \frontsup 1, Mingyue Cheng \frontsup 1 \corrauthor , Jie Ouyang \frontsup 1, Shuo Yu \frontsup 1, Qi Liu \frontsup 1, Enhong Chen \frontsup 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39233
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18401

Browse the full library · About the Tower of Babel