SeeUPO : Sequence-Level Agentic-RL with Convergence Guarantees

Tianyi Hu, Qingxu Fu, Yanxi Chen, Zhaoyang Liu, Bolin Ding · 2026

Read SeeUPO : Sequence-Level Agentic-RL with Convergence Guarantees by Tianyi Hu, Qingxu Fu, Yanxi Chen, Zhaoyang Liu, Bolin Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06554.txt.

Description

Training language model agents by reinforcement learning lacks convergence guarantees, especially across multiple turns of interaction. Tianyi Hu and 4 co-authors analyze how update rules and advantage estimators combine, and propose SeeUPO, a method with guarantees for multi-turn settings. They find one combination, REINFORCE with group relative advantage estimation, can reach global optima undiscounted.

Pairing the same estimator with the popular PPO algorithm, however, breaks PPO's monotonic improvement property. No mainstream backbone algorithm, they show, manages to be both critic-free and convergence-guaranteed across turns. SeeUPO fills that gap by treating multi-turn interaction as a sequence of bandit problems, updating turn by turn in reverse order to guarantee steady improvement.

Catalog record

CREATORTianyi Hu, Qingxu Fu, Yanxi Chen, Zhaoyang Liu, Bolin Ding
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE140 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-06554

Browse the full library · About the Tower of Babel