SeeUPO : Sequence-Level Agentic-RL with Convergence Guarantees
Tianyi Hu, Qingxu Fu, Yanxi Chen, Zhaoyang Liu, Bolin Ding · 2026
Read SeeUPO : Sequence-Level Agentic-RL with Convergence Guarantees by Tianyi Hu, Qingxu Fu, Yanxi Chen, Zhaoyang Liu, Bolin Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06554.txt.
Description
Training language model agents by reinforcement learning lacks convergence guarantees, especially across multiple turns of interaction. Tianyi Hu and 4 co-authors analyze how update rules and advantage estimators combine, and propose SeeUPO, a method with guarantees for multi-turn settings. They find one combination, REINFORCE with group relative advantage estimation, can reach global optima undiscounted.
Pairing the same estimator with the popular PPO algorithm, however, breaks PPO's monotonic improvement property. No mainstream backbone algorithm, they show, manages to be both critic-free and convergence-guaranteed across turns. SeeUPO fills that gap by treating multi-turn interaction as a sequence of bandit problems, updating turn by turn in reverse order to guarantee steady improvement.
Catalog record
| CREATOR | Tianyi Hu, Qingxu Fu, Yanxi Chen, Zhaoyang Liu, Bolin Ding |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 140 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-06554