Single-stream Policy Optimization
Zhongwen Xu, Zihan Ding · 2025
Read Single-stream Policy Optimization by Zhongwen Xu, Zihan Ding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13232.txt.
Description
Zhongwen Xu and Zihan Ding replace group baselines in 2025 with a single-stream policy method called SPO. Group methods cut variance but waste work when groups collapse. Sync waits also curb scale in long tool tasks.
SPO keeps one lasting value tracker that adapts with divergence. Gains are scaled across the whole batch, so each sample teaches. No groups means higher throughput and prized sampling for a natural curriculum. On a mid Qwen model, SPO beats the group rival on five hard math sets. Gains stem from baseline and scaling choices, per ablations.
Catalog record
| CREATOR | Zhongwen Xu, Zihan Ding |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-13232