Single-stream Policy Optimization

Zhongwen Xu, Zihan Ding · 2025

Read Single-stream Policy Optimization by Zhongwen Xu, Zihan Ding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13232.txt.

Description

Zhongwen Xu and Zihan Ding replace group baselines in 2025 with a single-stream policy method called SPO. Group methods cut variance but waste work when groups collapse. Sync waits also curb scale in long tool tasks.

SPO keeps one lasting value tracker that adapts with divergence. Gains are scaled across the whole batch, so each sample teaches. No groups means higher throughput and prized sampling for a natural curriculum. On a mid Qwen model, SPO beats the group rival on five hard math sets. Gains stem from baseline and scaling choices, per ablations.

Catalog record

CREATORZhongwen Xu, Zihan Ding
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-13232

Browse the full library · About the Tower of Babel