OPPO: Accelerating PPO-based RLHF via Pipeline Overlap
Kaizhuo Yan, Yingjie Yu, Yifan Yu, Haizhong Zheng, Fan Lai · 2025
Read OPPO: Accelerating PPO-based RLHF via Pipeline Overlap by Kaizhuo Yan, Yingjie Yu, Yifan Yu, Haizhong Zheng, Fan Lai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25762.txt.
Description
Kaizhuo Yan and co-authors speed up in 2025 the standard recipe for tuning language models on human feedback. The recipe runs several models in sequence, each waiting on the last, and a few very long replies stall every round. Their framework overlaps the steps instead of marching them in line.
Upstream output streams to the next model in right sized chunks, so scoring starts while writing continues. A second trick overbooks a few prompts and defers the longest jobs to later rounds. Training ran up to nearly three times faster, with graphics chips kept far busier. The wrapper drops into existing setups.
Catalog record
| CREATOR | Kaizhuo Yan, Yingjie Yu, Yifan Yu, Haizhong Zheng, Fan Lai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-25762