OPPO: Accelerating PPO-based RLHF via Pipeline Overlap

Kaizhuo Yan, Yingjie Yu, Yifan Yu, Haizhong Zheng, Fan Lai · 2025

Read OPPO: Accelerating PPO-based RLHF via Pipeline Overlap by Kaizhuo Yan, Yingjie Yu, Yifan Yu, Haizhong Zheng, Fan Lai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25762.txt.

Description

Kaizhuo Yan and co-authors speed up in 2025 the standard recipe for tuning language models on human feedback. The recipe runs several models in sequence, each waiting on the last, and a few very long replies stall every round. Their framework overlaps the steps instead of marching them in line.

Upstream output streams to the next model in right sized chunks, so scoring starts while writing continues. A second trick overbooks a few prompts and defers the longest jobs to later rounds. Training ran up to nearly three times faster, with graphics chips kept far busier. The wrapper drops into existing setups.

Catalog record

CREATORKaizhuo Yan, Yingjie Yu, Yifan Yu, Haizhong Zheng, Fan Lai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25762

Browse the full library · About the Tower of Babel