Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning

Xu Wan, Yansheng Wang, Wenqi Huang, Mingyang Sun · 2026

Read Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning by Xu Wan, Yansheng Wang, Wenqi Huang, Mingyang Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-20722.txt.

Description

Xu Wan and three co-authors introduce Batch Adaptation Policy Optimization, an off-policy method for training large language models with verifiable rewards. Standard on-policy methods waste past experience and give flat rewards, which slows learning on hard samples.

BAPO picks training batches by re-checking hard samples from the past and reusing high-quality ones, while keeping a lower bound guarantee for policy improvement. Tests show it beats GRPO by twelve and a half percent on average across math, planning, and visual reasoning tasks. It also solves about forty-one percent of problems that base models consistently fail.

Catalog record

CREATORXu Wan, Yansheng Wang, Wenqi Huang, Mingyang Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-20722

Browse the full library · About the Tower of Babel