Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
Xu Wan, Yansheng Wang, Wenqi Huang, Mingyang Sun · 2026
Read Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning by Xu Wan, Yansheng Wang, Wenqi Huang, Mingyang Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-20722.txt.
Description
Xu Wan and three co-authors introduce Batch Adaptation Policy Optimization, an off-policy method for training large language models with verifiable rewards. Standard on-policy methods waste past experience and give flat rewards, which slows learning on hard samples.
BAPO picks training batches by re-checking hard samples from the past and reusing high-quality ones, while keeping a lower bound guarantee for policy improvement. Tests show it beats GRPO by twelve and a half percent on average across math, planning, and visual reasoning tasks. It also solves about forty-one percent of problems that base models consistently fail.
Catalog record
| CREATOR | Xu Wan, Yansheng Wang, Wenqi Huang, Mingyang Sun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-20722