TAPERED OFF-POLICY REINFORCE Stable and efficient reinforcement learning for LLMs
Nicolas Le Roux Marc G. Bellemare, Jonathan Lebensold Arnaud Bergeron Joshua Greaves, Alex Fréchette Carolyne Pelletier Eric Thibodeau-Laufer, Sándor Toth Sam Work · 2025
Read TAPERED OFF-POLICY REINFORCE Stable and efficient reinforcement learning for LLMs by Nicolas Le Roux Marc G. Bellemare, Jonathan Lebensold Arnaud Bergeron Joshua Greaves, Alex Fréchette Carolyne Pelletier Eric Thibodeau-Laufer, Sándor Toth Sam Work (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14286.txt.
Introduction
Nicolas Le Roux and co-authors introduce TOPR, a reinforcement learning algorithm for fine-tuning large language models that stays stable without the usual safety leash. Standard methods learn only from fresh model output, which wastes compute and forbids reusing stored examples. Off-policy learning reuses data but often destabilizes.
Their tapered, asymmetric importance weighting lets the learner use old and new examples, good and bad, in one framework. No KL penalty is needed to keep training calm. On grade-school and competition math benchmarks, TOPR improved both solution generation and a generative verifier that checks answers. Simplicity is part of the pitch: the method is a Monte Carlo algorithm at heart.
Catalog record
| CREATOR | Nicolas Le Roux Marc G. Bellemare, Jonathan Lebensold Arnaud Bergeron Joshua Greaves, Alex Fréchette Carolyne Pelletier Eric Thibodeau-Laufer, Sándor Toth Sam Work |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14286
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TAPERED OFF-POLICY REINFORCE Stable and efficient reinforcement learning for LLMs Nicolas Le Roux Marc G. Bellemare, Jonathan Lebensold Arnaud Bergeron Joshua Greaves, Alex Fréchette Carolyne Pelletier Eric Thibodeau-Laufer, Sándor Toth Sam Work Abstract We propose a new algorithm for fine-tuning large language models using reinforcement learning. Tapered Off-Policy REINFORCE (TOPR) uses an asymmetric, tapered variant of importance sampling to speed up learning while maintaining stable learning dynamics, even without the use of KL regularization. TOPR can be applied in a fully offline fashion, allows the handling of positive and negative examples in a unified framework, and benefits from the implementational simplicity that is typical of Monte Carlo algorithms. We demonstrate the effectiveness of our approach with a series of experiments on the GSM8K and MATH reasoning benchmarks, finding performance gains for training both a model for solution generation and as a generative verifier. We show that properly leveraging positive and negative examples alike in the off-policy regime simultaneously increases test-time accuracy and training data efficiency, all the while avoiding the “wasted inference” that comes with discarding negative examples. We find that this advantage persists over multiple iterations of training and can be amplified by dataset curation techniques, enabling us to match 70B-parameter model performance with 8B language models. As a corollary to this work, we find that REINFORCE’s baseline parameter plays an important and unexpected role in defining dataset composition in the presence of negative examples, and is consequently critical in driving off-policy performance. 1 Introduction Reinforcement learning (RL) and EM-type methods are rapidly becoming the dominant paradigm for fine-tuning large language models on complex tasks such as chain-of-thought reasoning. These methods can amplify a base model’s performance without additional human data and can optimize for synthetic rewards ( Zhang et al. 2024a ) and non-differentiable objectives ( Chow et al. 2024 ) . While several popular methods rely solely on positive examples to fine-tune an LLM Zelikman et al. 2022 ; Gulcehre et al. 2023 , the “trial and error” nature of RL algorithms is especially well-positioned to leverage negative examples produced by the model, which are increasingly being recognized as key to efficient learning ( Rafailov et al. 2023 ; Tajwar et al. 2024 ; Zhang et al. 2024b ; Flet-Berliac et al. 2024 ) . In fact, there is mounting evidence that the simplest of all methods, REINFORCE ( Williams 1992 ) , is a highly effective approach to fine-tuning LLMs ( Ahmadian et al. 2024 ) . However, REINFORCE is essentially an on-policy algorithm. In the presence of negative rewards, its good behaviour can only be guaranteed when the training data distribution (or reference model) matches, or is close to the model’s own distribution. This limits its ability to reuse past data, and puts pressure on the experimenter to select a just-right set of hyperparameters to avoid problems. Indeed, evidence of the instability of REINFORCE-type in off-policy training of LLMs can be found everywhere in the literature, from early work Pang and He 2021 to key algorithmic choices made in the training of the Kimi \kappa 1.5 ( Team et al. 2025 ) and DeepSeek-R1 ( Guo et al. 2025 ) models. While KL regularization to the objective can mitigate this instability, it results in slower learning and requires additional hyperparameter tuning. As an alternative we propose Tapered Off-Policy REINFORCE (TOPR), an algorithm that is stable even when the model differs substantially from the data distribution, while leveraging positive and negative examples to their fullest. TOPR improves a language model \pi (or policy ) by means of the asymmetric policy gradient \displaystyle\nabla J_{\textsc{topr}}(\pi) \displaystyle=\sum_{\tau:R(\tau)\geq 0}\mu(\tau)R(\tau)\nabla\log\pi(\tau)+\sum_{\tau:R(\tau)<0}\mu(\tau)\left[\frac{\pi(\tau)}{\mu(\tau)}\right]_{0}^{1}R(\tau)\nabla\log\pi(\tau)\;, (1) where \tau is a response (or trajectory ) sampled from some data-generating policy \mu , R(\tau) is the reward associated with this trajectory, and [x]_{a}^{b} denotes the usual clipping function. Unlike other LLM algorithms in the REINFORCE family, TOPR does not require an explicit KL penalty to guarantee stable behaviour, making it both simpler to implement and computationally more efficient. Compared to PPO ( Schulman et al. 2017 ) , DPO ( Rafailov et al. 2023 ) , and the “naive” application of REINFORCE ( Ahmadian et al. 2024 ) , TOPR continues to improve reasoning performance even once \pi differs substantially from \mu (Figure 1 ). We characterize the stable off-policy performance of TOPR by using it to train language models to reason within the GSM8K and MATH benchmarks. We use these benchmarks to highlight the various factors at play in off-policy reinforcement learning of language models, in particular the importance of positive/negative balance in the training dataset. Surprisingly, we find that REINFORCE’s baseline parameter – commonly used as a variance reduction mechanism – plays an altogether different role of balancing the dataset in this regime, and is essential to good performance. Critically, our results indicate that choosing the best baseline requires taking more than only the mean return into account. We conclude with a series of multi-iteration experiments showing the ability of TOPR to fine-tune language models well beyond their base benchmark performance. Algorithm 1 TOPR (single iteration) Input: Language model \pi , reference \mu Input: Prompts x_{1},\dots,x_{m} y_{i}^{1},\dots,y_{i}^{n}\sim\mu(\cdot\,|\,x_{i}) for i=1,\dots,m Make dataset \mathcal{D}=\{x_{i},y_{i}^{j}\} for (x,y)\in\mathcal{D} do \alpha=\left[\tfrac{\pi(y\,|\,x)}{\mu(y\,|\,x)}\right]_{0}^{1} if R(x,y)<0 else 1 \ell=\textsc{stop-grad}(\alpha)r(x,y)\log\pi(y\,|\,x) Perform gradient step w.r.t. loss \ell end for Figure 1: Left: Test set accuracy (pass@1) on the GSM8K benchmark ( Cobbe et al. 2021 ) over the course of off-policy fine-tuning of the Llama 3 8B model. As training becomes increasingly off-policy, the naive use of the REINFORCE gradient causes substantial performance degradation and PPO stops improving. DPO, which handles negative examples through a preference-based formulation, fares better but still falls well short of TOPR’s performance. Pass@1 refers to the usual single-reasoning accuracy. See Section 4 for experimental details. Right: TOPR pseudo-code (Section 3 ). 2 Off-policy policy optimization We consider an autoregressive language model \pi that, given a prompt x , assigns a probability to a length- n response y according to \pi(y\,|\,x)=\prod_{i=1}^{n}\pi(y_{i}\,|\,x,y_{<i}). Given a reward function R(x,y) that measures the quality of the response y to x and a dataset of prompts x_{1},\dots,x_{m} , we consider the problem of maximizing the expected reward J(\pi)=\frac{1}{m}\sum_{j=1}^{m}\Big[\expect_{y\sim\pi(\cdot\,|\,x_{j})}R(x_{j},y)\Big]. In this paper we abstract the prompt-response relationship and view this problem through the lens of policy optimization, where \tau is a trajectory produced by the language model (i.e., the policy). With mild abuse of notation, we thus write J(\pi)=\expect_{\tau\sim\pi}R(\tau). The original REINFORCE algorithm ( Williams 1992 ) maximizes J(\pi) through the process of on-policy policy optimization . In the simplest form of the algorithm, a single trajectory \tau is sampled according to \pi , and the parametrized policy \pi is updated according to the unbiased gradient estimate \nabla\hat{J}(\pi)=R(\tau)\nabla\log\pi(\tau), (2) whose expectation is \nabla J(\pi)=\expect_{\tau\sim\pi}R(\tau)\nabla\log\pi(\tau). In practice, training is rarely truly on-policy, for example because data is generated in a parallel, asynchronous fashion ( Mnih et al. 2016 ) or in a separate “sidecar” process ( Noukhovitch et al. 2024 ; Guo et al. 2025 ) . It is obviously also desirable to reuse trajectories throughout training, especially when generating these trajectories incurs a substantial computational cost or because they have been generated by a different process (e.g., expert trajectories). In the off-policy policy optimization (OPPO) setting, we assume