A Comedy of Estimators: On KL Regularization in RL Training of LLMs

Vedant Shah, Johan Obando-Ceron, Vineet Jain, Brian Bartoldson, Bhavya Kailkhura, Sarthak Mittal, Glen Berseth, Pablo Samuel Castro, Yoshua Bengio, Nikolay Malkin, Moksh Jain, Siddarth Venkatraman, Aaron Courville · 2025

Read A Comedy of Estimators: On KL Regularization in RL Training of LLMs by Vedant Shah, Johan Obando-Ceron, Vineet Jain, Brian Bartoldson, Bhavya Kailkhura, Sarthak Mittal, Glen Berseth, Pablo Samuel Castro, Yoshua Bengio, Nikolay Malkin, Moksh Jain, Siddarth Venkatraman, Aaron Courville (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21852.txt.

Introduction

Vedant Shah, Yoshua Bengio, Aaron Courville, and a large team examine an implementation problem in reinforcement learning for language models. Training objectives include a KL penalty that keeps the trained policy near a reference policy. The divergence cannot be computed exactly, so practitioners plug in sample-based estimators, in several configurations, some inherited from open-source libraries.

Recent work showed some common setups give gradients that do not match the stated objective. This paper analyzes the gradient bias of the estimator configurations. Experiments fine-tuning three instruction models, from Qwen and Llama families, confirm the theory. In on-policy training, biased-gradient configurations caused instability, while unbiased configurations performed better both in-domain and out-of-domain. In off-policy, asynchronous setups, KL regularization helped stabilize training. The authors' message is practical: the estimator is not an implementation detail. Choosing its configuration changes what the training process is optimizing.

Catalog record

CREATORVedant Shah, Johan Obando-Ceron, Vineet Jain, Brian Bartoldson, Bhavya Kailkhura, Sarthak Mittal, Glen Berseth, Pablo Samuel Castro, Yoshua Bengio, Nikolay Malkin, Moksh Jain, Siddarth Venkatraman, Aaron Courville
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-21852

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Comedy of Estimators: On KL Regularization in RL Training of LLMs Vedant Shah, Johan Obando-Ceron, Vineet Jain, Brian Bartoldson, Bhavya Kailkhura, Sarthak Mittal, Glen Berseth, Pablo Samuel Castro, Yoshua Bengio, Nikolay Malkin, Moksh Jain, Siddarth Venkatraman, Aaron Courville Abstract The reasoning performance of large language models (LLMs) can be substantially improved by training them with reinforcement learning (RL). The RL objective for LLM training involves a regularization term, which is the reverse Kullback-Leibler (KL) divergence between the trained policy and the reference policy. Since computing the KL divergence exactly is intractable, various estimators are used in practice to estimate it from on-policy samples. Despite its wide adoption, including in several open-source libraries, there is no systematic study analyzing the numerous ways of incorporating KL estimators in the objective and their effect on the downstream performance of RL-trained models. Recent works show that prevailing practices for incorporating KL regularization do not provide correct gradients for stated objectives, creating a discrepancy between the objective and its implementation. In this paper, we further analyze these practices and study the gradients of several estimators configurations, revealing how design choices shape gradient bias. We substantiate these findings with empirical observations by RL fine-tuning Qwen2.5-7B , Llama-3.1-8B-Instruct and Qwen3-4B-Instruct-2507 with different configurations and evaluating their performance on both in- and out-of-distribution tasks. Through our analysis, we observe that, in on-policy settings: (1) estimator configurations with biased gradients can result in training instabilities; and (2) using estimator configurations resulting in unbiased gradients leads to better performance on in-domain as well as out-of-domain tasks. We also investigate the performance resulting from different KL configurations in off-policy settings and observe that KL regularization can help stabilize off-policy RL training resulting from asynchronous setups. Overall, our findings provide useful takeaways for using KL-regularized objectives during RL post-training of LLMs. 1 Introduction Reinforcement learning (RL) has become an indispensable component of present-day post-training pipelines for large language models (LLM). RL fine-tuning of LLMs was initially popularized for human preference alignment and instruction-following ( 27 ) . Since then, RL has played a critical role in reasoning-oriented post-training of LLMs. Recent work ( 15 ; 10 ) has shown that training LLMs as RL policies on verifiable reasoning tasks such as mathematics and coding leads to a substantial performance improvements. Subsequently, there has been rapid progress in approaches for reasoning-oriented RL training of LLMs. For instance, the performance of DeepSeek-R1 ( 10 ) is matched by much smaller models ( 46 ) . At the same time, this progress has been accompanied by inconsistent design choices and implementation errors in open source RL fine-tuning pipelines ( 39 ; 49 ) . One such design choice is the use of Kullback-Leibler (KL) divergence between the trained policy and the base policy as a regularization term in the objective ( 35 ; 27 ) . This regularization ensures that the policy explores within the space of coherent sequences by constraining it within the support of the base model, mitigating problems such as language drift, reward over-optimization ( 8 ) or catastrophic forgetting ( 25 ; 30 ) of the information present in the base model. Specifically, the reverse KL divergence is used for this regularization so that the learned policy assigns high probability mass to a narrow set of high-reward trajectories. This is opposed to the forward KL divergence, which tends to maintain probability mass over the entire support of the base model at the expense of performance. A regularization coefficient \beta controls the trade-off between reward maximization and proximity to the base model. However, it is intractable to compute the reverse KL divergence exactly owing to the high-dimensionality of the space of possible sequences. As a result, different sample-based estimators of the reverse KL divergence are used in practice ( 32 ; 49 ; 2 ) . In addition to differences in their approximations, these estimators may be incorporated into the objective in different ways: previous work on RLHF with PPO ( 27 ) adds the KL penalty to the task reward ( i.e. , no direct gradients); methods such as GRPO ( 35 ; 10 ) popularized adding the KL term directly to the loss. The choice of the estimator, regularization coefficient, and whether it is added to the reward or directly to the loss can have a significant effect on the training stability, convergence rate, and out-of-distribution generalization of the trained models. Moreover, 39 identified that some of these choices lead to biased estimates of the true gradient. For example, using the KL estimator in the loss function, as popularized by GRPO, results in biased gradients and therefore does not optimize the intended reverse KL-regularized objective. These issues have propagated to widely used public libraries, leading to potentially incorrect results when using KL regularization ( 36 ; 43 ; 13 ; 7 ) . These findings highlight that while KL regularization is ubiquitous in RL training of LLMs, the implementation details are poorly understood and often overlooked. In this work, we attempt to fill this gap by providing a systematic empirical exploration of the space of some design choices associated with the practical use of KL regularization . We study this in the context of reinforcement learning with verifiable rewards ( 41 ; 20 , RLVR;) , which has become the dominant paradigm for improving the reasoning abilities of LLMs. Specifically, we investigate two commonly used unbiased estimators of reverse KL divergence in an on-policy setting – the naïve or K1 estimator, and the Schulman or the K3 estimator ( 32 ) . First, we analytically study the bias of the gradients with respect to the true gradient of sequence-level reverse KL divergence, when these estimates are added to the reward versus when directly added to the loss ( section 3 , table 1 ). Next, we empirically investigate the bias of gradient estimates in each case in a synthetic setting ( Section 3.3 ), substantiating our prior discussion. Finally, we present empirical results studying how these choices affect RL fine-tuning of Qwen2.5-7B ( 47 ) and Llama-3.1-8B-Instruct ( 40 ) on a mathematical reasoning task in on-policy settings, across different values of the KL regularization coefficient \beta , and study both in- and out-of-domain performance of the resulting models ( section 4.1 ). We also study the effect of KL-regularization in highly asynchronous training setting ( 26 ; 3 ) , which is preferrable in practice owing to its efficiency. We train Qwen3-4B-Instruct-2507 ( 46 ) with RL on a mathematical reasoning task and a general reasoning task under high asynchronicity. In summary, our key findings are as follows. Key observations : • Unbiased estimates of the sequence level reverse KL divergence can result in biased gradients depending on their usage. • Configurations inducing biased gradients can lead to unstable training and even precipitate complete collapse. • Configurations that lead to unbiased gradient estimates result in better-performing models, across both in-domain and out-of-domain evaluation tasks. • KL-regularization can stabilize learning from highly off-policy data in asynchronous setups. 2 Background We study the problem of fine-tuning a base language model \pi_{\mathrm{ref}} with reinforcement learning. Given a reward function R(\cdot) and a set of observations {\mathcal{D}} comprising question-answer pairs (x,y) ,

Read the complete text