KL for a KL: On-Policy Distillation with Control Variate Baseline
Minjae OhGraduate School of Data ScienceSeoul National University [email protected] Sangjun SongGraduate School of Data ScienceSeoul National University [email protected] Gyubin ChoiGraduate School of Data ScienceSeoul National University [email protected] Yunho ChoiGraduate School of Data ScienceSeoul National University [email protected], Yohan JoGraduate School of Data ScienceSeoul National University [email protected] · 2026
Read KL for a KL: On-Policy Distillation with Control Variate Baseline by Minjae OhGraduate School of Data ScienceSeoul National University [email protected] Sangjun SongGraduate School of Data ScienceSeoul National University [email protected] Gyubin ChoiGraduate School of Data ScienceSeoul National University [email protected] Yunho ChoiGraduate School of Data ScienceSeoul National University [email protected], Yohan JoGraduate School of Data ScienceSeoul National University [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07865.txt.
Introduction
Minjae Oh and four co-authors stabilize on-policy distillation, the training method where a student language model learns from a teacher on its own generated outputs. In practice the method is unstable because its single-sample gradient estimator has high variance, and good recipes are still immature. Their fix imports a classic tool from reinforcement learning: a control variate baseline.
The neat part is the value function: it has a closed form as the per-token negative reverse KL divergence between student and teacher. It is already available from the forward pass, so no extra critic is needed. Subtracting it keeps gradients unbiased while cutting variance. Across math and science reasoning benchmarks, their vOPD beat plain distillation and matched the expensive full-vocabulary baseline at a fraction of the cost.
Catalog record
| CREATOR | Minjae OhGraduate School of Data ScienceSeoul National University [email protected] Sangjun SongGraduate School of Data ScienceSeoul National University [email protected] Gyubin ChoiGraduate School of Data ScienceSeoul National University [email protected] Yunho ChoiGraduate School of Data ScienceSeoul National University [email protected], Yohan JoGraduate School of Data ScienceSeoul National University [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07865
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KL for a KL: On-Policy Distillation with Control Variate Baseline Minjae OhGraduate School of Data ScienceSeoul National University [email protected] Sangjun SongGraduate School of Data ScienceSeoul National University [email protected] Gyubin ChoiGraduate School of Data ScienceSeoul National University [email protected] Yunho ChoiGraduate School of Data ScienceSeoul National University [email protected], Yohan JoGraduate School of Data ScienceSeoul National University [email protected] Abstract On-Policy Distillation (OPD) has emerged as a dominant post-training paradigm for large language models, especially for reasoning domains. However, OPD remains unstable in practice due to the high gradient variance of its single-sample Monte Carlo estimator, and recipes for stable training are still immature. We propose v OPD ( O n- P olicy D istillation with a control v ariate baseline), which casts OPD as policy-gradient RL and stabilizes it by introducing a control variate baseline—canonically a value function—from the RL literature. We show that the OPD value function admits a closed form as the per-token negative reverse KL divergence between the student and the teacher, available directly from the already-computed forward pass with no additional critic or inference. Existing stabilization methods either compute the full token-level reverse KL over the entire vocabulary, adding significant overhead, or restrict it to a top- k k support, biasing the objective. v OPD instead preserves the lightweight single-sample estimator, subtracting the value function as a detached baseline to keep the gradient unbiased while reducing variance. Furthermore, we show that a top- k k approximation of the baseline further lowers cost without compromising performance. Across mathematical and scientific reasoning benchmarks, v OPD consistently outperforms vanilla OPD and matches the most expensive full-vocabulary baseline, offering an efficient stabilization of On-Policy Distillation through principled RL variance reduction. 1 1 1 Code is available at https://github.com/holi-lab/vOPD . 2 2 footnotetext: Corresponding author. 1 Introduction Large Language Models have made remarkable advances in reasoning, accompanied by improvements in post-training recipes [ 12 , 43 , 38 ] . A key factor has been Reinforcement Learning with Verifiable Rewards (RLVR) [ 18 , 8 ] , which trains LLMs directly against easily verifiable rewards—answer correctness, code execution—sidestepping the noise and reward hacking introduced by learned reward models. RLVR has been successful thanks to its simple recipe, but this simplicity comes at a cost: LLMs generate thousands of intermediate tokens during reasoning before receiving a single scalar reward for the final answer. RLVR methods must perform credit assignment over long chains of thought from a single sparse scalar signal. This sparse supervision demands large rollouts and prolonged training, making training progress painfully slow. On-Policy Distillation (OPD) [ 7 , 1 ] has emerged as an attractive alternative to RLVR when a strong teacher is available. Rather than relying on a sparse terminal reward, OPD minimizes the reverse KL divergence between the student and the teacher via dense, token-level signals, enabling faster training [ 21 ] . Because it is on-policy and reward-driven, OPD can naturally be implemented using standard RL pipelines with a single-sample Monte Carlo estimator [ 23 ] , and empirically matches RLVR accuracy with a fraction of the compute [ 29 ] . Its effectiveness has been demonstrated in industrial-level post-training such as Qwen3, GLM-5, Nemotron-Cascade2, and DeepSeek-V4 [ 43 , 46 , 44 , 4 ] . Despite this success, OPD’s optimization recipe remains underdeveloped: training is unstable in practice, and stabilization techniques are still immature relative to the recipes that drive successful RLVR training [ 45 , 14 , 49 , 3 , 22 ] . The most widely adopted fix replaces the single-sample estimator with a full-vocabulary token-level KL, incurring additional compute overhead [ 1 ] ; a lighter-weight variant restricts the KL to a top- k support, which biases the gradient away from the true objective and still adds compute, yet yields only marginal gains [ 20 ] . In contrast, we turn to the RL interpretation of OPD and propose a principled, low-compute method that controls variance while preserving the efficient single-sample Monte Carlo estimator. We propose v OPD ( O n- P olicy D istillation with a control v ariate baseline), which leverages a standard tool from policy-gradient RL to reduce gradient variance: subtracting a control variate baseline [ 41 , 36 , 6 ] . Baseline subtraction for variance reduction underlies actor-critic methods such as PPO, and more recently GRPO and RLOO [ 33 , 26 , 2 , 34 ] (see § 3.1 ). v OPD reduces variance without biasing the gradient in expectation. The standard choice of baseline is the value function, and we show that for OPD this quantity admits a computable closed form: the per-token negative reverse KL between the student and the teacher at each token. The baseline is therefore available from the same forward pass that already computes the OPD objective—without an additional critic model, extra rollouts, or additional backward passes. We show that the baseline can be approximated using only the top- k student tokens at a lower cost; crucially, because this approximation does not depend on the sampled token, it preserves the unbiasedness of the gradient regardless of k (see § 3.2 ). Furthermore, we find empirically that the choice of k has little effect on performance (see § 4.3 ). We evaluate v OPD on four models from the Qwen3 [ 43 ] and Olmo-3 [ 27 ] families across six reasoning benchmarks spanning mathematics and science—MATH500 [ 9 , 21 ] , Minerva Math [ 19 ] , AMC23 [ 25 ] , AIME24/25 [ 24 ] , SciKnowEval [ 5 ] , and GPQA-Diamond [ 31 ] —demonstrating consistent improvements over baseline methods. v OPD delivers an absolute average accuracy gain of up to +3% on average over base OPD, with improvements of up to +6.2% on MATH500 (see §§ 4.2 and 4.4 ). Against the two stabilization variants, v OPD substantially outperforms top- k OPD and matches full-vocabulary OPD while reducing wall-clock time up to 57.7%. We further validate the stability of v OPD through consistently lower gradient norms, and show that it acts as a regularizer on destabilizing reward tokens (see § 4.3 ). Overall, v OPD bridges RL and knowledge distillation, providing a principled, efficient approach to stable On-Policy Distillation. 2 Preliminaries 2.1 On-Policy Distillation On-Policy Distillation (OPD) [ 1 , 7 ] trains the student by minimizing the reverse KL divergence between the student ( \pi_{\theta} ) and the teacher ( \pi_{T} ): \mathbb{D}_{\mathrm{KL}}\!\left(\pi_{\theta}\,\|\,\pi_{T}\right)\;=\;\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_{\theta}(\cdot\mid x)}\!\left[\log\frac{\pi_{\theta}(y\mid x)}{\pi_{T}(y\mid x)}\right], (1) where x is a prompt drawn from a dataset \mathcal{D} and y=(y_{1},\ldots,y_{|y|}) is a response of length |y| . Importantly, OPD samples from the student during generation to obtain an unbiased estimator of the KL. On-policy learning mitigates exposure bias [ 30 ] —the train-test discrepancy in off-policy training, where the model is trained on static data but conditions on its own outputs at test time. This has enabled effective training on long Chain-of-Thought reasoning tasks such as mathematics [ 40 , 43 , 23 ] . In practice, Eq. ( 1 ) is commonly optimized via a single-sample Monte Carlo estimate, by maximizing the following token-level objective, where c_{t}=(x,y_{<t}) denotes the context at step t [ 23 ] : \mathcal{J}_{\text{OPD}}(\theta)\;=\;\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_{\theta}(\cdot\mid x)}\!\left[\sum_{t=1}^{|y|}\log\frac{\pi_{T}(y_{t}\mid c_{t})}{\pi_{\theta}(y_{t}\mid c_{t})}\right]. (2) Following recent practice [ 23 ,