OPD+ : Rethinking the Advantage Design for On-Policy Distillation

Hanyang Zhao, Haoxian Chen, Han Lin, Genta Indra Winata, David Yao, Wenpin Tang · 2026

Read OPD+ : Rethinking the Advantage Design for On-Policy Distillation by Hanyang Zhao, Haoxian Chen, Han Lin, Genta Indra Winata, David Yao, Wenpin Tang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01039.txt.

Introduction

Zhao and co-authors fix a quiet bias in how students learn from teachers in this 2026 paper. On policy distillation trains a student on its own rollouts, scored against a teacher. The reward hangs on the student's own likelihoods. For stability, most work stops the gradient through part of that score. The team asks if that stop is sound.

They frame the task with f-divergence between student and teacher. The math says the stop gradient habit biases both the reward estimate and its gradient, for general divergence choices. Their fix, OPD+, removes the bias and keeps the freedom to pick a divergence. The classic KL version is one special case. On math reasoning and tool use benchmarks, OPD+ beats the KL baseline, they report. The paper is a caution and a repair kit for a widely copied training trick.

Catalog record

CREATORHanyang Zhao, Haoxian Chen, Han Lin, Genta Indra Winata, David Yao, Wenpin Tang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01039

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OPD+ : Rethinking the Advantage Design for On-Policy Distillation Hanyang Zhao, Haoxian Chen, Han Lin, Genta Indra Winata, David Yao, Wenpin Tang Abstract On-policy distillation (OPD) is a widely used technique to transfer capabilities from capable teacher language models to the base student models, and can be formulated in a reinforcement learning style objective using student generated rollouts. Yet, despite the divergence reward being dependent on student model likelihood, existing works usually adopt a stop gradient design primarily for stability, which makes the resulting advantage estimation questionable. In this work, we provide a generic optimization framework based on f f -divergence between the student and teacher, and mathematically revisit whether such design space is valid. We prove that general stop-gradient operation would lead to biased estimates of the reward objective and corresponding gradient for general divergence functions. We propose OPD+, the corrected version of OPD that demonstrates improved performance over the baseline KL approach and also supports the choice of various f f -divergence. We validate our findings on mathematical reasoning and tool-use benchmarks. 1 Introduction On-policy distillation ( OPD ) 1 ; 8 is a powerful technique to let student models learn from capable teacher models through dense token-level supervisions. It has also become a crucial part of the post training recipes of modern large language models (LLMs), e.g. in Deepseek V4 3 , to learn from teacher models without performance degeneracy. In the design space of OPD 1 1 1 For OPD implementation, existing approaches either adopt a logit (top-k) distillation or the single token likelihood. We focus on the simpler second approach without the need of additional infrastructure modification as in 3 . , existing baselines like Tinker 8 suggest to adopt the KL divergence between student and teacher to achieve the best performance. To derive the loss function, common practice also disables the gradient propagation of log ratio difference reward between student and teacher primarily for numerical stability, motivated by e.g., target networks in DQN 9 . However, in this paper, we revisit these two key designs for OPD. Our contributions are mainly two-folded: Firstly, we provide the framework of f -divergence on policy distillation and derive the corresponding gradient and loss functions dependent on whether gradients on the reward function is enabled. Our framework enriches the design space of on policy distillation with various f -divergence; Secondly, we demonstrate that stopping gradient in reward not only yields mathematically incorrectly gradient estimates but also harms the performance for various f -divergence functions. The corrected algorithm, which we name as OPD+, yields strong empirical gains over the baseline in math reasoning and tool use scenarios, and challenges the common belief of the advantage of KL divergence over other statistical distances in on policy distillation. 2 Preliminaries We first revisit the preliminaries of LLMs and OPD as a Reinforcement Learning (RL) objective. More discussions on some other relevant works are provided in Appendix A . LLMs. Denote by y\sim p_{\theta}(\cdot\mid x) a full sentence generated by language model p_{\theta} given the context/prompt x . For auto-regressive LLMs, we use p_{\theta} for a sequential model parameterized by \theta , whose likelihood can be factorized as the product of the next-token distribution over the vocabulary, conditioning on the prompt x and already generated tokens y_{<n} , i.e., p_{\theta}(y\mid x)=\prod_{n=1}^{L_{y}}p_{\theta}(y_{n}\mid y_{<n},x) . OPD. For OPD, we set the behavior policy the same as the student policy/model, and we are interested in optimizing a per-token level difference between student p_{\theta} and teacher policy q : \mathbb{E}_{x\sim X,\ y\sim p_{\theta}(\cdot\mid x)}\sum_{n=1}^{L_{y}}D(p_{\theta}(\cdot\mid y_{<n},x),q(\cdot\mid y_{<n},x)), where D(p,q) denotes a chosen statistical divergence between two probability distributions, and L_{y} is the length of the generation y . When choosing D as the KL divergence, i.e., D(P,Q):=D_{\mathrm{KL}}(P\|Q)=\sum_{x}p(x)(\log p(x)-\log q(x)) , we have an equivalent loss objective which can be simply estimated by student sample rollouts: \mathbb{E}_{x\sim X,\ y\sim p_{\theta}(\cdot\mid x)}\sum_{n=1}^{L_{y}}\log\frac{p_{\theta}(y_{n}\mid y_{<n},x)}{q(y_{n}\mid y_{<n},x)}. In practice, the log ratio difference is often ‘stop grad’ed as a fixed reward. Using a zero discount factor for the advantage function yields the PPO-like loss \mathcal{J}_{\text{opd}}(\theta):= \mathbb{E}_{y\sim p_{\theta_{0}}(\cdot\mid x)}\sum_{n=1}^{L_{y}}\rho_{\theta}(y_{n}\mid y_{<n},x)\log\frac{p_{\theta_{0}}(y_{n}\mid y_{<n},x)}{q(y_{n}\mid y_{<n},x)}, where \rho_{\theta}=p_{\theta}(y_{n}\mid y_{<n},x)/p_{\theta_{0}}(y_{n}\mid y_{<n},x) is the importance sampling ratio. We provide a more detailed full derivation in Appendix B . 3 OPD+: Correcting the Advantage Estimation for On Policy Distillaion Note that in our derivation of OPD loss objective for KL divergence, it is assumed that the reward \log\frac{p_{\theta}(y_{n}\mid y_{<n},x)}{q(y_{n}\mid y_{<n},x)} is fixed, and hence independent of \theta . Yet this leads to a potential mismatch between the true target and our loss objective: the gradient can be biased due to the stop gradient operations. 3.1 Beyond KL: f -divergence optimization for on-policy distillation We first present a generic framework of f -divergence optimization between the student and teacher policy, and derive the true gradient and the loss objective compared to those obtained from stop gradient operations. f -divergence objective. Let f:(0,\infty)\to\mathbb{R} be a convex function satisfying f(1)=0 , and define the token-level density ratio as u_{\theta,n}:=q(y_{n}\,|\,y_{<n},x)/p_{\theta}(y_{n}\,|\,y_{<n},x). Consider the f -divergence objective sampling from the student model: \displaystyle J^{f}(\theta)=\mathbb{E}_{x\sim X}\mathbb{E}_{y\sim p_{\theta}(\cdot\mid x)}\sum_{n=1}^{L_{y}}f\!\left(u_{\theta,n}\right). (1) Equivalently, minimizing J^{f}(\theta) corresponds to minimizing the f -divergence between the student’s and the teacher’s next-token distribution along prefixes sampled from the student. Stop-gradient policy-gradient estimator. Define the reward as the negative divergence value: \displaystyle r_{\theta}^{f}(y_{\leq n},x):=-f\!\left(u_{\theta,n}\right), (2) As in the earlier reverse-KL case, suppose we stop-gradient the reward term (denote \theta_{-} for preventing gradient back propagation), the objective is: J_{\text{sg}}^{f}(\theta)=\mathbb{E}_{x\sim X,\ y\sim p_{\theta}(\cdot\mid x)}\sum_{n=1}^{L_{y}}\gamma^{n}r_{\theta_{-}}^{f}(y_{\leq n},x). By REINFORCE 18 , \nabla_{\theta}J_{\text{sg}}^{f}(\theta)= \mathbb{E}_{x\sim X,\ y\sim p_{\theta}(\cdot\mid x)}\left[\sum_{k=1}^{L_{y}}\nabla_{\theta}\log p_{\theta}(y_{k}\mid y_{<k},x)A^{f}_{k}\right], where the reward-to-go advantage is A_{k}^{f}:=\sum_{n=k}^{L_{y}}\gamma^{n-k}r_{\theta_{-}}^{f}(y_{\leq n},x) . Taking the discount factor \gamma=0 yields the immediate token-level reward A_{k}^{f}=r_{\theta_{-}}^{f}(y_{\leq k},x) . Thus, the resulting gradient estimator of J_{\text{sg}}^{f} is: \displaystyle\mathbb{E}_{y\sim p_{\theta}(\cdot\mid x)}\sum_{n=1}^{L_{y}}\nabla_{\theta}\log p_{\theta}(y_{n}\mid y_{<n},x)r_{\theta_{-}}^{f}(y_{\leq n},x). (3) 3.2 OPD+: Corrected Formula for f -divergence Optimization. We derive our earlier formula by applying stop gradient to the reward term. However, the true reward r_{\theta}^{f}(y_{\leq n},x)=-f\!\left(u_{\theta,n}\right) does depend on \theta . So if we differentiate the true objective, we get two terms – the score-function term and the direct reward-gradient term : \nabla_{\theta}\mathbb{E}_{y_{n}\sim p_{\theta}(\cdot\mid y_{<n},x)}\left[r_{\theta}^{f}(y_{\leq n},x)\right]= \displaystyle\mathbb{E}[r_{\theta}^{f}(y_{\leq n},x)\nabla_{\theta}\log p_{\theta}(y_{n}|y_{<n},x)+\nabla_{\theta}r_{\theta}^{f}(y_{\leq n},x)]. The direct reward-gradient term is computed as: \displaystyle\nabla_{\theta}r_{\theta}^{f}(y_{\leq n},x)=u_{\theta,n}f^{\prime}(u_{\theta,n})\nabla_{\theta}\log p_{\theta}(y_{n}\mid y_{<n},x). So the correct immediate token-level gradient is: \displaystyle\mathbb{E}_{y\sim p_{\theta}(\cdot\mid x)}\sum_{n=1}^{L_{y}}\nabla_{\theta}\log p_{\theta}(y_{n}\mid y_{<n},x)w_{f}(u_{\theta,n}), where w_{f}(u_{\theta,n})=-f(u_{\theta,n})+u_{\theta,n}f^{\prime}(u_{\theta,n}). Thus, for a general f -divergence, the correct advantage function is not simply r_{\theta}^{f}(y_{\leq n},x)=-f(u_{\theta,n}) : there is an additional term of u_{\theta,n}f^{\prime}(u_{\theta,n}) due to the reward dependency on policy parameter \theta , and we name our method as OPD+. To implement OPD+, only one line change of code is needed. What is special about Reverse KL? So is the exisiting formula of reverse KL misused when we do not apply stop grad? Theoretically, the answer is no . Reverse KL is a special instance of f -divergence with f(u)=-\log u . As a result, w_{f}(u)=f(u)-uf^{\prime}(u)=-\log u+1. The additional +1 term is a constant baseline, and hence vanishes under the score identity \mathbb{E}_{y_{n}\sim p_{\theta}}\left[\nabla_{\theta}\log p_{\theta}(y_{n}\mid y_{<n},x)\right]=0 . Thus, the direct reward-gradient correction for reverse KL reduces to a constant baseline, and is hidden in the stop-gradient policy optimization. For a general f , the term uf^{\prime}(u) is not a constant, so the stop gradient operation leads to a

Read the complete text