OmniOPD : Logit-Free On-Policy Distillation via Speculative Verification
Yuhang Zhou, Lizhu Zhang, Yifan Wu, Mingyi Wang, Bo Peng, Jiayi Liu, Xiangjun Fan, Zhuokai Zhao · 2026
Read OmniOPD : Logit-Free On-Policy Distillation via Speculative Verification by Yuhang Zhou, Lizhu Zhang, Yifan Wu, Mingyi Wang, Bo Peng, Jiayi Liu, Xiangjun Fan, Zhuokai Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01476.txt.
Introduction
Zhou and co-authors distill teacher models without access to their logits. On-policy distillation trains a student on its own outputs. A stronger teacher gives dense token-level feedback. Standard versions need the teacher's logits directly. That excludes many capable proprietary models as teachers. Logit signals are also brittle and noisy. They hinge on a narrow overlap of plausible next tokens.
OmniOPD replaces logit matching with chunk-level supervision. Monte Carlo rollouts approximate the teacher's local preferences. A semantic similarity metric scores multi-token chunks. A peak-entropy scheduler audits the student at uncertain forks only. Bayesian priors and a KL anchor bound the variance. Math scores beat standard OPD by up to 28 point 64 percent.
Catalog record
| CREATOR | Yuhang Zhou, Lizhu Zhang, Yifan Wu, Mingyi Wang, Bo Peng, Jiayi Liu, Xiangjun Fan, Zhuokai Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01476
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniOPD : Logit-Free On-Policy Distillation via Speculative Verification Yuhang Zhou, Lizhu Zhang, Yifan Wu, Mingyi Wang, Bo Peng, Jiayi Liu, Xiangjun Fan, Zhuokai Zhao Abstract On-Policy Distillation (OPD) trains a student model on its own generative trajectories under dense token-level feedback from a stronger teacher, mitigating both the off-policy distribution shift of Supervised Fine-Tuning (SFT) and the sparse credit assignment of outcome-based Reinforcement Learning (RL). However, standard OPD faces two coupled limitations. First, it requires direct access to the teacher’s token-level logits, excluding a broad class of capable proprietary models from serving as teachers. Second, the token-level logit signal itself is brittle, depending on a narrow overlap of plausible next tokens between teacher and student, and prone to amplifying degenerate patterns such as repetition loops. In this paper, we introduce OmniOPD , a novel framework that addresses both limitations through a logit-free, chunk-level supervision signal. OmniOPD replaces deterministic logit matching with Monte Carlo rollouts that approximate the teacher’s local preferences through a continuous semantic similarity metric over multi-token chunks, and concentrates this supervision via a peak-entropy scheduler that audits the student only at its high-uncertainty reasoning forks. A Dirichlet-Multinomial Bayesian prior and a base-model KL anchor further bound the variance of discrete sampling and prevent policy collapse across unaudited tokens. Across mathematical reasoning and competitive programming benchmarks, OmniOPD outperforms SFT by up to +45.31% relative on math and +18.52% relative on code, and surpasses standard white-box OPD approach by up to +28.64% on mathematical reasoning, confirming that chunk-level semantic verification extracts a cleaner and more reliable learning signal than token-level logit matching, whose high information density is offset by significant noise and brittleness. Furthermore, when paired with stronger black-box teachers such as Claude-4.5-Haiku and Gemini-2.5-Flash, OmniOPD achieves an additional +9.54% relative on mathematical reasoning over its open-weight teacher counterpart, advancing the student past the performance of self-exploratory RL. † † affiliation: Meta AI † † date: August 10, 2026 † † correspondence: Yuhang Zhou and Zhuokai Zhao at {zyhang, zhuokai}@meta.com 1 Introduction On-Policy Distillation (OPD) has emerged as a central paradigm for post-training open-weight reasoning models ( Agarwal et al. 2024 ; Lu and Lab 2025 ; Song and Zheng 2026 ) . By having the student generate its own reasoning trajectories and querying a stronger teacher to score them, OPD combines the dense token-level supervision of Supervised Fine-Tuning (SFT) with the on-policy exploration of Reinforcement Learning (RL), mitigating both the distribution shift of imitating offline demonstrations ( Huan et al. 2025 ) and the sparse credit assignment of outcome-based RL methods such as GRPO ( Shao et al. 2024 ; Chen et al. 2025 ) . Despite these strengths, standard OPD inherits two coupled limitations that both stem from using the teacher’s token-level log-probabilities as the supervision signal. First, standard OPD approaches ( Agarwal et al. 2024 ; Lu and Lab 2025 ) requires direct read-out of the teacher’s exact next-token logits. This limitation is increasingly restrictive, as a broad class of capable proprietary models, including GPT ( Singh et al. 2025 ) , Claude ( Anthropic 2025b ) , Gemini ( Pichai et al. 2025 ) , and the wider commercial LLM ecosystem ( Aubakirova et al. 2026 ) , return only text, not logits, and are therefore excluded from the OPD pipeline entirely ( Song and Zheng 2026 ) . Second, even when full logit access is available, token-level matching is a surprisingly brittle target. Recent studies of OPD failure modes ( Li et al. 2026 ; Luo et al. 2026 ; Fu et al. 2026 ) show that the effective supervision signal lives in the narrow overlap between the teacher’s and student’s plausible next-token sets — a small set of plausible next tokens that concentrates the bulk of the gradient mass. This narrow region is fragile: it degrades when teacher and student reason in stylistically different ways, becomes uninformative or actively harmful on degenerate prefixes such as repetition loops, and is further disrupted by tokenizer mismatches between proprietary and open-weight model families ( Fu et al. 2026 ) . The high-dimensional logit distribution offers far less reliable supervision than its information density would suggest. This motivates our central question: Can dense on-policy distillation be performed without access to the teacher’s logits, and what alternative dense supervision signal could match, or surpass, logit-based matching? We answer both questions with OmniOPD , a logit-free framework that supervises the student through a multi-token, semantic-level signal estimated from text alone. Our approach builds on Monte Carlo estimation ( Kroese et al. 2014 ) . Rather than requiring deterministic logit distributions, OmniOPD approximates the teacher’s local preferences by sampling multiple discrete rollouts and estimating the teacher’s true probability density from their semantic similarity to the student’s generation. To make this computationally viable under restricted query budgets, we propose a peak-entropy chunk scheduling . Instead of querying the teacher at every token, OmniOPD selectively audits chunk of student tokens at the high-uncertainty decision boundaries in the student’s reasoning trajectory, which is the point where corrective supervision is most valuable. Finally, to counteract the high variance and zero-probability traps inherent in discrete sampling, we introduce a Bayesian smoothing step ( Robbins 1992 ) . By anchoring the empirical Monte Carlo estimates with the student’s base-model prior, OmniOPD extracts a variance-reduced gradient signal, while a separate base-model KL anchor on un-audited tokens prevents policy collapse across the trajectory’s unsupervised gaps. Across mathematical reasoning and competitive programming benchmarks, OmniOPD substantially outperforms SFT and, more strikingly, frequently outperforms white-box OPD ( Lu and Lab 2025 ) even when full teacher logits are available. This finding provides empirical support for our second motivation, where a coarser but cleaner signal yields stronger gradients than the high-bandwidth but noisy token-level distribution that white-box OPD optimizes. When paired with stronger black-box teachers such as Claude-4.5-haiku ( Anthropic 2025b ) and Gemini-2.5-flash ( Comanici et al. 2025 ) , OmniOPD further advances the student past the performance ceiling of self-exploratory GRPO ( Shao et al. 2024 ) on the same model. In summary, our main contributions are as follows: • We propose OmniOPD , a logit-free, chunk-level OPD framework that addresses both the white-box access requirement of standard OPD and the brittleness of its token-level supervision signal. • We introduce three components that make OmniOPD tractable and stable: a peak-entropy scheduler that concentrates teacher verification at the trajectory’s most uncertain reasoning forks, a Dirichlet-Multinomial Bayesian prior that stabilizes the discrete Monte Carlo estimates, and a trust-region KL penalty that prevents policy collapse across un-audited tokens. We provide formal guarantees on estimation error, gradient variance, and trajectory stability. • We demonstrate empirically that OmniOPD outperforms SFT by up to +45.31% relative on mathematical reasoning and +18.52% on competitive programming, exceeds white-box OPD by up to +28.64% on mathematical reasoning, and, when paired with stronger black-box teachers such as Claude-4.5-haiku and Gemini-2.5-flash, advances the student past the performance ceiling of self-exploratory GRPO. 2 Related Work Offline policy distillation of LLMs. The traditional paradigm for transferring complex reasoning capabilities from frontier models to smaller, more deployable models is offline knowledge distillation ( Hsieh et al. 2023 ; Shridhar et al. 2023 ; Kim and Rush 2016 ; Hao et al. 2026 ) . Early