Draft-OPD: On-Policy Distillation for Speculative Draft Models
Haodi Lei 1,2 Yafu Li 2,4,† Haoran Zhang 1,2 Shunkai Zhang 2,5 Qianjia Cheng 2,6 Xiaoye Qu 2 Ganqu Cui 2 Bowen Zhou 2,3 Ning Ding 3,2,† Yun Luo 2,† Yu Cheng 4,2 1 Shanghai Jiao Tong University 2 Shanghai AI Laboratory 3 Tsinghua University 4 The Chinese University of Hong Kong 5 Peking University 6 Zhejiang University · 2026
Read Draft-OPD: On-Policy Distillation for Speculative Draft Models by Haodi Lei 1,2 Yafu Li 2,4,† Haoran Zhang 1,2 Shunkai Zhang 2,5 Qianjia Cheng 2,6 Xiaoye Qu 2 Ganqu Cui 2 Bowen Zhou 2,3 Ning Ding 3,2,† Yun Luo 2,† Yu Cheng 4,2 1 Shanghai Jiao Tong University 2 Shanghai AI Laboratory 3 Tsinghua University 4 The Chinese University of Hong Kong 5 Peking University 6 Zhejiang University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29343.txt.
Introduction
Haodi Lei and colleagues present Draft-OPD, an on-policy distillation method. A target model pairs with a draft model checked at once, but fine-tuning on target-made trajectories plateaus: accept rate stops improving. Cause: a mismatch. The drafter learns from fixed runs but is judged on its own blocks. Draft-OPD uses target-aided rollout and replays drafting from check-exposed error positions, so the drafter learns from feedback on accepted and rejected drafts. Tests show more than five times lossless speedup.
Catalog record
| CREATOR | Haodi Lei 1,2 Yafu Li 2,4,† Haoran Zhang 1,2 Shunkai Zhang 2,5 Qianjia Cheng 2,6 Xiaoye Qu 2 Ganqu Cui 2 Bowen Zhou 2,3 Ning Ding 3,2,† Yun Luo 2,† Yu Cheng 4,2 1 Shanghai Jiao Tong University 2 Shanghai AI Laboratory 3 Tsinghua University 4 The Chinese University of Hong Kong 5 Peking University 6 Zhejiang University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41547 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29343
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Draft-OPD: On-Policy Distillation for Speculative Draft Models Haodi Lei 1,2 Yafu Li 2,4,† Haoran Zhang 1,2 Shunkai Zhang 2,5 Qianjia Cheng 2,6 Xiaoye Qu 2 Ganqu Cui 2 Bowen Zhou 2,3 Ning Ding 3,2,† Yun Luo 2,† Yu Cheng 4,2 1 Shanghai Jiao Tong University 2 Shanghai AI Laboratory 3 Tsinghua University 4 The Chinese University of Hong Kong 5 Peking University 6 Zhejiang University Abstract Speculative decoding accelerates large language model inference by pairing a target model with a lightweight draft model whose proposed tokens are verified in parallel. A common way to build draft models, like EAGLE3 or DFlash is supervised fine-tuning (SFT) on target-generated trajectories. However, we observe that SFT quickly plateaus: the draft model’s acceptance length on test data stops improving. The reason is an offline-to-inference mismatch: In SFT, the drafter learns from fixed target-generated trajectories, whereas during speculative decoding it is evaluated on blocks proposed under its own policy. This motivates on-policy distillation (OPD), where the target model supervises the drafter on draft-induced states. Yet OPD remains difficult for draft models, as they cannot reliably roll out complete sequences independently, whereas target-assisted generation makes the collected sequences follow the target distribution and thus eliminates the on-policy signal. We therefore propose Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions. This allows the drafter to learn from target feedback on both accepted and rejected proposals, focusing training on the draft-induced errors that limit speculative acceptance. Experiments show that Draft-OPD achieves over 5 × 5\times lossless acceleration for thinking models across diverse tasks, improving over EAGLE-3 and DFlash by 23% and 13%. 2 2 footnotetext: Corresponding authors. Project Page Code Models 1 Introduction Figure 1: Accepted length during draft-model training. After an initial SFT warm-up, continued offline SFT quickly plateaus, and simply applying OPD data for SFT can even reduce accepted length. In contrast, Draft-OPD continues to improve accepted length by online training. Recent advances in large language models (LLMs) have enabled strong performance across reasoning, coding, and general assistant tasks, but their growing model sizes and longer generations substantially increase inference cost ( Yang et al. 2025 ; DeepSeek-AI et al. 2025 ) . Speculative decoding (SD) mitigates this cost by using a lightweight draft model to propose tokens that are verified in parallel by a larger target model, preserving the target model’s output distribution ( Leviathan et al. 2023 ; Chen et al. 2023 ) . In SD, the speedup depends heavily on the quality of the draft model: when the drafter closely matches the target model, longer token spans can be accepted, reducing the number of expensive target-model decoding steps. Recent mainstream methods, such as EAGLE-3 and DFlash, have achieved strong acceleration by training lightweight draft models on target-generated trajectories. ( Li et al. 2025 ; Chen et al. 2026 ) . However, we find that offline supervised fine-tuning quickly reaches its limit for draft-model training. Figure 1 shows a representative training curve. After an initial SFT warm-up, continuing SFT does not improve the draft model’s acceptance length on test data; instead, the accepted length fluctuates around a fixed plateau. Moreover, continuing SFT on the data used for OPD can even reduce the accepted length. This suggests that the key limitation does not lie in offline training compute. Instead, the plateau points to a mismatch between the states used for training and the states that determine speculative acceptance. In SFT, the drafter learns from fixed target-generated trajectories, so every prefix is produced by the target model. During speculative decoding, however, the target model verifies token blocks proposed by the drafter itself. The accepted length is therefore determined by draft-induced inference states, rather than only by the static states seen in offline target trajectories. This offline-to-inference mismatch motivates on-policy distillation (OPD), where the target model supervises states induced by the current draft policy ( Agarwal et al. 2024 ) . Directly applying OPD to draft models, however, is not straightforward. Standard OPD assumes that the student can roll out full sequences under its own policy. But mainstream draft models such as EAGLE-style or DFlash-style drafters are designed to propose short token spans under target-model guidance, rather than to act as standalone autoregressive generators. As a result, draft-only rollouts can easily become repetitive or low quality. Using target-assisted rollouts can produce usable sequences, but the verified continuation follows the target distribution: target model discards the incorrect tokens proposed by draft model, even though these errors are the most valuable training signals. Consequently, draft model still learns on trajectories generated by target model rather than on its own inference-time states. To address this gap, we propose Draft-OPD , an on-policy distillation framework designed for speculative draft models. Draft-OPD uses target-assisted rollout to maintain stable continuations, and records error positions exposed by speculative verification. It then replays drafting from these positions and asks the target model to score the same draft-generated prefixes. This makes it possible to train on states where the draft model actually acted, including both accepted proposals and rejected proposals. Finally, Draft-OPD uses an acceptance-aware distillation objective that treats these two token groups differently: accepted tokens reinforce reliable agreement with the target model, while rejected tokens focus learning on draft-induced errors that limit speculative acceptance. In summary, our contributions are: • We identify a key limitation of offline SFT for draft models: SFT quickly plateaus because the drafter is trained on fixed target trajectories but evaluated on blocks induced by its own policy during speculative decoding. • We explain why standard OPD does not directly apply to draft models: draft-only rollouts are unstable, while target-assisted rollouts remove the on-policy signal. • We propose Draft-OPD , an on-policy distillation framework with error-position replay, making it feasible to efficiently post-train draft models on verification-time errors. • Experiments show that Draft-OPD achieves over 5\times lossless acceleration for thinking models and improves over EAGLE-3 and DFlash by 23\% and 13\% under matched FLOPs. 2 Related Work 2.1 Speculative Decoding Speculative decoding accelerates autoregressive inference by using a lightweight draft model to propose multiple future tokens and a larger target model to verify them in parallel while preserving the target distribution ( Leviathan et al. 2023 ; Chen et al. 2023 ) . Recent work further improves speculative decoding by leveraging feature-level context from the frozen target model, as in EAGLE ( Li et al. 2024 ; Li et al. 2025 ) , or by designing stronger block-level draft architectures such as DFlash ( Chen et al. 2026 ) . These methods substantially improve decoding efficiency, but they still train draft models with offline SFT on target-generated trajectories, which limits further improvements in draft-model acceptance length. In contrast, our work studies how to move draft models beyond this offline training recipe. 2.2 Draft Model Distillation Because speculative speedup depends on how closely the draft distribution matches the target distribution, several methods train draft models through distillation. DistillSpec aligns a compact draft model with the target model and highlights the importance of on-policy data and task-specific divergence choices ( Zhou et al. 2024 ) ; temperature-centric distillation studies further show that matching training and decoding configurations can improve speculative decoding under challenging sampling settings