Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, Yankai Lin · 2026
Read Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation by Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, Yankai Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12125.txt.
Description
On-policy distillation aligns a student with a teacher's outputs on the student's own trajectories, often beating off-policy distillation and reinforcement learning. Wenkai Yang and five co-authors first show theoretically that it is a special case of dense KL-constrained reinforcement learning with reward and regularization weighted equally.
Their generalized framework adds a flexible reference model and a reward scaling factor. Scaling reward above one, called reward extrapolation, consistently improves over standard distillation and can even let the student surpass its teachers when merging domain experts. In strong-to-weak distillation, choosing the teacher's pre-reinforcement-learning base model as reference further sharpens the reward signal, at the cost of needing that variant and extra compute.
Catalog record
| CREATOR | Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, Yankai Lin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-12125