Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation

Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, Yankai Lin · 2026

Read Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation by Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, Yankai Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12125.txt.

Description

On-policy distillation aligns a student with a teacher's outputs on the student's own trajectories, often beating off-policy distillation and reinforcement learning. Wenkai Yang and five co-authors first show theoretically that it is a special case of dense KL-constrained reinforcement learning with reward and regularization weighted equally.

Their generalized framework adds a flexible reference model and a reward scaling factor. Scaling reward above one, called reward extrapolation, consistently improves over standard distillation and can even let the student surpass its teachers when merging domain experts. In strong-to-weak distillation, choosing the teacher's pre-reinforcement-learning base model as reference further sharpens the reward signal, at the cost of needing that variant and extra compute.

Catalog record

CREATORWenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, Yankai Lin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-12125

Browse the full library · About the Tower of Babel