Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N.M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, Mehrdad Farajtabar · 2026

Read Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why by Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N.M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, Mehrdad Farajtabar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10889.txt.

Description

Mohammadreza Armandpour and eight colleagues probe on-policy distillation, a way to train reasoning models with dense, token-by-token guidance from a teacher. It is often unclear when that signal helps and when it harms, which teacher to use, and whether the best choice changes from token to token.

Instead of costly training runs that hide token-level effects in aggregate scores, the team builds a training-free diagnostic. They define an ideal per-token gradient, estimate it with targeted rollouts, and score any distillation setup by how closely its gradient aligns with the ideal. Across self-distillation and outside-teacher settings, the score reveals where the signal helps and where it hurts. Trainers get a cheap lens for choosing teachers token by token.

Catalog record

CREATORMohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N.M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, Mehrdad Farajtabar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64879
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10889

Browse the full library · About the Tower of Babel