Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N.M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, Mehrdad Farajtabar · 2026
Read Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why by Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N.M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, Mehrdad Farajtabar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10889.txt.
Description
Mohammadreza Armandpour and eight colleagues probe on-policy distillation, a way to train reasoning models with dense, token-by-token guidance from a teacher. It is often unclear when that signal helps and when it harms, which teacher to use, and whether the best choice changes from token to token.
Instead of costly training runs that hide token-level effects in aggregate scores, the team builds a training-free diagnostic. They define an ideal per-token gradient, estimate it with targeted rollouts, and score any distillation setup by how closely its gradient aligns with the ideal. Across self-distillation and outside-teacher settings, the score reveals where the signal helps and where it hurts. Trainers get a cheap lens for choosing teachers token by token.
Catalog record
| CREATOR | Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N.M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, Mehrdad Farajtabar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64879 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10889