TIP: Token Importance in On-Policy Distillation
Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard · 2026
Read TIP: Token Importance in On-Policy Distillation by Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14084.txt.
Description
Xu and colleagues ask which tokens teach the most in on-policy distillation. Informative tokens cluster in two regions: high student entropy, and low entropy where a sure yet wrong student diverges sharply from the teacher. Entropy-based sampling keeps half the tokens while matching or beating all-token training and cutting peak memory up to 47 percent. Training on under 10 percent of tokens, chosen for overconfidence, nearly matches full-token baselines.
Catalog record
| CREATOR | Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54686 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14084