TIP: Token Importance in On-Policy Distillation

Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard · 2026

Read TIP: Token Importance in On-Policy Distillation by Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14084.txt.

Description

Xu and colleagues ask which tokens teach the most in on-policy distillation. Informative tokens cluster in two regions: high student entropy, and low entropy where a sure yet wrong student diverges sharply from the teacher. Entropy-based sampling keeps half the tokens while matching or beating all-token training and cutting peak memory up to 47 percent. Training on under 10 percent of tokens, chosen for overconfidence, nearly matches full-token baselines.

Catalog record

CREATORYuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54686
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14084

Browse the full library · About the Tower of Babel