X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
Sharath Turuvekere Sreenivas*, Adithyakrishna Venkatesh Hanasoge*, Mingyu Yang*, Ali Taghibakhshi, Saurav Muralidharan, Ashwath Aithal, Pavlo Molchanov · 2026
Read X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation by Sharath Turuvekere Sreenivas*, Adithyakrishna Venkatesh Hanasoge*, Mingyu Yang*, Ali Taghibakhshi, Saurav Muralidharan, Ashwath Aithal, Pavlo Molchanov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21699.txt.
Description
Sharath Turuvekere Sreenivas and colleagues tackle distillation between language models whose vocabularies do not match. Existing logit methods fail in two ways. Rare tokens outside the matched set get suppressed in training, and strict one-to-one matching throws away near-equivalent tokens. Their X-Token method uses one shared projection matrix. One loss drops the partition entirely, while another merely softens it, and the method beats the prior best on Llama models taught by larger teachers.
Catalog record
| CREATOR | Sharath Turuvekere Sreenivas*, Adithyakrishna Venkatesh Hanasoge*, Mingyu Yang*, Ali Taghibakhshi, Saurav Muralidharan, Ashwath Aithal, Pavlo Molchanov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55100 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.21699