X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation

Sharath Turuvekere Sreenivas*, Adithyakrishna Venkatesh Hanasoge*, Mingyu Yang*, Ali Taghibakhshi, Saurav Muralidharan, Ashwath Aithal, Pavlo Molchanov · 2026

Read X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation by Sharath Turuvekere Sreenivas*, Adithyakrishna Venkatesh Hanasoge*, Mingyu Yang*, Ali Taghibakhshi, Saurav Muralidharan, Ashwath Aithal, Pavlo Molchanov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21699.txt.

Description

Sharath Turuvekere Sreenivas and colleagues tackle distillation between language models whose vocabularies do not match. Existing logit methods fail in two ways. Rare tokens outside the matched set get suppressed in training, and strict one-to-one matching throws away near-equivalent tokens. Their X-Token method uses one shared projection matrix. One loss drops the partition entirely, while another merely softens it, and the method beats the prior best on Llama models taught by larger teachers.

Catalog record

CREATORSharath Turuvekere Sreenivas*, Adithyakrishna Venkatesh Hanasoge*, Mingyu Yang*, Ali Taghibakhshi, Saurav Muralidharan, Ashwath Aithal, Pavlo Molchanov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55100
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21699

Browse the full library · About the Tower of Babel