Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching
Benjamin Minixhofer Ivan Vulić Edoardo M. Ponti · 2025
Read Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching by Benjamin Minixhofer Ivan Vulić Edoardo M. Ponti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20083.txt.
Description
Benjamin Minixhofer, Ivan Vulic, and Edoardo Ponti present a 2025 paper on moving knowledge between language models that split text in different ways. Teaching a small model from a large one works well, but past methods need both models to cut text into the same pieces. That need rules out most teacher-student pairs.
Their method matches the two models by how likely each finds a piece of text, even when the pieces differ. It is the first method to work across truly different text cutters, and it beat older methods in other cases too. Tests cover moving a model to a new cutter, sharpening a small model at math, and joining models for gains.
Catalog record
| CREATOR | Benjamin Minixhofer Ivan Vulić Edoardo M. Ponti |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20083