Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching

Benjamin Minixhofer Ivan Vulić Edoardo M. Ponti · 2025

Read Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching by Benjamin Minixhofer Ivan Vulić Edoardo M. Ponti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20083.txt.

Description

Benjamin Minixhofer, Ivan Vulic, and Edoardo Ponti present a 2025 paper on moving knowledge between language models that split text in different ways. Teaching a small model from a large one works well, but past methods need both models to cut text into the same pieces. That need rules out most teacher-student pairs.

Their method matches the two models by how likely each finds a piece of text, even when the pieces differ. It is the first method to work across truly different text cutters, and it beat older methods in other cases too. Tests cover moving a model to a new cutter, sharpening a small model at math, and joining models for gains.

Catalog record

CREATORBenjamin Minixhofer Ivan Vulić Edoardo M. Ponti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20083

Browse the full library · About the Tower of Babel