Conditional Unigram Tokenization with Parallel Data
Gianluca Vico, Jindřich Libovický · 2025
Read Conditional Unigram Tokenization with Parallel Data by Gianluca Vico, Jindřich Libovický (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07824.txt.
Description
Gianluca Vico and Jindrich Libovicky introduce a way to build tokenizers, the tools that chop text into pieces for language models, using parallel texts in two languages. Their paper came out in 2025. Standard tokenizers are trained on one language at a time, so the pieces rarely line up across languages.
Their method conditions the target language's piece choices on the source language's pieces, aiming for pieces that carry matching meaning. They tested four language pairs from different families and resource levels. The new tokenizer kept the same basic statistics as the standard one. It did not improve machine translation, but it consistently lowered perplexity, a measure of model surprise, in language modeling. The authors suspect a data-efficiency bottleneck and suggest other designs for cross-language tokenization.
Catalog record
| CREATOR | Gianluca Vico, Jindřich Libovický |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07824