Conditional Unigram Tokenization with Parallel Data

Gianluca Vico, Jindřich Libovický · 2025

Read Conditional Unigram Tokenization with Parallel Data by Gianluca Vico, Jindřich Libovický (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07824.txt.

Description

Gianluca Vico and Jindrich Libovicky introduce a way to build tokenizers, the tools that chop text into pieces for language models, using parallel texts in two languages. Their paper came out in 2025. Standard tokenizers are trained on one language at a time, so the pieces rarely line up across languages.

Their method conditions the target language's piece choices on the source language's pieces, aiming for pieces that carry matching meaning. They tested four language pairs from different families and resource levels. The new tokenizer kept the same basic statistics as the standard one. It did not improve machine translation, but it consistently lowered perplexity, a measure of model surprise, in language modeling. The authors suspect a data-efficiency bottleneck and suggest other designs for cross-language tokenization.

Catalog record

CREATORGianluca Vico, Jindřich Libovický
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-07824

Browse the full library · About the Tower of Babel