Tokenization is Sensitive to Language Variation

Anna Wegmann, Dong Nguyen, David Jurgens · 2025

Read Tokenization is Sensitive to Language Variation by Anna Wegmann, Dong Nguyen, David Jurgens (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15343.txt.

Description

Anna Wegmann, Dong Nguyen, and David Jurgens ask how word splitting affects language models in a 2025 study. Models first cut text into small units, and that step may treat rare spellings or local forms less well. The team test whether the best way to split words depends on the kind of task.

They train base models with a common splitting method and vary the source texts, the first split step, and the size of the word list. The first split step had the largest effect overall. They also offer a new way to judge a splitter by its effect on later tasks, which beats older scores.

Catalog record

CREATORAnna Wegmann, Dong Nguyen, David Jurgens
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15343

Browse the full library · About the Tower of Babel