SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers
Iaroslav Chelombitko, Ekaterina Chelombitko, Aleksey Komissarov · 2026
Read SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers by Iaroslav Chelombitko, Ekaterina Chelombitko, Aleksey Komissarov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04469.txt.
Description
Iaroslav Chelombitko, Ekaterina Chelombitko, and Aleksey Komissarov build a toolkit that grades the scissors language models cut words with, in a 2026 arXiv paper. Subword tokenizers decide how a model sees text, and for morphologically rich Uralic languages nobody had clean morpheme dictionaries to judge whether the cuts fall at real word joints. Their SampoNLP toolkit makes those dictionaries without any corpus, scoring candidate morphemes by a self-referential atomicity test inspired by minimum description length.
With high-purity lexicons for Finnish, Hungarian, and Estonian in hand, they sweep tokenizer sizes from small to very large and track a combined score balancing morpheme coverage against over-splitting. The score's elbow points give the first evidence-based recommendations for vocabulary size in these languages. The study also quantifies how standard byte-pair encoding struggles with highly agglutinative words, and all resources are public.
Catalog record
| CREATOR | Iaroslav Chelombitko, Ekaterina Chelombitko, Aleksey Komissarov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-04469