MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment
Sander Land · 2026
Read MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment by Sander Land (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27019.txt.
Description
Sander Land introduces MinGram, a minimalist tokenizer that splits text into pieces for language models. Tokenizers decide the units a model reads. The Unigram family keeps an editable token list with elegant scoring, but its training is heavy, needing suffix arrays and repeated pruning passes. MinGram keeps the list and slashes the training machinery.
A seed vocabulary from the rival BPE method starts the process. Hard expectation-maximisation on a shortest-token path refines it. One flat pruning step finishes the job. Token count is the main objective; the Unigram score only breaks ties. Across six languages, MinGram compressed text better than both BPE and standard Unigram. Its alignment with word morphology stayed competitive. In controlled model training, Unigram-family tokenizers beat BPE on efficiency, with MinGram among the best.
Catalog record
| CREATOR | Sander Land |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27019