MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment

Sander Land · 2026

Read MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment by Sander Land (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27019.txt.

Description

Sander Land introduces MinGram, a minimalist tokenizer that splits text into pieces for language models. Tokenizers decide the units a model reads. The Unigram family keeps an editable token list with elegant scoring, but its training is heavy, needing suffix arrays and repeated pruning passes. MinGram keeps the list and slashes the training machinery.

A seed vocabulary from the rival BPE method starts the process. Hard expectation-maximisation on a shortest-token path refines it. One flat pruning step finishes the job. Token count is the main objective; the Unigram score only breaks ties. Across six languages, MinGram compressed text better than both BPE and standard Unigram. Its alignment with word morphology stayed competitive. In controlled model training, Unigram-family tokenizers beat BPE on efficiency, with MinGram among the best.

Catalog record

CREATORSander Land
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27019

Browse the full library · About the Tower of Babel