The Tokenizer Tax Across 24 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty

Volodymyr Ovcharov · 2026

Read The Tokenizer Tax Across 24 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty by Volodymyr Ovcharov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24718.txt.

Introduction

Volodymyr Ovcharov maps the hidden tokenizer tax across 25 European tongues for ten base models. The tax spans 2.5 times from English to Greek and Maltese: Romance cheapest after English, then Germanic, Slavic, and Uralic and Baltic dearest. Ukrainian pays 15 to 18 percent more than cognate Slavic kin, showing thin training data. Rankings stay nearly identical across three text types. High-fertility tokenizers shred word-part boundaries instead of preserving them. Few-shot effects across languages trace to the model itself, not the language. All numbers ship as a public dataset.

Catalog record

CREATORVolodymyr Ovcharov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39278
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24718

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Tokenizer Tax Across 24 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty Volodymyr Ovcharov Abstract Tokenizer fertility – the number of tokens per word – imposes a hidden cost on non-English NLP. We measure fertility for ten foundation models across 25 European languages on parallel text, producing the first controlled tokenizer tax map for the continent. The tax spans 2.5 × \times from English (1.2 tokens/word) to Greek/Maltese ( ∼ {\sim} 3.1), following a clear hierarchy: Romance (1.5–1.7), Germanic (1.7–1.9), Slavic (2.2–2.5), Uralic/Baltic (2.7–3.0). Ukrainian (2.7) pays 15–18% more than cognate Slavic languages, reflecting underrepresentation in pre-training data. Fertility rankings are domain-invariant across three text registers ( ρ > 0.97 \rho>0.97 ). A subword analysis reveals that high-fertility tokenizers fragment morphological boundaries rather than preserving them. Cross-lingual few-shot evaluation on four Slavic languages shows that few-shot effects are model-intrinsic, not language-dependent. We release all measurements as a public dataset. Keywords: tokenizer fertility, language tax, cross-lingual evaluation, few-shot prompting, morphologically rich languages, Ukrainian NLP, Slavic languages 1 Introduction Every token costs money. When a tokenizer fragments Ukrainian text into 2.7 tokens per word but handles English at 1.2, the same API call costs more than twice as much – a hidden “tokenizer tax” that penalizes billions of speakers of morphologically rich languages. Petrov et al. 2023 and Ahia et al. 2023 documented this disparity, but existing measurements cover individual languages on heterogeneous text, making it impossible to disentangle language effects from content effects. This paper provides four things the literature lacks. First, a controlled cross-lingual fertility map : we measure ten foundation models on parallel text in 25 European languages, isolating the tokenizer’s contribution from content variation. Second, a domain invariance test : we compare fertility across three text registers – legal, news, and encyclopedic – to determine whether a single measurement suffices across applications. Third, a subword analysis that reveals how different tokenizers fragment morphologically complex Ukrainian words, explaining the fertility gap mechanistically. Fourth, a downstream consequence test : we evaluate few-shot classification across four Slavic languages to determine whether the tokenizer tax affects task performance. Ovcharov 2026 showed that tokenizer fertility varies by 1.6 \times across models on Ukrainian legal text and that few-shot prompting can degrade performance by up to 26 pp. We extend these findings along the cross-lingual axis and add three new model families (GPT-4o, Gemma 2, DeepSeek V3): 1. Domain invariance (Experiment 1): Fertility rankings are stable across three text registers – legal, news, and encyclopedic – with Spearman \rho>0.97 between domains. A single measurement predicts cost across all applications. 2. Cross-lingual fertility map (Experiment 2): Fertility across 25 EU languages on parallel text, revealing a 2.5 \times spread from English (1.23) to Greek/Maltese (3.1). Ukrainian (2.66) pays 15–18% more than cognate Slavic languages. 3. Subword analysis (Experiment 2b): Morphological decomposition analysis showing that high-fertility tokenizers split at arbitrary byte boundaries rather than morpheme boundaries, explaining the performance gap. 4. Cross-lingual few-shot (Experiment 3): Few-shot classification on SIB-200 across Ukrainian, Polish, Russian, and Czech. The effect is model-intrinsic: the same models that benefit in one language benefit in all four. 5. Linguistic competence (Experiment 4): ULP benchmark (347 grammar questions) tests whether fertility predicts grammatical accuracy. 2 Related Work 2.1 Few-Shot Learning and Its Limits Brown et al. 2020 established few-shot in-context learning as a core capability of large language models. Subsequent work has shown that few-shot performance depends on example selection ( Liu et al. 2022 ) , format consistency ( Min et al. 2022 ) , and label distribution ( Zhao et al. 2021 ) . Min et al. 2022 demonstrated that even random labels can improve performance, suggesting that demonstrations primarily specify task format rather than input–output mappings. However, these studies focus overwhelmingly on English. Lai et al. 2023 found significant performance variation across languages but did not systematically investigate few-shot effects. Ovcharov 2026 documented systematic few-shot degradation on Ukrainian legal text, showing that the effect depends on model architecture rather than input language. Our work extends the cross-lingual dimension: testing whether the same patterns hold across four Slavic languages and 25 European languages. 2.2 Tokenizer Fertility and Multilingual Fairness Petrov et al. 2023 formalized the “language tax” imposed by suboptimal tokenization, showing 2–15 \times token cost variation across languages. Ahia et al. 2023 demonstrated that API cost varies by an order of magnitude across languages for equivalent content. Rust et al. 2021 showed that monolingual performance correlates with pre-training data proportion, with fertility as a proxy. These studies examine general-domain text or single languages. Niklaus et al. 2024 introduced LEXTREME, a multi-lingual legal benchmark, but did not measure tokenizer fertility or few-shot effects. Alternative approaches bypass the tokenizer entirely: CANINE ( Clark et al. 2022 ) operates on Unicode code points and ByT5 ( Xue et al. 2022 ) on raw bytes, eliminating the fertility disparity at the cost of longer sequences. Zheng et al. 2021 showed that vocabulary reallocation can reduce fertility by 15–30% for underserved languages. We provide the first controlled cross-lingual comparison of six tokenizers across 25 languages on parallel text, isolating the tokenizer’s contribution from content variation, and test domain invariance across three text registers. 2.3 Morphologically Rich Languages in NLP Ukrainian, Polish, Russian, and Czech are Slavic languages with rich inflectional morphology: 7 cases, grammatical gender, and extensive verb conjugation. This morphological complexity interacts with subword tokenizers, potentially producing more fragmented representations that interfere with in-context pattern matching. Conneau et al. 2020 showed that multilingual model performance correlates with pre-training data volume, with low-resource languages suffering disproportionately. Chaplynskyi 2023 showed consistent underperformance of multilingual models on Ukrainian compared to English, a finding our cross-lingual experiments extend to the few-shot setting. 3 Methodology 3.1 Models We extend the seven API models from Ovcharov 2026 with three additional models whose tokenizers are publicly available, bringing the total to ten (Table 2 ). The original seven are accessed via the AWS Bedrock API; the three additions are measured using their HuggingFace tokenizers and OpenAI’s tiktoken library. We validate the local tokenizer approach by comparing it against API measurements on Ukrainian news text for the three models available in both settings (Table 1 ). Local and API fertility values agree within 1.8% (mean absolute difference 0.030 tokens/word), confirming that local tokenizer measurement is a reliable and cost-free substitute. Table 1 : Validation: API-reported vs. local tokenizer fertility on Ukrainian SIB-200 news text. \Delta = absolute difference. Model API Local \Delta % Llama 4 Maverick 2.197 2.158 0.039 1.8 Llama 3.3 70B 2.333 2.317 0.016 0.7 Qwen3 32B 3.583 3.617 0.034 0.9 Mean 0.030 1.1 Table 2 : Models evaluated. Original seven via AWS Bedrock (April–May 2026); three additions (†) via local tokenizer. MoE = mixture of experts; active = parameters per forward pass. Model Provider Tokenizer Vocab Llama 4 Maverick Meta SentencePiece 200K Llama 3.3 70B Meta SentencePiece 128K Mistral Large 3 Mistral SentencePiece 32K Nemotron Super 3 NVIDIA SentencePiece 128K Nova Pro Amazon Proprietary — Qwen3 32B Qwen tiktoken-derived 151K GPT-4o† OpenAI tiktoken (o200k) 200K Gemma 2 27B† Google SentencePiece 256K DeepSeek V3† DeepSeek Custom BPE 128K 3.2 Datasets SIB-200 ( Adelani et al. 2024 ) is a topic classification benchmark covering 205 languages with

Read the complete text