The African Language Tax

Olaoye Anthony Somide, DataLens Africa Research · CipherSense AI Technologies Ltd · 2026

Read The African Language Tax by Olaoye Anthony Somide, DataLens Africa Research · CipherSense AI Technologies Ltd (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24460.txt.

Introduction

Somide measures what speaking an African language costs in the age of per token billing. Commercial language models charge by the token. Tokenizers chop the same meaning into far more pieces in some languages than others. Speakers thus pay a structural penalty before any model runs. With DataLens Africa Research, Somide quantifies it for twenty African languages across five families and three scripts, using parallel corpora so language, not content, explains the difference.

Every African language measured carries a tokenisation premium over English, across eleven frontier and open tokenizers. The median premium sits near double on a leading commercial tokenizer. Far higher extremes appear for the N'Ko script. Ethiopic and N'Ko scripts suffer most, reaching seven to ninefold token counts. The penalty is nearly identical across corpora: a tokenizer property, not a data artefact. In deployment terms, the tax means severalfold inference costs and shrunken context.

Catalog record

CREATOROlaoye Anthony Somide, DataLens Africa Research · CipherSense AI Technologies Ltd
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE102 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24460

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The African Language Tax Olaoye Anthony Somide, DataLens Africa Research · CipherSense AI Technologies Ltd Abstract Abstract Commercial large language models bill, scale latency, and budget context per token. Yet tokenizers assign more subword tokens to the same meaning in some languages than in others, so speakers of languages with high token-fertility pay a structural penalty before a model is ever invoked. This penalty is documented for multilingual settings in general, but it has not been measured systematically for African languages at the level of enterprise deployment economics and cognitive context capacity. We measure it across 20 African languages spanning five language families and three scripts (Latin, Ge’ez/Ethiopic, N’Ko; 19 appear in the primary FLORES-200+ corpus, with Nigerian Pidgin measured via MAFAND-MT only), using parallel corpora so that the language effect is isolated from content. Across 11 frontier and open tokenizers on FLORES-200+, every African language carries a tokenization premium above English (median 1.88× on GPT-5 / o200k_base, up to 8.92× for N’Ko); the penalty is largest for Ethiopic and N’Ko scripts (reaching 7–9×) and is near-invariant across corpora (FLORES vs SIB-200 Pearson r = 0.9998). Translated into deployment terms, this results in up to 8.9× inference cost and an equivalent generation-latency multiplier (N’Ko vs English on GPT-5; 7.4× for Amharic), and as little as 11% of English’s effective context window. The best currently available tokenizer for African languages, Gemma 4, reduces the mean premium from 3.31× (cl100k_base) to 2.38×, but no tokenizer eliminates the penalty. We release an open measurement tool ( afri-fertility ), a public leaderboard, a results dataset, and mitigation guidance for African builders. The penalty falls hardest on the languages whose speakers can least afford it, a digital divide encoded directly into the subword vocabulary. Keywords tokenization, subword fertility, African languages, LLM inference cost, multilingual NLP, tokenizer fairness, low-resource languages, AI equity 1. Introduction 1.1 The Pre-Inference Cost Layer Before a large language model reasons about a single word of a prompt, it must first convert that text into tokens. Commercial LLM providers price their APIs per token, scale generation latency with the number of tokens produced, and bound the conversation by a fixed token context window. Tokenization is therefore not a neutral preprocessing step: it is the layer at which the economics of using a model are set. Two prompts that carry exactly the same meaning, but that tokenize into different numbers of tokens, cost different amounts to process, take different lengths of time to answer, and consume different fractions of the available context (effectively reducing the model’s “operational memory” and forcing earlier truncation of conversation history) before the model’s quality, accuracy, or capability enters the picture at all. This matters because tokenizers do not segment all languages equally. A subword vocabulary learned predominantly from English and other high-resource, web-abundant languages represents those languages compactly, while fragmenting others into many short pieces. The standard measure of this effect is fertility (the number of subword tokens a tokenizer emits per word) and the disparity across languages is large and well documented. Petrov et al. 2023 show that the same content can require up to roughly fifteen times as many tokens in one language as in another under current tokenizers, and Ahia et al. 2023 demonstrate that this length disparity translates directly into unequal API cost and utility across typologically diverse languages. The penalty is structural: it is fixed by the tokenizer’s vocabulary, applies on every request, and cannot be engineered away by the downstream user. 1.2 Why African Languages, and Why Now The cross-lingual tokenization gap has been documented in general multilingual settings and, most directly, for European languages by Ovcharov 2026 , who measures the “tokenizer tax” across 25 languages on parallel text, finding that English encodes at roughly 1.2 tokens per word while Greek and Maltese reach about 3.1×, with efficiency rankings near-invariant across text registers. That study is the closest methodological precedent to ours, but its largest penalties fall on European languages that still share the Latin or Greek/Cyrillic script families and substantial web presence. African languages have been left unmeasured at this resolution, even though they combine exactly the properties that should push the penalty well past that ceiling: non-Latin scripts (Geʿez/Ethiopic, Arabic/Ajami, N’Ko), agglutinative and tonal morphology, and far thinner representation in tokenizer training corpora. The gap is consequential: African builders deploying customer-service, clinical-triage, and agricultural advisory services must increasingly operate in local languages at production scale, in markets where compute is least affordable; and the African NLP community has invested heavily in accuracy benchmarks (IrokoBench Adelani et al. 2025 , AfroBench Ojo et al. 2023 ) while the pre-inference cost layer beneath them has gone unquantified. A model can score well on a task and still be uneconomic to run in the language it is written in. We address this with a parallel-corpus measurement. Because parallel corpora express the same meaning across every language, differences in token counts for a fixed tokenizer reflect the language and its script, not the content, isolating exactly the effect we want to measure. We translate these results into the terms a decision-maker uses: USD and local currency costs, latency multipliers, and the effective capacity of the model’s context window. 1.3 Contributions This paper makes three contributions: 1. The first comprehensive audit of the African-language tokenization premium , establishing a baseline for digital equity across 20 African languages and 11 current frontier and open tokenizers. We report fertility and an English-relative premium for each language–tokenizer pair across multiple script families, with corpus-level aggregation and bootstrap confidence intervals. 2. An enterprise cost model that converts fertility into USD and local-currency (NGN, ZAR, KES) cost, accounting for both token-fertility and the compounding effect of local currency volatility against USD-denominated API pricing. We further quantify generation-latency multipliers and context-window erosion, which reduces the “operational memory” of the model and forces African-language applications to handle shorter conversation histories for the same cost. We instantiate this model on three concrete deployment scenarios: a bank customer-service assistant, a clinical-triage line, and an SMS agricultural-advisory service. 3. Open artifacts : afri-fertility , a deterministic measurement tool with an African test suite built in; a public African Tokenization Tax Leaderboard; a released results dataset; and a one-page mitigation guide for African builders. Together these turn a known but abstract inequality into a measured, reproducible, and economically legible penalty for the African deployment context, with tooling for anyone to re-run and extend the measurement. 2. Background & Related Work 2.1 Subword Tokenization and Fertility Modern LLMs operate over subword vocabularies learned from a training corpus, typically by byte-pair encoding (BPE), unigram/SentencePiece, or a byte-level BPE variant such as the tiktoken family used by recent OpenAI models. The vocabulary fixes how any string is split: sequences frequent in the training corpus are merged into single tokens, while rarer sequences are left as many short pieces, down to individual bytes for scripts the vocabulary barely covers. The standard scalar summary of this behaviour is fertility (tokens emitted per word Rust et al. 2021 ), which Rust et al. link to downstream model quality and which we adopt here as the basis for an English-relative premium . Because a vocabulary is a frozen artifact of its training

Read the complete text