Causal Estimation of Tokenisation Bias
Pietro Lesci ✉ {}_{\text{{{\char 0\relax}}}} Clara Meister Thomas Hofmann Andreas Vlachos ✉ {}_{\text{{{\char 0\relax}}}} Tiago Pimentel ✉ {}_{\text{{{\char 0\relax}}}} · 2025
Read Causal Estimation of Tokenisation Bias by Pietro Lesci ✉ {}_{\text{{{\char 0\relax}}}} Clara Meister Thomas Hofmann Andreas Vlachos ✉ {}_{\text{{{\char 0\relax}}}} Tiago Pimentel ✉ {}_{\text{{{\char 0\relax}}}} (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03149.txt.
Introduction
Language models read text chopped into subword pieces, and the chopping itself can tilt what the model believes. Pietro Lesci and co-authors define this tokenisation bias and measure it as a causal effect. The question: how much does adding one subword to the vocabulary change the odds the model gives its characters?
Each model trains with only one tokenizer, so no direct comparison exists. The authors exploit the sharp cutoff where tokenizer builders stop adding ranked subwords. Pieces just inside and just outside the line are near twins, which allows a clean causal estimate. Across model sizes and tokenizers, the effect is real and sometimes huge: one piece's presence raised its characters' odds up to 17-fold in a small model. Tokenizer choice, they argue, is a first-class design decision.
Catalog record
| CREATOR | Pietro Lesci ✉ {}_{\text{{{\char 0\relax}}}} Clara Meister Thomas Hofmann Andreas Vlachos ✉ {}_{\text{{{\char 0\relax}}}} Tiago Pimentel ✉ {}_{\text{{{\char 0\relax}}}} |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-03149
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Causal Estimation of Tokenisation Bias Pietro Lesci ✉ {}_{\text{{{\char 0\relax}}}} Clara Meister Thomas Hofmann Andreas Vlachos ✉ {}_{\text{{{\char 0\relax}}}} Tiago Pimentel ✉ {}_{\text{{{\char 0\relax}}}} Abstract Modern language models are typically trained over subword sequences, but ultimately define probabilities over character-strings. Ideally, the choice of the tokeniser—which maps character-strings to subwords—should not affect the probability assigned to the underlying character-string; in practice, it does. We define this mismatch as tokenisation bias . In this work, we quantify one particular type of tokenisation bias: the effect of including or not a subword (e.g., ⟨ ℎ𝑒𝑙𝑙𝑜 ⟩ \mathit{\langle hello\rangle} ) in a tokeniser’s vocabulary on the probability a trained model assigns to the corresponding characters (i.e., “ ℎ𝑒𝑙𝑙𝑜 ” \mathit{\text{``}hello\text{''}} ). Estimating this effect is challenging because each model is trained with only one tokeniser. We address this by framing tokenisation bias as a causal effect and estimating it using the regression discontinuity design. Specifically, we exploit the fact that tokenisation algorithms rank subwords and add the first K K to a tokeniser’s vocabulary, where K K is an arbitrary cutoff point. As such, we can estimate a causal effect by comparing similar subwords around this cutoff. Experimentally, we find that tokenisation consistently affects models’ outputs across scales, vocabularies, and tokenisers. Notably, a subword’s presence in a small model’s vocabulary may increase its characters’ probability by up to 17 17 times, highlighting tokenisation as a key design choice in language modelling. 1 Introduction Language models (LMs) define probability distributions over character-strings (e.g., \mathit{\text{``}hello\text{''}} ), i.e., finite sequences of characters 1 1 1 We use characters as a generic term to refer to either raw bytes (e.g., in ASCII), Unicode symbols, or graphemes. from an alphabet. Directly modelling character-strings, however, can be inefficient, as it might require processing long sequences. To improve computational efficiency, modern LMs (e.g., Touvron et al. 2023 ) typically model subwords-strings instead (e.g., \mathit{\langle he,llo\rangle} ); these subword-strings are produced by a tokeniser , where each subword represents a sequence of characters. While in practice LMs thus output distributions over subword-strings, we can still map them back to distributions over character-strings ( Pimentel and Meister 2024 ; Phan et al. 2025 ) . Figure 1: Tokenisation bias. Consider subword {\color[rgb]{0.75,0,0.25}\mathit{\langle hello\rangle}} . If it is included in the vocabulary, then each occurrence of {\color[rgb]{0.3281,0.4961,0.9375}\mathit{\text{``}hello\text{''}}} in the training data is represented as a single subword; otherwise, it is split into two subwords, i.e., {\color[rgb]{0.75,0,0.25}\mathit{\langle he,llo\rangle}} . If two models are trained under these two settings, the difference in the probability they assign to {\color[rgb]{0.3281,0.4961,0.9375}\mathit{\text{``}hello\text{''}}} is the tokenisation bias we aim to estimate. Notably, a tokeniser’s behaviour is constrained by its vocabulary : a finite set of subwords which it may produce. Given a character-string, a tokeniser deterministically maps it to a sequence of subwords from its vocabulary. Different tokenisers (with different vocabularies) thus map character-strings to different subword-strings (e.g., map \mathit{\text{``}hello\text{''}} to either \mathit{\langle he,llo\rangle} or \mathit{\langle hello\rangle} ). While an ideal LM would assign the same probability to a string regardless of tokenisation ( section 3.2 ), in practice, tokenisation is an influential design choice for model performance ( Rust et al. 2021 ; Toraman et al. 2023 ; Ali et al. 2024 , inter alia ) . How tokenisation affects trained models, however, remains poorly understood. In this work, we first define the effect of a tokenisation choice on model behaviour as tokenisation bias . We then quantify one particular type of tokenisation bias: the effect of including or not a subword (e.g., \mathit{\langle hello\rangle} ) in a tokeniser’s vocabulary on the log-probability a trained model assigns to its characters (i.e., \mathit{\text{``}hello\text{''}} ). While tokenisation bias may be an intuitive concept, estimating it is not straightforward. We cannot simply compare a model’s log-probabilities on in- vs. out-of-vocabulary words (e.g., \mathit{\text{``}hello\text{''}} vs. \mathit{\text{``}appoggiatura\text{''}} ); as vocabularies are built based on, e.g., frequency, there are systematic differences between these groups. Further, we cannot compare log-probabilities on the same word under different tokenisations (e.g., \mathit{\langle he,llo\rangle} or \mathit{\langle hello\rangle} ); as the model only sees one of them during training. 2 2 2 A brute-force approach—e.g., training separate models under the different possible tokenisers—is (i) computationally impractical and (ii) estimates expected bias for a LM family, rather than the tokenisation bias in a specific model instance. To quantify tokenisation bias, we first frame it as a causal effect , asking: how would a word’s log-probability change if its subword was removed from the vocabulary (see fig. 1 )? This question requires comparing two values: (i) an observed value , the log-probability the model assigns the word in question; (ii) a counterfactual value , the log-probability the model would assign the word had it not been in the vocabulary. We estimate this causal effect by noting that tokenisers’ vocabularies are typically built incrementally. This effectively creates a ranking over subwords, which is arbitrarily cut off by a fixed vocabulary size: e.g., the first 32\text{true}\mathrm{k} subwords under this ranking are included in the vocabulary; later ones are excluded. Regression discontinuity (RD) design ( Thistlewaite and Campbell 1960 ) then gives us a principled way to estimate our causal effect: we can compare subwords near the cutoff—as they are bound to have similar features—using subwords before the cut-off to estimate our observed values, and the ones after the cutoff to input the counterfactual value. Experimentally, we show that tokenisation consistently affects LM performance across vocabulary sizes, model scales, and tokenisers. Text represented as a single subword receives more log-probability (on average) than if split into two. Further, this bias grows during training and in models under 100\text{true}\mathrm{M} parameters, it reaches 2.88 nats for BPE and 2.51 for WP . In 850\text{true}\mathrm{M} -parameter models, this bias is around 1 nat: a subword’s characters would thus be assigned roughly 2.7 times less probability if it were not in the tokeniser’s vocabulary. 2 Tokenisation Let {\color[rgb]{0.3281,0.4961,0.9375}c}\in{\color[rgb]{0.3281,0.4961,0.9375}\Sigma} be a character , where {\color[rgb]{0.3281,0.4961,0.9375}\Sigma} is an alphabet representing either a set of bytes, or the graphemes in a language (including whitespace and punctuation). Further, let \mathbf{\color[rgb]{0.3281,0.4961,0.9375}c}\in{\color[rgb]{0.3281,0.4961,0.9375}{\color[rgb]{0.3281,0.4961,0.9375}\Sigma}^{*}} be a character-string , i.e., a finite sequences of characters; we will represent these strings as \mathbf{\color[rgb]{0.3281,0.4961,0.9375}c}={\color[rgb]{0.3281,0.4961,0.9375}\text{``}{\color[rgb]{0.3281,0.4961,0.9375}c}_{{1}}{\color[rgb]{0.3281,0.4961,0.9375}c}_{{2}}\cdots{\color[rgb]{0.3281,0.4961,0.9375}c}_{{{\left|\mathbf{\color[rgb]{0.3281,0.4961,0.9375}c}\right|}}}\text{''}} . The goal of language modelling is to learn a language’s probability distribution over character-strings. We write these probabilities autoregressively as: \displaystyle p(\mathbf{\color[rgb]{0.3281,0.4961,0.9375}c})=p(\textsf{{eos}}\mid\mathbf{\color[rgb]{0.3281,0.4961,0.9375}c})\,\prod_{t=1}^{{\left|\mathbf{\color[rgb]{0.3281,0.4961,0.9375}c}\right|}}p({\color[rgb]{0.3281,0.4961,0.9375}c}_{{t}}\mid\mathbf{\color[rgb]{0.3281,0.4961,0.9375}c}_{{<t}}) (1) where \textsf{{eos}}\notin{\color[rgb]{0.3281,0.4961,0.9375}\Sigma} is a special symbol denoting the end of a string. 3 3 3 The eos symbol is required to define probability distributions over strings autoregressively ( Du et al. 2023 ) . Most modern LMs, however, do not model eq. 1 directly. Rather, they model distributions over subword-strings \mathbf{{\color[rgb]{0.75,0,0.25}v}}\!\in\!{\color[rgb]{0.75,0,0.25}{\color[rgb]{0.75,0,0.25}\mathcal{V}}^{*}} : \displaystyle p(\mathbf{{\color[rgb]{0.75,0,0.25}v}})=p(\textsf{{eos}}\mid\mathbf{{\color[rgb]{0.75,0,0.25}v}})\,\prod_{t=1}^{{\left|\mathbf{{\color[rgb]{0.75,0,0.25}v}}\right|}}p({\color[rgb]{0.75,0,0.25}v}_{t}\mid\mathbf{{\color[rgb]{0.75,0,0.25}v}}_{<t}) (2) In words, subword-strings are finite sequences of subwords {\color[rgb]{0.75,0,0.25}v}\in{\color[rgb]{0.75,0,0.25}\mathcal{V}} , where {\color[rgb]{0.75,0,0.25}\mathcal{V}} is a finite set typically called a vocabulary; we thus represent these strings as \mathbf{{\color[rgb]{0.75,0,0.25}v}}={\color[rgb]{0.75,0,0.25}\langle{\color[rgb]{0.75,0,0.25}v}_{1},...,{\color[rgb]{0.75,0,0.25}v}_{{\left|\mathbf{{\color[rgb]{0.75,0,0.25}v}}\right|}}\rangle} . As we explain later, subwords represent character spans, denoted here as \mathbf{\color[rgb]{0.3281,0.4961,0.9375}c}_{{\color[rgb]{0.75,0,0.25}v}}={\color[rgb]{0.75,0,0.25}v} . How do eqs. 1 and 2 connect? This is the job of a tokeniser, which we describe next. 2.1 Characters-strings \leftrightarrow Subword-strings Tokenisers map character-strings to subword-strings and back, and can be formally defined as a tuple \mathbbm{T}\mathrel{\stackrel{{\scriptstyle\textnormal{def}}}{{=}}}({\color[rgb]{0.75,0,0.25}\mathcal{V}},\tau,\rotatebox[origin]{180.0}{$\tau$}) . The first item in this tuple is a vocabulary : a finite set of character-spans {\color[rgb]{0.75,0,0.25}\mathcal{V}}\subset\color[rgb]{0.3281,0.4961,0.9375}{\color[rgb]{0.3281,0.4961,0.9375}\Sigma}^{+}