Effective Context in Transformers: An Analysis of Fragmentation and Tokenization
Amirmehdi J. Fesharaki, Mohammadamin Rami, Aslan Tchamkerten · 2026
Read Effective Context in Transformers: An Analysis of Fragmentation and Tokenization by Amirmehdi J. Fesharaki, Mohammadamin Rami, Aslan Tchamkerten (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13485.txt.
Introduction
Amirmehdi Jafari Fesharaki, Mohammadamin Rami and Aslan Tchamkerten analyse representation choice in finite-context transformer prediction. The same data can be bytes, characters or tokens, yet a fixed window may hide some information. First, smaller units can hurt prediction even when the window covers the relevant source history. Fragmentation, replacing each symbol by several smaller units, can strictly raise optimal log-loss. Second, greedy tokenisation groups symbols into larger units, so a short window acts like a longer window. They give a loss guarantee and a diagnostic for real tokenisers.
Catalog record
| CREATOR | Amirmehdi J. Fesharaki, Mohammadamin Rami, Aslan Tchamkerten |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77675 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13485
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Effective Context in Transformers: An Analysis of Fragmentation and Tokenization Amirmehdi J. Fesharaki, Mohammadamin Rami, Aslan Tchamkerten Abstract Transformers predict over a representation of a sequence. The same data can be written as bytes, characters, or subword tokens, and these representations may be lossless. Yet, under a fixed context window, they need not expose the same information to the model. This raises a basic question: how does the choice of representation change what a finite-context predictor can achieve? We study this question on Markov sources and uncover two complementary phenomena. First, we observe that moving to smaller representation units can hurt prediction even when the context window is enlarged to cover the relevant source history. To explain this, we introduce fragmentation: a lossless recoding that replaces each source symbol by several smaller units. We prove that fragmentation can strictly increase the optimal finite-context log-loss, showing that the gap is not merely an optimization or capacity issue, but can be intrinsic to the representation. This gives a theoretical account of the finite-context gap observed in byte- and character-level models such as ByT5 and CANINE relative to subword-tokenized models. Second, we study the opposite direction: greedy tokenization—BPE, WordPiece, and related methods—which groups source symbols into larger units. We show that tokenization can make a short token window behave like a longer source-context window, and we give a loss guarantee describing when this is achievable. The guarantee depends on how reliably token windows span the needed source history, together with the compression rate of the tokenizer. This also yields a simple diagnostic for real tokenizers: measuring how much source context a fixed token window reliably contains. Together, the two directions establish a finite-context information-theoretic framework for reasoning about representation choices in Transformers. 1 Introduction Sequence prediction is at the core of modern language modeling, and the Transformer architecture has proven remarkably effective for this task ( 23 , 6 , 2 , 22 ) . Most analyses of these models focus on architecture, optimization, scaling, or data. Yet there is a more basic choice made before training begins: how the sequence is represented. The same text can be represented as bytes, characters, words, or subword tokens. These encodings may be lossless, but a fixed model window over them can cover very different amounts of the original sequence. Thus, even lossless representations can lead to different finite-context prediction problems. This tension appears directly in modern language modeling. Token-free models such as ByT5 and CANINE avoid a fixed subword tokenizer by operating on bytes or characters, but they must process longer sequences and face a more expensive context problem ( 25 , 4 ) . Other approaches reduce this cost using latent subword blocks or multiscale byte-level modeling ( 21 , 26 ) . At the same time, most mainstream language models still use a fixed discrete tokenizer learned before model training, often based on subword or byte-level variants of BPE, WordPiece, or unigram language-model tokenization ( 19 , 18 , 6 , 10 ) . This creates a basic representation tradeoff: token-free models avoid committing to a fixed tokenizer, but they operate on longer sequences; tokenized models shorten the sequence, but they introduce a preprocessing choice which changes the prediction problem. To study this tradeoff cleanly, we focus on finite-context prediction on stationary finite-order Markov sources. In this setting, the relevant history is explicit: once the context reaches the Markov order, the optimal finite-context loss reaches the entropy rate. This lets us isolate how the representation changes what is available inside a fixed window. We ask: How does representation shape what a finite-context predictor can achieve? We show that this question has two sides. Moving to smaller representation units can make prediction harder, even when the recoding is lossless. Moving to larger token units can make the same finite window carry more information about the original source. The paper formalizes both effects and identifies the quantities that determine when they arise. In short, our contributions are: • We uncover a fragmentation effect motivated by Figure 1 : when a source is recoded into sub-symbol units, a transformer can converge to a strictly worse loss even after its window is enlarged to cover the relevant source history. This mirrors the prediction problem faced by byte- or character-level models. We formalize the effect as a lossless recoding into smaller units and prove that the resulting excess loss can be intrinsic to the representation. • We develop a finite-context analysis of greedy tokenization. We show that if a token window spans enough source symbols, then a token-level predictor can match the performance of a longer source-context predictor with no asymptotic loss. We then extend this to typical token windows, giving an achievability bound controlled by the lower tail of the source-span distribution and the tokenizer compression rate. • We identify the effective source context of a tokenizer as a measurable quantity for finite-context prediction. The theory lets us compare tokenizers by the guarantees they provide: for a fixed token window, which tokenizer can reliably match a source-level predictor with the largest possible context, up to small excess loss? This criterion depends on the lower tail of the token-window source-span distribution, not on average token length alone. 1.1 Related Work Beyond the standard tokenizer algorithms cited above, recent work studies tokenization as a modeling and efficiency choice. 8 connect tokenizer compression to downstream model performance, while Tao et al. ( 20 ) study how vocabulary size should scale with model size and compute. Other work highlights limitations of fixed subword tokenization, including robustness failures and sensitivity to token structure ( 3 ) , as well as broader effects on meaning, reasoning, and multilingual efficiency ( 9 , 27 ) . Our results are complementary: rather than evaluating tokenizer choices empirically, we characterize how a lossless representation changes the best achievable finite-context loss. The closest theoretical line studies Transformers and related sequence models on Markov sources ( 12 , 16 , 15 , 7 , 1 ) . These works use Markov data as a controlled setting to reveal what sequence models learn in context. In particular, ( 12 ) show that single-layer Transformers can behave like unigram predictors on higher-order Markov chains, while ( 16 ) show that tokenization can make a unigram-style predictor near-optimal under suitable learned dictionaries. Our work stays in the Transformer finite-context view, but shifts the question from model behavior to representation: how does a lossless recoding change the best loss achievable by any fixed-window predictor? Finally, our greedy tokenizer model is related to dictionary parsing and compression, including Lempel–Ziv and LZW methods ( 28 , 29 , 24 ) , and recent compression/coding views of tokenization ( 30 , 17 ) . Both settings trade vocabulary size for longer emitted units; we translate this tradeoff into an achievability guarantee based on the source-span distribution of token windows. 2 Background Throughout the paper, \{Y_{i}\}_{i\in\mathbb{Z}} is an ergodic stationary finite order- k Markov process defined over a finite alphabet \mathcal{Y} . Recall that a process \{Y_{i}\}_{i\in\mathbb{Z}} is k th-order Markov if, for every i , \Pr(Y_{i}\mid Y_{-\infty}^{i-1})=\Pr(Y_{i}\mid Y_{i-k}^{i-1})\quad\text{almost surely} . We shall also encounter processes that are not stationary but retain a periodic form of