The Impact of Token Granularity on the Predictive Power of Language Model Surprisal
Byung-Doh Oh, William Schuler · 2024
Read The Impact of Token Granularity on the Predictive Power of Language Model Surprisal by Byung-Doh Oh, William Schuler (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11940.txt.
Introduction
Byung-Doh Oh and William Schuler test how token size shapes a language model's power to mimic human reading in a 2024 paper. Models split words into tokens, and the split size is usually treated as a mere engineering choice. Surprisal, the model's measure of how unexpected each word is, is used to predict reading effort.
They trained models with varied token grain and checked them against reading times and classic garden-path sentences. A mid-size vocabulary of 8,000 tokens gave the most human-like surprisal on natural text. Coarser tokens made models more sensitive to garden-path traps than past work found. The grain choice, they argue, belongs in any reading model.
Catalog record
| CREATOR | Byung-Doh Oh, William Schuler |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11940
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Impact of Token Granularity on the Predictive Power of Language Model Surprisal Byung-Doh Oh, William Schuler Abstract Word-by-word language model surprisal is often used to model the incremental processing of human readers, which raises questions about how various choices in language modeling influence its predictive power. One factor that has been overlooked in cognitive modeling is the granularity of subword tokens, which explicitly encodes information about word length and frequency, and ultimately influences the quality of vector representations that are learned. This paper presents experiments that manipulate the token granularity and evaluate its impact on the ability of surprisal to account for processing difficulty of naturalistic text and garden-path constructions. Experiments with naturalistic reading times reveal a substantial influence of token granularity on surprisal, with tokens defined by a vocabulary size of 8,000 resulting in surprisal that is most predictive. In contrast, on garden-path constructions, language models trained on coarser-grained tokens generally assigned higher surprisal to critical regions, suggesting a greater sensitivity to garden-path effects than previously reported. Taken together, these results suggest a large role of token granularity on the quality of language model surprisal for cognitive modeling. 1 Introduction In cognitive modeling, word-by-word surprisal is often used as a predictor of processing difficulty, under a theoretical framework that emphasizes the predictive aspect of real-time language processing ( Hale, 2001 ; Levy, 2008 ) . In recent years, neural network-based language models (LMs) have been used to calculate and evaluate surprisal against human reading times ( Wilcox et al., 2020 ; Merkx and Frank, 2021 ) , which has opened possibilities for refining them as computational models of language processing and using them to study how predictive processing interacts with other cognitive processes. Therefore, a core question in this area is how various aspects of language modeling such as the LMs’ architecture or training data influence the learned probabilities and their alignment to human-like processing difficulty. ‘If you were to journey’ Finer granularity, more character-like ( |V|=256 ) ␣ I f ␣ y o u ␣w er e ␣to ␣ j o ur n e y Coarser granularity, more word-like ( |V|=128000 ) ␣If ␣you ␣were ␣to ␣journey Figure 1: Smaller subword vocabulary sizes result in longer sequences of finer-granularity tokens that are more character-like (top), and larger vocabulary sizes result in shorter sequences of coarser-granularity tokens that are more word-like (bottom). One such variable that has been overlooked in cognitive modeling is the granularity of the tokens over which LMs are trained to define probability distributions. To allow LMs to flexibly handle unseen word forms and keep the vocabulary size manageable, it has become a standard practice in language modeling to use ‘subword’ tokenizers ( Sennrich et al., 2016 ; Kudo, 2018 , e.g.) . Such tokenizers are often trained on corpus statistics such that their vocabularies contain a fixed number of frequent sequences (which may or may not correspond to words) as independent tokens. As a consequence, less frequent word forms are split into multiple subword tokens during both training and inference. This suggests that there are at least two ways in which token granularity will influence the quality of LM surprisal in the context of cognitive modeling. The first is through the different initial biases about word probabilities that the different levels of token granularity embody. For example, a tokenizer with a very fine token granularity (Figure 1 , top) tokenizes the word journey into seven tokens, and therefore a uniform initial distribution over tokens predicts its probability to be six magnitudes lower than that of to . In contrast, a tokenizer with coarser granularity (Figure 1 , bottom) keeps both journey and to intact, and a uniform initial distribution over these tokens predicts their probabilities to be equal. As such subword tokenization is informed by word length and frequency, which are variables that are well known to influence real-time processing ( Barton et al., 2014 ; Just and Carpenter, 1980 ) , some tokenization schemes are more likely to lead to word-level surprisal that is more predictive of processing difficulty. The second way in which token granularity will impact LM surprisal is through the quality of the token representations that are learned. That is, LMs trained on coarse-grained tokens that are more word-like will learn token representations that align closer to lexical co-occurrence statistics like those from earlier neural network LMs ( Mikolov et al., 2013 ) . In contrast, fine-grained tokens will cause words to be split across multiple vector representations, which may make learning word-to-word associations more challenging. For instance, the LM would need to attend to seven separate vector representations for journey to predict travel later in the sequence. Motivated by these observations, this work presents experiments that manipulate the token granularity of LMs and evaluate its influence on the ability of surprisal to account for processing difficulty of both naturalistic text and garden-path constructions. First, regression experiments across multiple reading time corpora reveal a strong influence of token granularity on the predictive power of surprisal prior to any LM training, with tokens defined by a vocabulary size of 4,000 resulting in surprisal that is most predictive. This initial bias appears to persist throughout training in smaller LMs, and eventually yield surprisal that is more predictive than that of larger LMs. In contrast, LMs trained with tokens of coarser granularity generally assigned higher surprisal to critical regions of garden-path constructions, suggesting a greater sensitivity to garden-path effects. These results suggest a large role of token granularity on the quality of LM surprisal for cognitive modeling, with different levels of granularity being more appropriate for modeling broad-coverage comprehension and garden-path effects. 2 Related Work The choice of subword tokenizers has mostly been studied in terms of performance on natural language processing tasks ( Bostrom and Durrett, 2020 , e.g.) , with more recent work aiming to provide explanations for their downstream impact ( Zouhar et al., 2023 ; Schmidt et al., 2024 ) or the discrepancy in performance across languages ( Arnett and Bergen, 2025 ) . Other work proposes methods for mitigating ‘quirks’ introduced by subword tokenization, such as LMs’ sensitivity to misspelling ( Vieira et al., 2024 ) or ‘under-trained’ tokens with poor quality representations ( Land and Bartolo, 2024 ) . In psycholinguistic modeling, Oh et al. (2021) incorporate a character model to an incremental left-corner parser and show improvements in surprisal’s fit to naturalistic reading times. Nair and Resnik (2023) compare a subword tokenization scheme informed by concatenative morphology against a word-level and a byte-pair encoding scheme and report its impact on LM surprisal’s predictions of reading times. Giulianelli et al. (2024) apply Vieira et al. ’s ( 2024 ) algorithm to derive character-level probabilities from GPT-2’s ( Radford et al., 2019 ) token-level probabilities and demonstrate the potential of using them to model the processing difficulty of specific focal areas (e.g. the first three characters of a word) more flexibly. 3 Experiment 1: Impact on Fit to Naturalistic Reading Times The first experiment evaluates the effect of token granularity on the fit of LM surprisal to naturalistic reading times. LM surprisal is first evaluated prior to any training to examine the extent to which word-level surprisal determined