How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor?
Christian Clark, Byung-Doh Oh, William Schuler · 2025
Read How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor? by Christian Clark, Byung-Doh Oh, William Schuler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22209.txt.
Introduction
Christian Clark, Byung-Doh Oh, and William Schuler tested in 2025 a shortcut that reading science takes when it estimates how unsure a reader is before a word arrives. That uncertainty, contextual entropy, predicts processing effort, and it complements the better-known surprise measure. For convenience, entropy is usually read from the model's odds for the word's first token only.
The shortcut understates and can distort true word entropy, since words span varying token counts. The team built Monte Carlo estimates of whole-word entropy and reran reading time regressions with both versions. The two predictors parted ways in the results. Readers get a methods caution with data: an approximation borrowed for ease may answer a different question than the one reading science is asking.
Catalog record
| CREATOR | Christian Clark, Byung-Doh Oh, William Schuler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22209
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor? Christian Clark, Byung-Doh Oh, William Schuler Abstract Contextual entropy is a psycholinguistic measure capturing the anticipated difficulty of processing a word just before it is encountered. Recent studies have tested for entropy-related effects as a potential complement to well-known effects from surprisal. For convenience, entropy is typically estimated based on a language model’s probability distribution over a word’s first subword token. However, this approximation results in underestimation and potential distortion of true word entropy. To address this, we generate Monte Carlo (MC) estimates of word entropy that allow words to span a variable number of tokens. Regression experiments on reading times show divergent results between first-token and MC word entropy, suggesting a need for caution in using first-token approximations of contextual entropy. 1 Introduction Recent studies of human sentence processing have explored potential psycholinguistic effects from the contextual entropy of the current word being processed Cevoli et al. 2022 ; Pimentel et al. 2023 ; Wilcox et al. 2023 ; Giulianelli et al. 2024 . Contextual entropy is an information-theoretic measure quantifying a reader’s level of uncertainty about the current word based on the preceding context; it is computed purely based on the conditional probability distribution of the current word without reference to the word’s identity. As such, it contrasts with surprisal, another commonly used psycholinguistic measure whose effects can be understood as integration costs for an already observed word Cevoli et al. 2022 ; Pimentel et al. 2023 . Contextual entropy is commonly estimated using a language model (LM) like GPT2 Radford et al. 2019 . However, because words can span multiple subword tokens in an LM’s vocabulary—making a full summation over their probability distribution intractable—entropy is typically calculated over the probability distribution of each word’s first token instead. This practice results in a systematic underprediction of true word entropy Pimentel et al. 2023 , which is magnified in contexts in which multi-token words are probable. This mismatch between the phenomenon of interest (word-level predictive processing) and the metric of choice (token-level entropy) may lead to challenges in drawing psycholinguistic conclusions from experimental results. To address this issue, we calculate LM-based entropy estimates using a technique based on Monte Carlo ( Metropolis and Ulam 1949 , MC;) sampling that allows words to span multiple tokens. While this method still only provides an approximation of true word entropy, the MC method produces unbiased estimates ( Giulianelli et al. 2024 ) —unlike first-token approximation—and it can produce higher-quality (lower-variance) estimates if the number of samples is increased. To evaluate the difference between first-token entropy and MC word entropy, both measures are evaluated in a set of regression experiments using naturalistic self-paced reading and eye-tracking data. Results show contrasting predictions from the two measures, with the MC word entropy especially making stronger estimates of self-paced reading times. The gap between the two entropy estimates suggests that first-token entropy may not provide a reliable approximation of true word entropy for psycholinguistic modeling. 1 1 1 Code for first-token and Monte Carlo entropy estimation is available at https://github.com/christian-clark/word-entropy . 2 Related Work Much of the interest in information-theoretic predictors of human sentence processing traces back to surprisal theory ( Hale 2001 ; Levy 2008 ) , which posits a direct link between processing difficulty and a word’s surprisal (negative log probability). Robust effects from surprisal have been observed across multiple languages and psycholinguistic measures ( Wilcox et al. 2023 ; Shain et al. 2024 , e.g.,) . Entropy-based predictors are thus often evaluated as possible complements to well-established surprisal predictors; we follow this practice by including surprisal predictors in our regression models. Several forms of entropy have been evaluated in earlier work. Word-level (or token-level) contextual entropy, the predictor of focus in the present study, is studied by van Schijndel and Linzen 2019 as well as several more recent studies ( Cevoli et al. 2022 ; Pimentel et al. 2023 ; Wilcox et al. 2023 ; Giulianelli et al. 2024 ) . Other work considers the total entropy of the remainder of a sentence—its raw value ( Roark et al. 2009 ) , the reduction in this entropy at each incoming word ( Hale 2003 ; Hale 2006 ) , or both ( Linzen and Jaeger 2016 ) . Because total entropy is difficult to exactly compute—especially with contemporary language models—some additional work calculates the entropy of a bounded number of future words as a middle ground ( Frank 2013 ; Giulianelli et al. 2024 ) . The work by Giulianelli et al. 2024 comes closest to our proposed method of MC estimation of contextual word entropy. These authors approximate the entropy of a continuation of a sentence by generating samples of up to L\in\{5,10,15\} tokens following a context string. This differs from our work in that they aim to approximate continuation entropy rather than next-word entropy. 3 Formulations of Contextual Entropy 3.1 Shannon Entropy Shannon entropy ( Shannon 1948 ) is the standard form of entropy studied in previous psycholinguistic work. A word’s contextual Shannon entropy is defined as its expected surprisal, i.e., its expected negative log probability given the preceding words w_{1..i-1} : \displaystyle\ent(W_{i}\mid w_{1..i-1})= (1) \displaystyle-\sum_{w\in V}P(w\mid w_{1..i-1})\log_{2}P(w\mid w_{1..i-1}), where V is the vocabulary of all possible words and W_{i} is a random variable over V . 3.2 Rényi Entropy Pimentel et al. 2023 discuss the possibility that readers’ anticipatory processing may be guided by strategies other than considering the expected surprisal of the next word (i.e., Shannon entropy). For instance, readers might rely on the surprisal of the single most likely word; or, at the other extreme, they might consider the number of possible next words regardless of each word’s exact probability. Rényi entropy ( Rényi 1961 ) is a generalization of Shannon entropy which captures this spectrum of possible anticipatory reading strategies. A word’s contextual Rényi entropy of order \alpha is defined as follows: \displaystyle\ent_{\alpha}( \displaystyle W_{i}\mid w_{1..i-1})= (2) \displaystyle\lim_{\beta\rightarrow\alpha}\frac{1}{1-\beta}\log_{2}\sum_{w\in V}\Big(P(w\mid w_{1..i-1})\Big)^{\beta}. When \alpha=0 , Rényi entropy considers the number of possible next words. If \alpha=1 , Rényi entropy equals Shannon entropy (via an application of L’Hôpital’s rule to Eq. ( 2 )). And when \alpha=\infty , Rényi entropy measures the surprisal of the single most likely word in context. Pimentel et al. 2023 test several values of \alpha and find that Rényi contextual entropy with \alpha=1/2 is a relatively strong predictor of reading times. Our regression experiments evaluate both Shannon entropy and Rényi entropy with \alpha=1/2 to get a fuller picture of how first-token and MC approximation affect metrics describing anticipatory word processing. 3.3 First-Token and MC Approximations Contemporary LMs typically work with a finite subword vocabulary—defined using a method like Byte-Pair Encoding ( Sennrich et al. 2016 ) —that supports an infinite word vocabulary, as words can span a variable number of subwords. Because summing over an infinite vocabulary is intractable, recent work instead considers the entropy of a word’s first subword token. First-token contextual Shannon entropy is defined as follows: \displaystyle\ent(W_{i}\mid w_{1..i-1})\approx (3) \displaystyle-\sum_{t\in T}P(t\mid w_{1..i-1})\log_{2}P(t\mid w_{1..i-1}), where t is a token drawn from the LM’s subword vocabulary T . First-token Rényi entropy can be defined analogously. These approximation are a lower bound on