Information-Theoretic Storage Cost in Sentence Comprehension Kohei Kajikawa, Shinnosuke Isono, Ethan Gotlieb Wilcox Abstract Real-time sentence comprehension imposes a significant load on working memory, as comprehenders must maintain contextual information to anticipate future input. While measures of such load have played an important role in psycholinguistic theories, they have been formalized, largely, using symbolic grammars, which assign discrete, uniform costs to syntactic predictions. This study proposes a measure of processing storage cost based on an information-theoretic formalization, as the amount of information previous words carry about future context, under uncertainty. Unlike previous discrete, grammar-based metrics, this measure is continuous, theory-neutral, and can be estimated from pre-trained neural language models. The validity of this approach is demonstrated through three analyses in English: our measure (i) recovers well-known processing asymmetries in center embeddings and relative clauses, (ii) correlates with a grammar-based storage cost in a syntactically-annotated corpus, and (iii) predicts reading-time variance in two large-scale naturalistic datasets over and above baseline models with traditional information-based predictors. 1 Introduction A large body of evidence shows that language comprehension is highly incremental ( 33 ; 52 ; 28 ) . As the input unfolds, the parser integrates incoming words into the evolving representation while generating expectations about upcoming content. However, because working memory capacity is limited, maintaining these expectations incurs a cognitive cost: as unresolved predictions accumulate, they consume memory resources and create a processing bottleneck ( 27 ) . It has been argued that this bottleneck gives rise to processing difficulty, for example, in center-embedding sentences such as 1 , below, where the processing of an outer clause is interrupted by an inner one, forcing the parser to hold the incomplete outer structure in working memory. ( 57 ; 35 ; 18 ) : \ex . The reporter [ who the senator [ who Mary met ] attacked ] ignored the president. The memory burden of maintaining predicted syntactic elements has been discussed previously, often under the more general term of storage cost ( 57 ; 35 ; 1 ; 43 ; 18 ; 19 ; 30 ; 26 ) . The hypothesis that storage cost actively influences online processing has received empirical support in both controlled experiments across languages ( 6 ; 36 ; 51 ; 44 ) and naturalistic reading ( 25 ) . Figure 1: Illustration of the proposed storage cost measure. It quantifies the predictive potential shared between a target word w_{i} and the future \boldsymbol{w}_{[k:N]} , conditioned on the remaining context . Storage cost is defined as the sum of these predictive potentials across all context words, representing the load of pending information. The reporter who the senator who Mary met t attacked t ignored the president Mary met the senator who t attacked the reporter who t ignored the president Figure 2: Illustration of DLT storage cost as the predicted syntactic head hypothesis. In the center-embedded structure (top), the storage cost at Mary is five memory units, because five syntactic heads (the blue -colored words) are predicted to form a grammatical sentence at Mary . In contrast, the right-branching structure (bottom) imposes a minimal storage load. t denotes a trace of extraction. This example is adopted from 6 . Previously, most formalizations of storage costs have been derived from the incremental states of symbolic parsers ( 19 , e.g.,) . While grammar-based storage cost successfully captures the memory burden of structural prediction, it relies on specific syntactic theories and assumes discrete, uniform costs for predicted elements. In this paper, we propose a grammar-independent reformulation of storage cost grounded in information theory. The key idea is that maintaining a syntactic prediction corresponds to retaining information about a context word that is relevant to future input. We argue that this measure can be formalized as the extent to which this word reduces the surprisal of future material under uncertainty. Taken in context, this is therefore the contextualized half-pointwise mutual information between a word and future words. The resulting measure requires no explicit syntactic formalism, can be estimated from neural language models, and is continuous, quantified in bits rather than discrete counts. We validate the proposal through three complementary analyses. First, we examine whether the measure recovers well-known processing asymmetries in center-embedding and relative clause sentences, establishing theoretical plausibility. Second, we confirm a positive correlation with grammar-based storage cost, showing that the two measures capture overlapping structural intuitions. Third, we evaluate its predictive power on two large-scale naturalistic English reading-time datasets and compare it to that of a grammar-derived measure. We find significant improvements across multiple reading-time measures, establishing information-theoretic storage cost as a viable, theory-neutral alternative. At the same time, the variance each measure explains is largely independent, suggesting that storage cost estimated from formal grammars and neural language models accounts for partially distinct sources of reading-time variance. 2 Grammar-based Storage Cost The idea that maintaining predicted syntactic elements consumes working memory resources has long been assumed in psycholinguistics across different grammar formalisms ( 57 ; 35 ; 1 ; 43 ; 18 ; 19 ; 30 ; 26 ) . As an example, we take Dependency Locality Theory ( 19 , DLT;) . Grounded in dependency grammar, DLT defines storage cost as the number of predicted syntactic heads required to complete the current input as a grammatical sentence. Consider the following center-embedded sentence 2 and its right-branching variant 2 : \ex . Ṫhe reporter [ who the senator [ who Mary met ] attacked ] ignored the president. .̱ Mary met the senator [ who attacked the reporter [ who ignored the president ]] . At the point of Mary in 2 , five syntactic heads are required to form a grammatical sentence: (i) three verbs to which the three subject NPs connect, and (ii) two extraction traces anticipated in object positions (see also Figure 2 ). Thus, DLT storage cost at Mary incurs a cost of five “memory units.” In contrast, at each word in the right-branching variant of this sentence 2 , at most one syntactic head is unresolved. Consequently, DLT storage cost correctly predicts the well-known difficulty of processing center-embedded structures compared to right-branching ones ( 57 ; 35 ; 18 ) . Storage cost vs. integration cost. Storage cost in DLT constitutes a complementary component to integration cost . While storage cost quantifies the memory burden of maintaining structural predictions before they are resolved, integration cost arises from the resource consumption required to resolve syntactic dependencies when new input is encountered. They represent different cognitive demands imposed by the same dependency structure. Integration cost has long been a focal point of research ( 19 ; 15 , e.g.,) because it offers a straightforward account of locality effects , the processing difficulty associated with longer syntactic distances ( 19 ; 21 ; 49 ; 2 ; 45 ) . However, its empirical robustness is debated because null effects or even unexpected negative correlations (i.e., facilitation) are reported in naturalistic reading ( 9 ; 47 ; 11 ; 26 ) . In this context, lossy-context surprisal ( 13 ) , defined as prediction error resulting from noisy context representations, can be seen as an information-theoretic generalization of integration cost. Within this framework, locality effects emerge as a consequence of memory noise. Linking storage cost to reading behavior. The linking hypothesis between storage cost and reading times is that storage and integration share the same pool of cognitive resources: as more resources are allocated to storage, fewer remain available for integration, resulting in slower reading times ( 18 ; 19 ) . Controlled experiments have validated this prediction across languages, including English, Japanese, Slovenian, and Spanish ( 6 ; 36 ; 51 ; 44 ) . Furthermore, 25 demonstrate that storage costs based on both dependency grammar 1 1 1 25 adopt the Universal Dependencies framework ( 37 ; 8 ) and operationalize DLT storage cost as the number of unseen tokens whose co-dependents are already seen at a given word. and combinatory categorial grammar ( 50 , CCG;) yield better predictions on a held-out test set of naturalistic Japanese reading times than integration cost. Limitations of grammar-based storage cost. Storage cost metrics based on symbolic grammars have several limitations. Theoretically, they assume that all syntactic predictions carry the same cost. Practically, it can be challenging to count the heads needed to complete a sentence. This often requires specific assumptions about grammatical completion and accurate representation of phonologically null elements (e.g., traces), which are absent in surface-based annotations like Universal Dependencies ( 37 ; 8 ) . Our information-theoretic approach eliminates the need for specific syntactic theories and assumptions while capturing the probabilistic nature of sentence comprehension and replacing the discrete fixed-cost assumption with a continuous measure quantified in bits. 3 Information-Theoretic Storage Cost This section develops an information-theoretic formalization of storage cost. We first motivate the approach and provide the formal definition, then describe how to estimate the measure using a pre-trained masked language model. 3.1 From syntactic to information storage Grammar-based storage cost counts the number of predicted syntactic elements that must be held in memory. The implicit assumption is that each prediction represents information about the future that the comprehender must maintain in working memory until it is resolved and integrated into a longer-term memory store. We propose that this can be directly quantified as the amount of information content that a word carries about future words, or the extent to which that word reduces the surprisal of future words. We view information-based storage as a generalization of grammar-based storage cost. Grammar-based approaches are a special case where (i) information about the future is mediated exclusively by syntactic predictions and (ii) each prediction carries a uniform cost. Our formalization relaxes both assumptions—information is measured over surface forms, and each word’s contribution is a continuous quantity in bits. 3.2 Predictive potentials as expected PMI We operationalize the information a word carries about the future using contextualized pointwise mutual information ( 23 , \operatorname{pmi} ;) . The \operatorname{pmi} measures the association between outcomes of random variables (RVs) X and Y as the log-ratio of their joint probability to the product of their marginals: \displaystyle{\color[rgb]{0,0,0}\operatorname{pmi}}(x;y)\coloneqq{\color[rgb]{0,0,0}\log_{2}}\frac{p(X=x,Y=y)}{p(X=x)p(Y=y)} (1) Consider a word-valued RV W , which ranges over a vocabulary \Sigma , and a sentence-valued RV \boldsymbol{W} , which ranges over strings drawn from its Kleene closure, \Sigma^{*} . For a sentence {\color[rgb]{0,0,0}\boldsymbol{w}_{[1:N]}}=[w_{1},w_{2},\ldots,w_{N} ], at position k , the observed context is \boldsymbol{w}_{[1:k)}=[w_{1},\ldots,w_{k-1}] , and the future sequence is \boldsymbol{w}_{[k:N]}=[w_{k},\ldots,w_{N} ]. For a target word at position i (where i