The Frequency Confound in Language-Model Surprisal and Metaphor Novelty

Omar Momen, Sina Zarrieß · 2026

Read The Frequency Confound in Language-Model Surprisal and Metaphor Novelty by Omar Momen, Sina Zarrieß (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06506.txt.

Introduction

Omar Momen and Sina Zarrieß question a reported link between language model surprisal and how novel a metaphor feels. Surprisal measures how unexpected a word is in context and is often used as a stand-in for predictability. It is also tightly tied to how frequent a word is, which may be the real driver.

Using two frequency measures and surprisal from eight Pythia model sizes across many training checkpoints, they find word frequency predicts metaphor novelty more strongly than surprisal does. The surprisal link peaks early in training, then fades as its tie to frequency grows. Settings once read as optimal for predictability may mostly reflect word frequency.

Catalog record

CREATOROmar Momen, Sina Zarrieß
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06506

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Frequency Confound in Language-Model Surprisal and Metaphor Novelty Omar Momen, Sina Zarrieß Abstract Language-model (LM) surprisal is widely used as a proxy for contextual predictability and has been reported to correlate with metaphor novelty judgments. However, surprisal is tightly intertwined with lexical frequency. We explore this interaction on metaphor novelty ratings using two different word frequency measures. We analyse surprisal estimates from eight Pythia model sizes and 154 training checkpoints. Across settings, word frequency is a stronger predictor of metaphor novelty than surprisal. Across training stages, the surprisal--novelty association peaks at an early stage and then falls again, mirroring a similarly timed increase in the surprisal--frequency association. These results suggest that the often-reported optimal LM surprisal settings may incorrectly associate contextual predictability with metaphor novelty and processing difficulty, whereas lexical frequency may be the major underlying factor. The experiment resources are publicly available. 1 1 1 Data and code: https://github.com/OmarMomen14/surprisal_frequency_novelty 1 Introduction Conceptual Metaphor Theory 8 argues that a metaphor is not only a linguistic phenomenon but also a cognitive mechanism through which abstract concepts are mapped into more concrete domains. Many of such metaphorical mappings are highly conventionalised, e.g., ARGUMENT \rightarrow WAR, TIME \rightarrow MONEY, or LIFE \rightarrow JOURNEY. But metaphorical mappings can vary in their degree of novelty: whereas we ‘‘live by’’ many conventionalised metaphors in everyday language, novel metaphors can stand out as instances of linguistic creativity. For example, the metaphorical mappings of WATER in the sentence: ‘‘The arrested water shone and danced.’’ 2 2 2 Example from the VU Amsterdam Metaphor Corpus (VUAMC) 18 , and the sentence is traced back to the novel “Still Life” by A. S. Byatt, 1985. are considered relatively novel. Novel (creative) metaphors introduce less familiar mappings and require greater interpretive effort to understand 7 ; 2 ; 15 . Commonly, metaphor novelty is measured by human ratings 4 . Recent studies have shown that metaphor novelty ratings by humans correlate significantly with both lexical frequency 4 ; 17 and LM surprisal 10 . Interestingly, 10 report that, within the same model family, surprisal estimates from smaller LMs consistently correlate more strongly with metaphor novelty ratings than those from larger variants. This mirrors the inverse scaling effect observed in studies of reading times, where smaller models often provide better fits to human reading behaviour ( 11 ; 12 ; 3 ) . Lexical frequency has been proposed as a potential explanation for the inverse scaling effect in reading time studies. 13 argue that larger LMs are better than humans at predicting low-frequency words, which weakens the correlation between their surprisal estimates and human reading-time data. Also, 14 report that non-contextual frequency has a stronger effect on predicting reading time than surprisal. Surprisal Theory ( 6 ; 9 ) is a common approach to studying processing difficulties in humans and LMs. The theory predicts that processing difficulty increases as a word becomes less predictable in context. Recently, the consistent inverse scaling pattern reported in studies of different phenomena such as reading time or metaphor novelty ( 11 ; 3 ; 10 ) has motivated an essential assumption that smaller LMs or shorter pretraining provide more human-aligned surprisal estimates. However, whether this pattern reflects genuinely more human-like contextual prediction or confounding lexical properties remains unclear. In this paper, we analyse the associations between the three quantities: “Metaphor Novelty Ratings”, “Lexical Frequency” and “LM Surprisal”. We propose two different methods for computing lexical frequency. Then, we compute LM surprisal estimates across (i) eight Pythia model sizes and (ii) 154 pretraining checkpoints. Our results show that word frequency is a substantially stronger predictor of novelty than surprisal across settings, and that the configurations where surprisal performs “best” are also those where it aligns most closely with frequency. We therefore caution against interpreting strong early-training or small-model surprisal–novelty associations as straightforward support for surprisal-based accounts of metaphor novelty, and we argue that progress requires clearer theoretical and methodological separation between contextual predictability and lexical-frequency effects. 2 Data & Methods This Section describes our experimental set-up to measure the association between word frequency and metaphor novelty on the one hand and LM surprisal on the other hand. 2.1 Dataset We base our study on the VU Amsterdam Metaphor Corpus (VUAMC) 18 . Every word in VUAMC is annotated as either a metaphoric word or not. Building on VUAMC, 4 collected crowd-sourced metaphor novelty ratings for the 15,155 metaphoric content words in VUAMC and converted them into continuous scores in the range (-1, +1), where -1 denotes the most conventional and +1 the most novel. They additionally binarised these scores using a 0.5 threshold, resulting in 353 metaphors labelled as novel out of the 15,155 content metaphoric words (see Appendix A ). In our experiment, we use these 15,155 instances, each consisting of a single sentence, a target word within the sentence (a content metaphoric word), and an associated metaphor novelty score. 2.2 Model Suite To examine the effects of model scale and pretraining progress (data/steps), we use the Pythia model suite 1 . Pythia consists of decoder-only causal LMs at 8 sizes (70M--12B parameters), all trained on the same 300B-token pretraining corpus 3 3 3 The Pile 5 in the same order. For each model, Pythia provides 154 intermediate checkpoints saved every 1,000 training steps (corresponds to additional \approx 2M tokens seen during these steps), and denser early checkpoints at steps \{1,2,4,8,16,32,64,128,256,512\} . The exact pretraining sequences seen at each checkpoint can be reconstructed using available scripts. 2.3 Surprisal For causal LMs, surprisal, computed for a word w_{i} within a sequence W_{0:n} is \mathrm{Surprisal}(w_{i})=-\log p(w_{i}\mid w_{<i}) 4 4 4 We use \log of base e for all log computations in our study. . In our experiment, we measure word-level surprisal for metaphoric target word(s) in their sentence-level context by running an independent, teacher-forced forward pass per sentence and recording the target word surprisal. To map token probabilities to a target word, we locate the word’s character offsets in the sentence and sum token-level surprisals over the minimal token span in the sequence’s tokenisation that covers these offsets. We additionally apply word-probability corrections for leading-whitespace tokenisation confounds 16 , and we prepend a BOS token so surprisal is defined even when the target is the first word in a sentence. 2.4 Word Frequency Estimating word frequency is non-trivial because frequency is not directly observed as a property of a word, but estimated from a reference corpus. Different corpora approximate different kinds of language exposure, and there is no single corpus that can be assumed to represent the linguistic experience of all speakers. To tackle this problem, we utilise two different estimates of word frequency reflecting both human and LM lexical memory. In both estimates, we compute the negative log frequency of each target metaphoric word for easier numerical comparability with surprisal. 1. Negative Log Frequency in General Language Use: We compute the negative log frequency of each target metaphoric word using the Python library wordfreq 5 5 5 https://pypi.org/project/wordfreq/ , which provides corpus-independent frequency estimates aggregated from multiple large-scale sources, rather than deriving counts from a single corpus. We treat this as an estimate of word frequency for an “average English speaker”, and hereafter denote it as

Read the complete text