On Linear Representations and Pretraining Data Frequency in Language Models

Jack Merullo, Noah A. Smith, Sarah Wiegreffe, Yanai Elazar · 2025

Read On Linear Representations and Pretraining Data Frequency in Language Models by Jack Merullo, Noah A. Smith, Sarah Wiegreffe, Yanai Elazar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12459.txt.

Introduction

Some facts inside a language model sit along straight lines in its inner space, and Jack Merullo and co-authors ask what puts them there in this 2025 paper. Their answer points at training data counts. For fact triples of subject, relation, and object, how often the pair appeared together tracks strongly with whether the fact formed a clean linear pattern.

The link held across all stages of training in two open models. Rough thresholds emerged: about a thousand pairings for subjects and two thousand for objects. A regression trained on one model's patterns could even guess term frequencies in another model's secret training set. Frequency, the authors argue, is a lever for shaping what models know cleanly.

Catalog record

CREATORJack Merullo, Noah A. Smith, Sarah Wiegreffe, Yanai Elazar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12459

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Linear Representations and Pretraining Data Frequency in Language Models Jack Merullo, Noah A. Smith, Sarah Wiegreffe, Yanai Elazar Abstract Pretraining data has a direct impact on the behaviors and quality of language models (LMs), but we only understand the most basic principles of this relationship. While most work focuses on pretraining data’s effect on downstream task behavior, we investigate its relationship to LM representations. Previous work has discovered that, in language models, some concepts are encoded ‘linearly’ in the representations, but what factors cause these representations to form (or not)? We study the connection between pretraining data frequency and models’ linear representations of factual relations (e.g., mapping France to Paris in a capital prediction task). We find evidence that the formation of linear representations is strongly connected to pretraining term frequencies; specifically for subject-relation-object fact triplets, both subject-object co-occurrence frequency and in-context learning accuracy for the relation are highly correlated with linear representations. This is the case across all phases of pretraining, i.e., it is not affected by the model’s underlying capability. In OLMo-7B and GPT-J (6B), we discover that a linear representation consistently (but not exclusively) forms when the subjects and objects within a relation co-occur at least 1k and 2k times, respectively, regardless of when these occurrences happen during pretraining (and around 4k times for OLMo-1B). Finally, we train a regression model on measurements of linear representation quality in fully-trained LMs that can predict how often a term was seen in pretraining. Our model achieves low error even on inputs from a different model with a different pretraining dataset, providing a new method for estimating properties of the otherwise-unknown training data of closed-data models. We conclude that the strength of linear representations in LMs contains signal about the models’ pretraining corpora that may provide new avenues for controlling and improving model behavior: particularly, manipulating the models’ training data to meet specific frequency thresholds. We release our code to support future work. 1 1 1 Code is available at https://github.com/allenai/freq , and for efficient batch search at https://github.com/allenai/batchsearch . 1 Introduction Understanding how the content of pretraining data affects language model (LM) behaviors and performance is an active area of research ( Ma et al. 2024 ; Xie et al. 2023 ; Aryabumi et al. 2025 ; Longpre et al. 2024 ; Wang et al. 2025 ; Seshadri et al. 2024 ; Razeghi et al. 2023 ; Wang et al. 2024 ) . For instance, it has been shown that for specific tasks, models perform better on instances containing higher frequency terms than lower frequency ones ( Razeghi et al. 2022 ; Mallen et al. 2023 ; McCoy et al. 2024 ) . However, the ways in which frequency affects the internal representations of LMs to cause this difference in performance remain unclear. We connect dataset statistics to recent work in interpretability, which focuses on the emergence of simple linear representations of factual relations in LMs Hernandez et al. 2024 ; Chanin et al. 2024 . Our findings demonstrate a strong correlation between these linear representations and the frequency of terms in the pretraining corpus. Linear representations in LMs have become central to interpretability research in recent years ( Ravfogel et al. 2020 ; Elazar et al. 2021 ; Elhage et al. 2021 ; Slobodkin et al. 2023 ; Olah et al. 2020 ; Park et al. 2024 ; Jiang et al. 2024 ; Black et al. 2022 ; Chanin et al. 2024 ) . Linear representations are essentially linear approximations (linear transforms, directions in space) that are simple to understand, and strongly approximate the complex non-linear transformations that networks are implementing. These representations are crucial because they allow us to localize much of the behavior and capabilities of LMs to specific directions in activation space. This allows for simple interventions to control model behaviors, i.e., steering ( Todd et al. 2024 ; Subramani et al. 2022 ; Hendel et al. 2023 ; Rimsky et al. 2024 ) . Recent work by Hernandez et al. 2024 and Chanin et al. 2024 highlight how the linearity of different types of relations varies greatly depending on the specific relationships being depicted. For example, over 80% of entities in the “ country-largest-city ” relation, but less than 30% of entities in the “ star-in-constellation ” relation can be approximated this way ( Hernandez et al. 2024 ) . Such findings complicate the understanding of the Linear Representation Hypothesis, which proposes that LMs will represent features linearly ( Park et al. 2024 ) without providing when/why these form. While Jiang et al. 2024 provide both theoretical and empirical evidence that the training objectives of LMs implicitly encourage linear representations, it remains unclear why some features are represented this way while others are not. This open question is a central focus of our investigation. Whether linear representations for “common” concepts are more prevalent in models or simply easier to identify (using current methods) than those for less common concepts remains unclear. We hypothesize that factual relations exhibiting linear representations are correlated with higher mention frequencies in the pretraining data (as has been shown with static embeddings, see Ethayarajh et al. 2019 ), which we confirm in Section 4 . Our results also indicate that this can occur at any point in pretraining, as long as a certain average frequency is reached across subject-object pairs in a relation. In order to count the appearance of terms in data corpora throughout training, we develop an efficient tool for counting tokens in tokenized batches of text, which we release to support future work in this area. We also explore whether the presence of linear representations can provide insights into relation term frequency. In Section 5 , we fit a regression model to predict the frequency of individual terms (such as “The Beatles”) in the pretraining data, based on metrics measuring the presence of a linear representation for some relation. For example, how well a linear transformation approximates the internal computation of the “ lead-singer-of ” relation mapping “John Lennon” to “The Beatles” can tell us about the frequency of those terms in the pretraining corpus. Our findings indicate that the predictive signal, although approximate, is much stronger than that encoded in log probabilities and task accuracies alone, allowing us to estimate the frequencies of held-out relations and terms within approximate ranges. Importantly, this regression model generalizes beyond the specific LM it was trained on without additional supervision. This provides a valuable foundation for analyzing the pretraining corpora of closed-data models with open weights. To summarize, in this paper we show that: 1. The development of linear representations for factual recall relations in LMs is related to frequency as well as model size. 2. Linear representations form at predictable frequency thresholds during training, regardless of when this frequency threshold is met for the nouns in the relation. The formation of these representations also correlates strongly with recall accuracy. 3. Measuring the extent to which a relation is represented linearly in a model allows us to predict the approximate frequencies of individual terms in the pretraining corpus of that model, even when we do not have access to the model’s training data. 4. We release a tool for

Read the complete text