Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
Tina Behnia, Puneesh Deora, Christos Thrampoulidis · 2025
Read Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization by Tina Behnia, Puneesh Deora, Christos Thrampoulidis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16096.txt.
Introduction
Tina Behnia, Puneesh Deora, and Christos Thrampoulidis probe how the mix of training text shapes what a language model learns. Models train on text that blends plain word patterns, which make speech fluent, with facts linking one word to another. The team built a fake training set where each part can be tuned alone. They can vary how often each fact shows up in new settings.
Richer settings slow a model's grip on facts it was trained on. Gains on facts in new settings hinge on how those settings are built. Sometimes rich variety is the only path to any recall. Some setups break fact recall, others break word patterns, and some break both. By swapping model parts, they trace the new-setting failures to the layers that code words in and out. The testbed lets researchers study effects that large real models blur.
Catalog record
| CREATOR | Tina Behnia, Puneesh Deora, Christos Thrampoulidis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-16096
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization Tina Behnia, Puneesh Deora, Christos Thrampoulidis Abstract Language models are pretrained on sequences that blend statistical regularities (structures making text fluent) with factual associations between specific tokens (corresponding to knowledge of facts). While recent work suggests that the variability of their interaction, such as paraphrases of factual associations, critically determines generalization ability, we lack a systematic analysis of these impacts. This paper introduces a flexible synthetic testbed that combines a statistical stream of generic tokens with an abstract factual stream of source-target token pairs, enabling fine-grained control over their interaction. The design enables the independent control of diversity nature by manipulating stream composition (contextual structure) and the level of diversity by varying which statistical streams each fact appears in. Through controlled experiments, we find that while higher contextual diversity delays in-distribution (ID) factual accuracy, its effect on out-of-distribution (OOD) factual generalization depends critically on contextual structure. In some cases, OOD performance follows the same trend as ID, but in others, diversity becomes essential for non-trivial factual learning. Even when low diversity prohibits factual recall, optimal diversity levels depend on training duration. Beyond factual recall failures, we identify structures where statistical generalization fails independently, and others where both capabilities degrade simultaneously. This demonstrates how the interplay between contextual design and diversity level impacts different aspects of generalization. Furthermore, through a series of controlled interventions on the model components, we trace the OOD generalization failures to distinct optimization bottlenecks, highlighting the importance of the learned embedding and unembedding layers. Overall, our synthetic framework allows us to isolate effects that would be confounded in large-scale studies, thus offering a controlled testbed for future investigations. † † Code available at https://github.com/TinaBehnia/FactStat Corresponding author: [email protected] 1 Introduction Modern transformer-based language models (LMs), trained on massive corpora of natural-text sequences, simultaneously learn to generate contextually plausible sequences that follow statistical and linguistic patterns, while also learning significant amount of real-world knowledge. Their parameters effectively become an implicit knowledge base that can rival structured knowledge bases in question-answering and recall tasks [ Petroni et al. 2019 , Roberts et al. 2020 , Dai et al. 2021 , Allen-Zhu and Li 2023 , Akyürek et al. 2022 , Yang et al. 2024 , Mallen et al. 2023 , Meng et al. 2023 ] . This dual capability is remarkable precisely because the LM receives no explicit fact supervision during training: it never encounters knowledge graphs, labeled facts, or any explicit differentiation between linguistic and factual content. Yet somehow, it develops the ability to both follow linguistic patterns and encode world knowledge. How does the conceptually-simple next-token prediction objective enable a language model to implicitly learn both linguistic structure and factual knowledge? Are there inherent trade-offs in acquiring these two types of information? Intuitively, the repetition of factual information across varying and diverse linguistic contexts during training, essentially appearing as paraphrases of the same underlying facts, likely plays a key role in facilitating factual recall from pure next-token prediction supervision. But, this raises several questions: How frequently must a model encounter a fact during training to reliably learn it, and how does this depend on the diversity of linguistic contexts (paraphrases) in which the fact appears? Under what training conditions is it possible to isolate factual associations from linguistic structures? Moreover, is diversity only possibly impacting factual recall, or can it also impact the model’s ability to learn statistical patterns? Figure 1 : Proposed testbed and summary of key findings. (A) Sequences are generated by combining a factual stream , a set of atomic pairs, and a statistical stream of templates, each a statistical distribution over sequences with two fact placeholder positions. Final sequences insert a fact into a sequence sampled from a template (Sec. 2 ). (B) We control two key data properties. (Left) Contextual structure: The degree of statistical and (fact) positional variation across templates. We explore three structures: Stats Var (varied statistics, fixed positions), Pos Var (fixed statistics, varied positions), and Stats-Pos Var (varied on both axes). (Right) Diversity level: The number of unique templates each fact is paired with in the training corpus. The table shows an exposure matrix , where dots indicate template-fact pairs used for training (Sec. 2.1 - 2.2 ). (C) Low diversity impairs out-of-distribution generalization, but the specific aspects affected (factual recall vs. statistical pattern generation) depend on contextual structure. At high diversity, all structures enable both types of generalization, though with varying efficiency (colored boxes: darker means more training iterations needed with higher DIV ) (Sec. 3 ). (D) While training on low-diversity data impairs statistical and factual generalization, when the embeddings or unembeddings from a high-diversity model are “patched” in, the remaining modules can still learn a generalizing solution even when trained on that same low-diversity data (Sec. 4 ). The intriguing factual-recall capability of LLMs has motivated several recent studies probing how models acquire, represent, and recall facts using controlled setups. Allen-Zhu and Li 2023 and very recently Zucchet et al. 2025 study synthetic biography corpora with fixed templates and varying factual associations (e.g., name, age, occupation) to examine how paraphrastic diversity affects factual recall. While their data setup provides more control than raw natural-text corpora, where linguistic patterns and facts are intricately interleaved, they still operate at considerable scale, requiring transformers with 8-12 layers/8-12 heads/512-768 dimension for empirical analysis. This limits accessibility and makes comprehensive parameter sweeps over controllable variables computationally challenging. Additionally, by employing fixed templates, these setups isolate focus on factual learning while ignoring the model’s ability to simultaneously acquire linguistic structure and leaving fine-grained impacts of diversity on learning of both components underexplored. Motivated by these, we develop a small-scale synthetic testbed for studying at a fine-grained level the interactions between statistical and factual learning streams, specifically as captured in terms of context diversity . Our framework offers three key advantages. Tractability: Its minimal scale enables comprehensive parameter sweeps and in-depth analysis that would be computationally challenging in large-scale setups. All our experiments use 4-layer/4-head/32-dim transformers, and phenomena can be reproduced even with 1-layer transformers. All experiments are performed on on single Tesla V100-SXM2-GPU(16GB) and can be replicated on the free-version of Google-Colab. Reproducibility: Despite its minimalism, our framework reproduces empirical phenomena from large-scale studies, including stage-wise factual learning [ Zucchet et al. 2025 ] and improved factual recall with larger diversity [ Allen-Zhu and Li 2023 ] . Experimental control: The explicit statistical stream enables exhaustive exploration of how pretraining data diversity, both in terms of exposure level and structural variety, impacts transformer out-of-distribution generalization under next-token prediction training. In broader perspective, our methodology mirrors research turning to deliberately small-scale tasks and transformers to study other next-token prediction phenomena, such as in-context learning [ Garg et al. 2022 , Edelman et al. 2024b ] , grokking [ Power et al. 2022 , Nanda et al. 2023 ] , and arithmetic reasoning [ Lee et al. 2023 ] . Within factual recall research, our data setup bears some abstract similarity to Nichani et al. 2024 , who use synthetic tasks to analyze scaling laws and dynamics of factual recall. Yet, their setup lacks statistical learning, any account of diversity effects, and operates in