How new data permeates LLM knowledge and how to dilute it

Chen Sun, Renat Aksitov, Andrey Zhmoginov, Nolan Andrew Miller, Max Vladymyrov, Ulrich Rueckert, Been Kim, Mark Sandler · 2025

Read How new data permeates LLM knowledge and how to dilute it by Chen Sun, Renat Aksitov, Andrey Zhmoginov, Nolan Andrew Miller, Max Vladymyrov, Ulrich Rueckert, Been Kim, Mark Sandler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09522.txt.

Introduction

Chen Sun and seven co-authors study how new facts leak through a language model's knowledge, in a 2025 paper. When a model learns a new fact, it may start applying that fact in places it does not belong, a priming effect that can seed false statements. To study the leak, the team built a test set of one thousand three hundred twenty varied samples designed to be outlandish probes. They then taught models new facts and watched where the facts surfaced.

The strength of the leak could be predicted in advance from how likely the model found key words before learning. That pattern held across many settings. The paper also explores how to dilute unwanted spread, a practical concern for anyone updating models with fresh data.

Catalog record

CREATORChen Sun, Renat Aksitov, Andrey Zhmoginov, Nolan Andrew Miller, Max Vladymyrov, Ulrich Rueckert, Been Kim, Mark Sandler
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-09522

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How new data permeates LLM knowledge and how to dilute it Chen Sun, Renat Aksitov, Andrey Zhmoginov, Nolan Andrew Miller, Max Vladymyrov, Ulrich Rueckert, Been Kim, Mark Sandler Abstract Large language models learn and continually learn through the accumulation of gradient-based updates, but how individual pieces of new information affect existing knowledge, leading to both beneficial generalization and problematic hallucination, remains poorly understood. We demonstrate that when learning new information, LLMs exhibit a "priming" effect: learning a new fact can cause the model to inappropriately apply that knowledge in unrelated contexts. To systematically study this phenomenon, we introduce "Outlandish," a carefully curated dataset of 1320 diverse text samples designed to probe how new knowledge permeates through an LLM’s existing knowledge base. Using this dataset, we show that the degree of priming after learning new information can be predicted by measuring the token probability of key words before learning. This relationship holds robustly across different model architectures (PALM-2, Gemma, Llama), sizes, and training stages. Finally, we develop two novel techniques to modulate how new knowledge affects existing model behavior: (1) a “stepping-stone” text augmentation strategy and (2) an “ignore-k” update pruning method. These approaches reduce undesirable priming effects by 50-95% while preserving the model’s ability to learn new information. Our findings provide both empirical insights into how LLMs learn and practical tools for improving the specificity of knowledge insertion in language models. Further materials: https://sunchipsster1.github.io/projects/outlandish/ . Figure 1: Outlandish dataset and main finding on “priming”. (a) Sample texts within the Outlandish dataset. (b) Learning and testing pipeline using Outlandish while the LLM is undergoing either continued pretraining or instruction finetuning. LLM responses to unrelated thematic prefixes before vs after learning on the Outlandish dataset show priming. (c) The degree of priming after learning (score formalized in eq. 1 ) can be predicted from the keyword probability before learning. 1 Introduction The ability of large language models (LLMs) to integrate new knowledge is central to their utility. Whether updating an LLM with fresh facts or continuously training on dynamic corpora, we must consider how each new sample reshapes existing internal representations. Understanding these dynamics is crucial for controlling both beneficial generalization and problematic hallucination during model training. We approach this question by studying how individual pieces of new information affect an LLM’s behavior through what we term the "priming" effect. “Priming”, originating from experimental psychology, is the phenomenon whereby an agent’s exposure to a particular event will influence their response to a subsequent closely related event ( Doyen 2012 ; Meyer and Schvaneveldt 1971 ; Tulving et al. 1982 ) . We formalize it for the study of large language models in equation ( 1 ). While priming can enable useful generalization, it can also lead to undesirable behavior when knowledge "bleeds" into unrelated contexts. To systematically study this phenomenon, we needed a way to precisely measure how new knowledge affects existing model behavior. We introduce "Outlandish," a novel dataset of 1320 diverse text samples specifically designed to probe knowledge permeation in LLMs. Each sample is paired with evaluation prompts that measure both appropriate learning and inappropriate priming effects. Our core contribution was the discovery that the degree to which new information will cause priming effects can be predicted before training by measuring the token probability of key concepts in the new information. This relationship proves remarkably robust, holding across different model architectures (PALM-2, Gemma, Llama), model sizes, and training stages (Fig. 1 , 2 , Appendix Fig. 11 , 13 , 14 , 15 ). Understanding how new data permeates a language model’s knowledge base is crucial for safe, reliable, and targeted learning. While many aspects—from architectural design to algorithmic choices—affect model updates ( Meng et al. 2022a ; Hase et al. 2023 ; Nanda et al. 2023 ; Geva et al. 2023 ) , our work underscores the powerful influence of the data itself. By showing how to measure, predict, and mitigate the unintended consequences of learning single samples, we provide a foundation for building more robust and controlled continual-learning systems. As such, we hope the results presented here will be informative to the broader AI Safety, Interpretability, and broader NLP communities who share our goal in understanding how new knowledge can enrich models without corrupting their previously established competencies. Our contributions are as follows: • We investigate how new texts, when inserted into an LLM by gradient updates, affect existing knowledge. We discover that the impact ("priming") of new text after learning on existing knowledge can be predicted by metrics (i.e. token probability) measured before learning (Fig. 1 , 2 ). This observation was robust across models (Fig. 2 , Fig. 13 , 14 ), model sizes (Fig. 16 ), learning stages (Fig. 15 ). • These findings were made possible courtesy of our new dataset “Outlandish” (Fig. 1 ). • Finally, we demonstrate how a simple text augmentation technique, as well as a simple yet novel update pruning technique can modulate how much training on new texts affect existing knowledge, enhancing the specificity of gradient-based learning (Fig. 6 , 5 ). 2 Related Work The nature of new learning and their impact on the existing language model is of central importance to understanding how large language models learn, and is therefore of great interest to several areas of machine learning research. 2.1 Knowledge insertion, Memory and Interpretability Our work is related to contemporary work on knowledge insertion and memory, which has most often been conducted within the framework of the rapidly growing research on Interpretability. Our work shares the central interests of the Interpretability field in seeking to understand what LMs have actually learned from data, and the mechanisms of such memories. In Interpretability, important works have sought to reconstruct minimalist working circuits to recapitulate such functions ( Geva et al. 2020 ; Geva et al. 2022 ; Roberts et al. 2020 ; Geva et al. 2023 ; Nanda et al. 2023 ; Ghandeharioun et al. 2024 ) . These works painstakingly dissect, characterize, and reconstruct LLM memory, finding the consequences of knowledge injection in LLM function (and even what happens when they are injected at non-matched localizations ( Hase et al. 2023 ) ), the mechanisms of retrieval ( Nanda et al. 2023 ; Geva et al. 2023 ) , the surprising sparse localization of memories ( Meng et al. 2022a ; Meng et al. 2022b ) , as well as the oftentimes surprising extent to which injection of new texts into LMs can cause hallucinations ( Gekhman et al. 2024 ; Wan et al. 2023 ; Yin et al. 2023 ; Huang et al. 2023 ) , or cause mistakes in downstream reasoning ( Huang et al. 2023 ; Cohen et al. 2023a ) . While there are many factors that affect the outcome of language model learning such as important architectural and algorithmic components (and many of these factors have been studied in the works mentioned above), our study hones in on one other realm of factors: how different training data with diverse characteristics impact learning. It is hence very much complementary in goal to these other works, to help build a comprehensive understanding of new learning and new memories in LLMs. 2.2 Learning dynamics

Read the complete text