How do language models learn facts? Dynamics, curricula and hallucinations

Nicolas Zucchet, Jörg Bornschein, Stephanie Chan, Andrew Lampinen, Razvan Pascanu, Soham De · 2025

Read How do language models learn facts? Dynamics, curricula and hallucinations by Nicolas Zucchet, Jörg Bornschein, Stephanie Chan, Andrew Lampinen, Razvan Pascanu, Soham De (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21676.txt.

Introduction

Nicolas Zucchet, Soham De and their co-authors study how language models learn facts during training in 2025. Models store vast knowledge, yet the steps by which they gain it are still poorly understood.

On a made-up recall task, learning came in three phases, with a flat stretch before exact facts clicked into place. That flat phase matched the forming of attention circuits that support recall. Uneven training data shortened the flat phase. False claims, or hallucinations, arose at the same time as true knowledge. Adding new facts later by fine-tuning also quickly harmed old stored facts.

Catalog record

CREATORNicolas Zucchet, Jörg Bornschein, Stephanie Chan, Andrew Lampinen, Razvan Pascanu, Soham De
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-21676

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How do language models learn facts? Dynamics, curricula and hallucinations Nicolas Zucchet, Jörg Bornschein, Stephanie Chan, Andrew Lampinen, Razvan Pascanu, Soham De Abstract Large language models accumulate vast knowledge during pre-training, yet the dynamics governing this acquisition remain poorly understood. This work investigates the learning dynamics of language models on a synthetic factual recall task, uncovering three key findings: First, language models learn in three phases, exhibiting a performance plateau before acquiring precise factual knowledge. Mechanistically, this plateau coincides with the formation of attention-based circuits that support recall. Second, the training data distribution significantly impacts learning dynamics, as imbalanced distributions lead to shorter plateaus. Finally, hallucinations emerge simultaneously with knowledge, and integrating new knowledge into the model through fine-tuning is challenging, as it quickly corrupts its existing parametric memories. Our results emphasize the importance of data distribution in knowledge acquisition and suggest novel data scheduling strategies to accelerate neural network training. \reportnumber 001 Large language models offer a powerful and intuitive interface to access the vast amounts of knowledge on which they were trained. The learning process acts as a lossy compression algorithm turning training data into neural network parameters, thereby implicitly determining what information is preserved within the final model. The extent and nature of this information loss depend on numerous factors, including architecture, training objective, and data distribution, dependencies which remain poorly understood. Deciphering the mechanisms underlying knowledge compression is increasingly important as these models are becoming our main gateway to human knowledge. Our work addresses this challenge by systematically investigating how language models acquire factual knowledge. Inspired by a long history of work in neuroscience focusing on associative recall to study ( Pavlov 1927 ) and model ( Hopfield 1982 ) intelligence, we focus on a synthetic factual recall task to study in depth the learning dynamics of language models. Our contributions are: – We adapt the synthetic task of Allen-Zhu and Li 2023 , which generates artificial biographies for testing factual recall, to make it suitable for studying the formation of associative memories (Section 1 ). – We find that language models learn in multiple phases on this task. They first learn overall distribution statistics, then their performance plateaus, and, finally, they acquire individual-specific knowledge (Section 2.1 ). Leveraging a novel attention patching technique, we demonstrate that attention-based recall circuits develop during this plateau (Section 2.2 ). – We analyze the impact of data distribution properties on these dynamics, revealing that imbalanced distributions accelerate the transition through the intermediate plateau phase (Section 3 ) but lead to overfitting. We further demonstrate that data scheduling strategies can exploit this accelerated transition while mitigating overfitting, providing a rare example in which data curricula benefit (self-)supervised learning. – We highlight the ineffectiveness of fine-tuning to incorporate new knowledge in the model (Section 4 ). This stems from two related factors: First, models hallucinate (overconfident predictions on unseen individuals) as soon as they acquire individual-specific knowledge. Second, associative memories stored in feed-forward layers are rapidly corrupted when training on new individuals. 1 An experimental setup to track knowledge over the course of learning This study investigates the learning dynamics underlying factual recall and knowledge acquisition in language models. This presents two main methodological challenges: First, we need to isolate the model’s knowledge from other abilities. Second, we want to evaluate the models over the course of learning, which requires computationally efficient measurement techniques. Following Allen-Zhu and Li 2023 , we train language models on synthetic biographies that feature key properties of datasets used to train large language models. By carefully designing these synthetic biographies, we can attribute the model’s ability to predict specific tokens solely to its acquired knowledge and efficiently measure its knowledge through its performance on these tokens. In this section, we first define knowledge and contrast it with memory, then describe our synthetic dataset and introduce the metrics we use to track knowledge during training, and finally describe training details. 1.1 Knowledge, and how it differs from memory For our analysis, we define knowledge to be the information a model has internally stored about its training data, abstracted from the specific form in which it was encountered. It is important to distinguish it from memorization. While knowledge involves accessing and applying information flexibly across different contexts, memorization is tied to particular training instances. In this view, knowledge can be understood as a form of memorization in a latent semantic space. For instance, knowing that Paris is France’s capital represents knowledge, while remembering the exact sentence ‘Paris is the capital of France’ would constitute memorization. From a practical perspective, this distinction carries significant implications. Memorization can be problematic due to potential data leakage ( Nasr et al. 2023 , e.g.,) , while knowledge is a more desirable property that enables models to ground their outputs in factual reality, making their responses both accurate and generalizable. 1.2 Synthetic biographies as a framework for studying knowledge An important methodological challenge in our study is to isolate knowledge from other model capabilities. To address this, we adapt the synthetic biography dataset of Allen-Zhu and Li 2023 , which offers several appealing properties. First, all facts in the dataset are atomic, meaning no fact can be derived from others, thus allowing us to disambiguate knowledge recall from other abilities like reasoning. Second, it is fully synthetic, which allows for precise control of the data distributional properties, while employing natural language with realistic statistics. For example, sufficient textual variation is crucial for enabling genuine knowledge acquisition rather than mere memorization ( Allen-Zhu and Li 2023 ) , cf. Figure A in the appendix. Third, and most importantly, previous work has established that relatively small language models trained on this dataset ( Allen-Zhu and Li 2023 ) store and use knowledge in a similar way to large language models ( Geva et al. 2021 ; Meng et al. 2022 ; Geva et al. 2023 ; Nanda et al. 2023b ) . Figure 1: Data generation process underlying the synthetic biography dataset we train on. We measure the knowledge stored within these models through the loss they achieve when predicting the attribute tokens, highlighted in blue. See Section 1 for more details. We summarize the data generation process in Figure 1 . The dataset contains a population of N randomly sampled individuals, each with a unique name and six attributes: birthdate, birthplace, university, major, company, and location. To generate a biography, we first sample an individual from the population. For each of attribute, we then sample a template from a finite pool, fill it with the individual’s information, and concatenate the resulting sentences in random order to form a complete biography. It should be noted that to make the dataset suitable for our study, our template generation and manipulation differs from the original setup. We detail these modifications in the next section and provide additional details in the appendix. 1.3 Measuring knowledge at scale Measuring knowledge in language models typically relies on factual question-answering tasks ( Lin et al. 2021 ) . However, this approach is computationally intractable in our setup, as it requires repeatedly fine-tuning models to handle question-answer prompt formats throughout the training process. Our framework offers a simple solution to this challenge:

Read the complete text