How much do language models memorize?

John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexander M. Rush, Kamalika Chaudhuri, Saeed Mahloujifar · 2025

Read How much do language models memorize? by John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexander M. Rush, Kamalika Chaudhuri, Saeed Mahloujifar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24832.txt.

Introduction

John Morris and co-authors ask a direct question in a 2025 paper: how much do language models memorize? Past work found it hard to split true recall of training data from broad skill. The team built a way to split the two. They define the part tied to a set of data apart from the part tied to how data is made.

When the broad part is removed, total recall gives a gauge of model size limits. The team puts that limit at about 3.6 bits per unit of the model. As training data grows, models fill that space, then start to generalize. Tests on hundreds of small models back the pattern.

Catalog record

CREATORJohn X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexander M. Rush, Kamalika Chaudhuri, Saeed Mahloujifar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24832

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How much do language models memorize? John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexander M. Rush, Kamalika Chaudhuri, Saeed Mahloujifar Abstract We propose a new method for estimating how much a model “knows” about a datapoint and use it to measure the capacity of modern language models. We formally separate memorization into two components: unintended memorization , the information a model contains about a specific dataset, and generalization , the information a model contains about the true data-generation process. By eliminating generalization, we can compute the total memorization of a given model, which provides an estimate of model capacity: our measurements estimate that models in the GPT family have an approximate capacity of 3.6 bits-per-parameter . We train language models on datasets of increasing size and observe that models memorize until their capacity fills, at which point “grokking” begins, and unintended memorization decreases as models begin to generalize. We train hundreds of transformer language models ranging from 500 ​ K 500K to 1.5 ​ B 1.5B parameters and produce a series of scaling laws relating model capacity and data size to membership inference. † † date: August 24, 2026 † † correspondence: Saeed Mahloujifar at [email protected] Figure 1: Unintended memorization of uniform random data (Section 3 ). Memorization plateaus at the empirical capacity limit of different-sized models from the GPT-family, approximately 3.6 bits-per-parameter. Figure 2: Unintended memorization of text across model and dataset sizes (Section 4 ). All quantities are calculated with respect to a large oracle model trained on the full data distribution. 1 Introduction Figure 3: In our experiments on synthetic bitstrings, double descent occurs exactly when the dataset size begins to exceed the model’s capacity, when unintended memorization is no longer beneficial for lowering the loss. Figure 4: Train and test losses of different model and dataset sizes trained on text. Double descent occurs when dataset size exceeds model capacity. For the past several years, modern language models have been trained on increasingly large amounts of data, while parameter counts stay stagnant in the billions. For example, one recent state-of-the-art model ( Dubey & et al, 2024 ) has 8 billion parameters (around 32GB on disk) but is trained on 15 trillion tokens (around 7TB on disk). A long line of work ( Carlini et al., 2019 ; Mireshghallah et al., 2022 ; Nasr et al., 2023 ; Zhang et al., 2023 ; Carlini et al., 2023b ; Schwarzschild et al., 2024 ) questions whether such pretrained language models memorize their training data in a meaningful way. Most research approaches this problem either through the lens of extraction, aiming to recover full training datapoints from model weights, or membership inference, classifying whether a training point was present in the training data of a given model. Studies of language model extraction argue that a datapoint is memorized if we can induce the model to generate it ( Carlini et al., 2023b ; Nasr et al., 2023 ; Schwarzschild et al., 2024 ) . We argue that such generation does not necessarily serve as a proof of memorization. Language models can be coerced to output almost any string ( Geiping et al., 2024 ) ; hence the fact that a model outputs something is not necessarily a sign of memorization. To address this issue, some researchers have suggested regularizing the input to the language model, such as by limiting its length ( Schwarzschild et al., 2024 ) or matching it to the prefix ( Carlini et al., 2023b ) preceding the memorized sentence. However, none of these constraints allow us to distinguish whether a model outputs a string due to memorization or good generalization. For example, a language model prompted to add two numbers can output the answer without having seen the equation before. To address this issue, we propose a definition of memorization that quantifies the extent to which a model retains information about a specific datapoint. Our approach leverages the concept of compression rate in bits: a model is considered to have memorized an input if the input can be compressed into a shorter encoding when the model is available. This framework draws inspiration from Kolmogorov information theory ( Kolmogorov, 1963 ) and Shannon information theory ( Shannon, 1948 ) , but remains practical by estimating information content using model likelihoods. We address the fundamental challenge of distinguishing memorization from generalization ( Prashanth et al., 2024 ) by decomposing memorization into two distinct components: unintended memorization , which captures the information the model retains about a specific dataset, and generalization , which represents the knowledge the model acquires about the underlying data-generating process. This separation is similar to the approach in ( Brown et al., 2021 ) , which defines memorization using conditional mutual information between the dataset and the trained model, conditioned on the true concept. However, our notion differs in that it enables this separation at the instance level using algorithmic definitions of information. To understand our new quantities, we measure unintended memorization and generalization by training language models of varying capacity on datasets of different sizes. We first eliminate the question of generalization entirely by training on a dataset of random uniformly-sampled bitstrings. In this setting, we can exactly measure the amount of information contained about the data inside the model. This gives us a principled way to measure language model capacity when trained on uniform datasets of exact known information content. We find that GPT-style transformers can store between 3.5 and 4 bits of information in each model parameter, depending on model architecture and precision. We then repeat our experiments with real text, where generalization is possible and even beneficial for learning. On real text, language models memorize up to a certain capacity, at which point they substitute unintended memorization for generalization, and begin to learn general, reusable patterns as opposed to sample-level specifics. Our framework shows that double descent phenomenon begins to occur at this point, when the data size exceeds the model capacity in bits. Finally, we use our results to predict a scaling law for membership inference performance based on model capacity and dataset size. We show that membership inference follows a clean relationship based on model capacity and dataset size: bigger models can memorize more samples, and making datasets bigger makes membership inference harder. Our scaling laws extrapolate to larger models, and predict most modern language models are trained on too much data to do reliable membership inference on the average datapoint. Figure 5: Bits memorized across training. This particular model is a GPT-style transformer with 6.86M parameters and a capacity of 23.9 MB. Figure 6: Capacity in bits-per-parameter for models trained on synthetic data. We estimate \alpha=3.64 bits-per-parameter for GPT models trained in half precision. 2 Memorization, intended and unintended When a model \theta=L(x) is trained using a training algorithm L and a dataset x\sim X , some information is transferred from the sample x to the model \theta . A key question in the memorization literature is determining how much of this stored information is intended versus unintended. In this work, we aim to provide a rigorous definition of memorization that satisfies certain properties: 1. Separation from generalization. Our notion of unintended

Read the complete text