Too Big to Think: Capacity, Memorization, and Generalization in Pre‑Trained Transformers

Joshua Barron, Devin White · 2025

Read Too Big to Think: Capacity, Memorization, and Generalization in Pre‑Trained Transformers by Joshua Barron, Devin White (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09099.txt.

Introduction

Joshua Barron and Devin White train small transformer models from scratch to probe how size trades memory against generalization, in a paper from two thousand twenty-five. Two toy tasks split the skills: arithmetic stretch for generalization and fact recall for memory. Models of many sizes faced each task alone and then both together.

Small models stretched arithmetic to unseen cases but failed to memorize facts. Large models memorized yet failed to stretch. A mid-size model drifted toward memory as well. Trained on both tasks, no model of any size could stretch. The authors say pretraining itself may force a choice between the two modes.

Catalog record

CREATORJoshua Barron, Devin White
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-09099

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Too Big to Think: Capacity, Memorization, and Generalization in Pre‑Trained Transformers Joshua Barron, Devin White Abstract The relationship between memorization and generalization in large language models (LLMs) remains an open area of research, with growing evidence that the two are deeply intertwined. In this work, we investigate this relationship by pre-training a series of capacity-limited Transformer models from scratch on two synthetic character-level tasks designed to separately probe generalization (via arithmetic extrapolation) and memorization (via factual recall). We observe a consistent trade-off: small models extrapolate to unseen arithmetic cases but fail to memorize facts, while larger models memorize but fail to extrapolate. An intermediate-capacity model exhibits a similar shift toward memorization. When trained on both tasks jointly, no model (regardless of size) succeeds at extrapolation. These findings suggest that pre-training may intrinsically favor one learning mode over the other. By isolating these dynamics in a controlled setting, our study offers insight into how model capacity shapes learning behavior and offers broader implications for the design and deployment of small language models. † † affiliationnotice: \icmlDisclaimer 1 Introduction Large language models (LLMs) have shown impressive capabilities in both factual recall and pattern-based reasoning ( Hurst et al. 2024 ; Team et al. 2023 ) . However, in classical machine learning, there is a long-standing tradeoff between memorization and generalization, where improving one often comes at the cost of the other. We hypothesize that this same trade‐off governs LLM pre‐training. We propose that limited-capacity models are biased toward learning underlying rules, whereas larger models tend to memorize training examples at the expense of extrapolation. To explore this hypothesis, we draw on the classical machine learning view of regularization, where models with limited capacity are naturally encouraged to generalize. Our experiments are based on nanoGPT ( Karpathy 2024 ) , a lightweight implementation of the GPT architecture ( Radford et al. 2019 ) that serves as the starting point for our study. We introduce targeted modifications to the codebase that support a broader range of model sizes and configurations, allowing us to systematically evaluate models across a range of capacity constraints. We train a sequence of Transformer models ( Vaswani et al. 2017 ) with increasing size, ranging from minimal single-layer networks to a multi-layer GPT-style model. All experiments are conducted at the character level, ensuring that generalization and memorization arise from raw sequence modeling rather than token-level abstractions. These models are evaluated on two synthetic datasets specifically designed to isolate generalization and memorization. The first dataset assesses algorithmic generalization by holding out specific arithmetic expressions during training. The second dataset tests pure memorization using a small set of factual statements about capital cities. Together, these tasks allow us to isolate and analyze how model capacity influences learning behavior. Our results show a consistent pattern: smaller models are able to learn and extrapolate the underlying arithmetic rules but fail to memorize factual content, while larger models excel at factual recall but struggle with extrapolation. Intermediate models display a potential shift between these behaviors. Across all configurations, no model achieves both generalization and memorization simultaneously. These findings suggest that the trade-off between memorization and generalization is an inherent consequence of model scale and task composition, rather than a byproduct of overfitting or optimization. Our contributions include: (1) a controlled setup that cleanly isolates generalization and memorization, (2) an empirical analysis showing how increasing capacity shifts learning from rule-based generalization to pattern memorization, and (3) evidence that simultaneous exposure to both task types suppresses generalization across all model sizes. 2 Background The trade-off between memorization and generalization in LLMs remains an open question. Nezhurina et al. ( Nezhurina et al. 2025 ) demonstrate that even state-of-the-art models (e.g., GPT-4 ( Achiam et al. 2023 ) , Claude 3 Opus) can fail catastrophically on simple reasoning tasks explicitly designed to test generalization. These failures raise concerns about the robustness of LLM reasoning and suggest a potential tension between generalization and pattern memorization. In contrast, Tirumala et al. ( Tirumala et al. 2022 ) show that large models can memorize substantial portions of their training data without exhibiting classical signs of overfitting, suggesting that memorization does not always interfere with generalization. This apparent contradiction highlights the need for controlled experiments that isolate these behaviors during model training. During pre-training, LLMs absorb vast and diverse corpora to build broad linguistic competence and world knowledge, but this process can conflate rote memorization with deeper generalization. Wang et al. ( Wang et al. 2025a ) show that knowledge‑intensive tasks benefit from distributional memorization, while reasoning‑intensive tasks benefit from distributional generalization. Maltoni et al. ( Maltoni & Ferrara 2024 ) show that nanoGPT, when pre-trained on binary‑arithmetic data, generalizes to 7‑bit addition and multiplication rather than merely memorizing examples. Together, these results suggest that systematic generalization in LLMs depends not only on the scale of pre-training, but also critically on the nature of the training distribution and the way tasks are represented. After pre-training, LLMs are typically adapted via supervised fine-tuning (SFT) ( Pareja et al. 2024 ) on labeled examples. While effective at aligning model outputs, SFT can reinforce patterns in the fine-tuning data and amplify memorization. Chu et al. ( Chu et al. 2025 ) find that SFT models often replicate training examples verbatim, especially on narrow or repetitive datasets. Zeng et al. ( Zeng et al. 2024 ) further show that memorization increases with longer training and smaller datasets. These behaviors raise concerns about the robustness and generalization of SFT-aligned models. The final stage of LLM training is reinforcement learning. Reinforcement learning with human feedback (RLHF) ( Ouyang et al. 2022 ) improves generalization via preference alignment, but can reduce output diversity ( Kirk et al. 2024 ) and often reverses memorization from SFT ( Chu et al. 2025 ) . Recently, reinforcement learning with verifiable rewards (RLVR) has emerged as a scalable alternative. Wang et al. ( Wang et al. 2025b ) show that a single verifiable example can dramatically improve math reasoning. To investigate LLM dynamics in a controlled setting, we take a bottom up approach focused on model pre-training. 3 Methods 3.1 Datasets We design two fully synthetic, character-level benchmarks to isolate generalization versus memorization: Arithmetic: This dataset consists of input–output pairs representing simple arithmetic expressions for addition and subtraction: <a+b=c> and <a-b=c> , where a,b\in\{0,\dots,9\} ; ”<” and ”>” are start and stop characters respectively. To test for true algorithmic extrapolation, all expressions involving both 5 and 7 (i.e., 5+7 , 7+5 , 5-7 , 7-5 ) are withheld from training and validation. Each line in the dataset is 9 characters long; shorter expressions are padded accordingly. We use a block size of 9 to provide full context per math sequence and a batch size of 882. This corresponds to half of the total dataset (196 expressions \times 9 characters = 1764), ensuring that each batch covers a representative subset of the data without passing the full dataset each iteration. Capital Cities: This dataset contains 50 factual statements of the form “Capital of X is Y,” designed to probe pure memorization. The same 50 examples are used for both training and validation. The longest sequence is 24 characters, so we set block size to

Read the complete text