From Memorization to Reasoning in the Spectrum of Loss Curvature

Jack Merullo, Srihita Vatsavaya, Lucius Bushnaq, Owen Lewis · 2025

Read From Memorization to Reasoning in the Spectrum of Loss Curvature by Jack Merullo, Srihita Vatsavaya, Lucius Bushnaq, Owen Lewis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24256.txt.

Introduction

Jack Merullo and three co-authors show how memorization in transformer models can be separated from general learning by looking at loss curvature. Prior work found that memorized training points sit in much sharper parts of the loss landscape than other points. Sorting weight components by curvature can thus reveal memorized content without explicit labels.

Based on this, the team builds a weight editing method that suppresses recitation of untargeted memorized data more effectively than a recent unlearning method, while keeping perplexity lower. The decomposition works in both language models and vision transformers. The paper frames a spectrum from memorization to reasoning in terms of curvature.

Catalog record

CREATORJack Merullo, Srihita Vatsavaya, Lucius Bushnaq, Owen Lewis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24256

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Memorization to Reasoning in the Spectrum of Loss Curvature Jack Merullo, Srihita Vatsavaya, Lucius Bushnaq, Owen Lewis Abstract We characterize how memorization is represented in transformer models and show that it can be disentangled in the weights of both language models (LMs) and vision transformers (ViTs) using a decomposition based on the loss landscape curvature. This insight is based on prior theoretical and empirical work showing that the curvature for memorized training points is much sharper than non-memorized, meaning ordering weight components from high to low curvature can reveal a distinction without explicit labels. This motivates a weight editing procedure that suppresses far more recitation of untargeted memorized data more effectively than a recent unlearning method (BalancedSubnet), while maintaining lower perplexity. Since the basis of curvature has a natural interpretation for shared structure in model weights, we analyze the editing procedure extensively on its effect on downstream tasks in LMs, and find that fact retrieval and arithmetic are specifically and consistently negatively affected, even though open book fact retrieval and general logical reasoning is conserved. We posit these tasks rely heavily on specialized directions in weight space rather than general purpose mechanisms, regardless of whether those individual datapoints are memorized. We support this by showing a correspondence between task data’s activation strength with low curvature components that we edit out, and the drop in task performance after the edit. Our work enhances the understanding of memorization in neural networks with practical applications towards removing it, and provides evidence for idiosyncratic, narrowly-used structures involved in solving tasks like math and fact retrieval. 1 1 1 Code is available at https://github.com/goodfire-ai/memorization_kfac 1 Introduction To what degree do models generate genuinely new knowledge, as opposed to simply reassembling snippets of data memorized from their training sets? Much discussion about the current utility and future prospects of large neural networks has centered on this question. On the one hand, a growing trophy case of model accomplishments on novel tasks near the frontier of human capabilities argues strongly against memorization strictly construed as an explanation of the full range of model behavior. But on the other hand, recent papers have argued convincingly that models do in fact memorize large volumes of their training data verbatim, and a surprisingly large fraction of naturalistic interactions with language-model chatbots contain significant verbatim recitations ( Aerni et al. 2024 ; Carlini et al. 2022 ; Stoehr et al. 2024 ) , behaviors which have significant implications for copyright and data privacy ( Karamolegkou et al. 2023 ; Carlini et al. 2019 ; Shokri et al. 2017 ) . Models thus seem to have a significant and frequently used capacity for both memorization and generalization. The question is not whether models generalize or recite, but rather how these capabilities are represented, how they interact and trade off ( Nguyen & Reddy 2025 ) , and how they might be modulated. These are the questions we seek to address in this work. We build on existing work that characterizes memorization in terms of the curvature of the loss landscape as a function of a model’s weights ( Foret et al. 2021 ; Hochreiter & Schmidhuber 1997 ; LeCun et al. 1989 ; Hassibi et al. 1993 ; Keskar et al. 2017 ; Garg et al. 2024 ; Ravikumar et al. 2024 ; Jeon et al. 2024 ; Kim et al. 2023 ) . This prior work argues theoretically and empirically that the loss landscape has highly curved directions in the neighborhood of memorized points, while generalization corresponds to flatter basins. We exploit this insight while extending it in several ways. We study models’ behavior in aggregate, rather than for individual examples. Figure 1 (left). Are there model structures that account for memorization and generalization across large swaths of training data? We find that there are: in both language and vision models, the eigenbasis of the approximated Hessian of weight matrices uncovers distinct disentanglement of memorization and generalization, in a way that extends across a range of subdistributions of memorized data. In extending from studying per-example to bulk memorization, we propose a novel inversion of the previous interpretation of loss curvature: while individual memorized points are associated with high curvature, the direction of curvature varies across examples, meaning that, averaged across multiple examples, memorization directions are actually flatter than generalizing directions, which maintain a consistent moderate curvature across points. We propose an effective recitation-reduction technique based on ablating memorized directions in weight space. We compare our results to a recent supervised memorization removal technique (BalancedSubnet (BSN); Sakarvadia et al. 2025 ) and find that our method matches the suppression of the targeted forget set of BSN, and is similarly robust to stress tests ( Huang et al. 2024 ) , while removing far more unseen memorized data and achieving lower perplexity 5.2 . Figure 1: Overview of our approach. We collect activations and gradients from a sample of training data (a), which allows us to approximate loss curvature w.r.t. a weight matrix using K-FAC (b). We decompose these weight matrices into components (each the same size as the matrix), ordered from high to low curvature. In language models, we show that data from different tasks interacts with parts of the spectrum of components differently (c). We go beyond pure memorization and generalization and find curvature signatures of intermediate behaviors like fact retrieval and arithmetic. Figure 1 (right). Many classical analyses of memorization and generalization have studied classification models, where the distinction between the two is clear. In modern language models, though, there is a much richer spectrum of behaviors between the poles of pure memorization (verbatim recitation of long passages) and pure generalization (de novo reasoning). For example, facts like “Paris is the capital of France” are memorized in the sense that they are specific pieces of information that the model knows, but are general in the sense that they are not tied to specific syntactic instantiations seen in the training data. Similarly, arithmetic and logical reasoning test a model’s ability to generate novel inferences, but the inference rules and base axioms may be remembered from training. Going beyond prior work, we extend the loss curvature analysis to cover these reasoning types, situating them on a continuum between memorization and generalization. We find that, besides memorization, arithmetic and factual recall exhibit weight activation with low-curvature weight directions and are sensitive to their removal. On the other hand, logical reasoning, which does not necessarily require precise recall or calculation is robust to removing flat directions, which in some cases even improving performance. Our general approach is illustrated in Figure 1 . For all of our analyses, we measure curvature with the Kronecker-Factored Approximate Curvature (K-FAC) approximation to the model’s Hessian, a computational technique that makes curvature analyses tractable at scale. K-FAC has been widely used for Hessian approximation in other settings, particularly for natural gradient optimization, but to our knowledge, our use of it to study memorization and generalization via curvature is novel. 2 Related Work Memorization, especially as a special case of overfitting, is a widely studied topic in both modern and classical machine learning. In the modern era of extremely large overparameterized models, there is particular interest in quantifying the

Read the complete text