Deep sequence models tend to memorize geometrically; it is unclear why.
Shahriar Noroozizadeh, Vaishnavh Nagarajan, Elan Rosenfeld, Sanjiv Kumar · 2025
Read Deep sequence models tend to memorize geometrically; it is unclear why. by Shahriar Noroozizadeh, Vaishnavh Nagarajan, Elan Rosenfeld, Sanjiv Kumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26745.txt.
Introduction
Shahriar Noroozizadeh, Vaishnavh Nagarajan, Elan Rosenfeld, and Sanjiv Kumar report a surprising way deep sequence models store facts. The usual picture is associative memory: a lookup of things seen together. The team identifies geometric memory instead, where a model's embeddings encode global relations among all entities, even ones that never appeared together in training.
Noroozizadeh and his three co-authors show this storage is powerful: it can turn a hard multi-step reasoning task into a one-step navigation task. Oddly, the geometry arises even when it is more complex than brute-force lookup, and standard pressures do not explain it. By linking the effect to Node2Vec, a graph embedding method, they trace it to a spectral bias that arises naturally. They also point out headroom for making Transformer memory more strongly geometric.
Catalog record
| CREATOR | Shahriar Noroozizadeh, Vaishnavh Nagarajan, Elan Rosenfeld, Sanjiv Kumar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 121 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26745
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deep sequence models tend to memorize geometrically; it is unclear why. Shahriar Noroozizadeh, Vaishnavh Nagarajan, Elan Rosenfeld, Sanjiv Kumar Abstract In sequence modeling, the parametric memory of atomic facts has been predominantly abstracted as a brute-force lookup of co-occurrences between entities. We contrast this associative view against a geometric view of how memory is stored. We begin by isolating a clean and analyzable instance of Transformer reasoning that is incompatible with memory as strictly a storage of the local co-occurrences specified during training. Instead, the model must have somehow synthesized its own geometry of atomic facts, encoding global relationships between all entities, including non-co-occurring ones. This in turn has simplified a hard reasoning task involving an ℓ \ell -fold composition into an easy-to-learn 1 1 -step geometric task. From this phenomenon, we extract fundamental aspects of neural embedding geometries that are hard to explain. We argue that the rise of such a geometry, despite optimizing over mere local associations, cannot be straightforwardly attributed to typical architectural or optimizational pressures. Counterintuitively, an elegant geometry is learned even when it is not more succinct than a brute-force lookup of associations. Then, by analyzing a connection to Node2Vec , we demonstrate how the geometry stems from a spectral bias that—in contrast to prevailing theories—indeed arises naturally despite the lack of various pressures. This analysis also points to practitioners a visible headroom to make Transformer memory more strongly geometric. We hope the geometric view of parametric memory encourages revisiting the default intuitions that guide researchers in areas like knowledge acquisition, capacity, discovery and unlearning. 1 Introduction Neat representations materialize when a deep network needs to compress redundancies in data. On the other extreme, if the data is a set of atomic facts (like the birth date of a celebrity), the network would simply memorize these incompressible associations as a lookup table [ 126 , 17 ] . These two narratives have so far roughly guided our understanding of how neural networks fit sequential data. This paper fleshes out a third phenomenon in sequence modeling—glimpses of which were observed recently in Khona et al. 2024 , Ye et al. 2025 —where a neat representation materializes from memorizing incompressible atomic facts. We argue that this phenomenon implies a geometric form of parametric memory, one that encodes global relationships between non-co-occurring entities. We point out that this is dramatically different from the common associative view of parametric memory as storing mere local atomic co-occurrences. From this geometry of atomic facts, we extract aspects of neural geometries that are hard to explain, posing fundamental questions about memorization in deep sequence models. To these questions, we offer some preliminary answers. Our discussion starts at a seemingly tangential point. We begin by consolidating a fragmented set of recent demonstrations [ 76 , 43 , 172 , 158 ] that the Transformer exhibits some level of implicit in-weights reasoning , i.e., reasoning over knowledge from the weights without emitting an explicit chain of thought. We sharpen these results by crafting a scenario where this ability is less expected, plays out vividly, and can be cleanly isolated and analyzed. Specifically, we study path-finding on path-star graphs, a (symbolic) implicit reasoning task. The task was adversarially designed [ 12 ] to cause failure of next-token trained deep sequence models—both the Transformer [ 156 ] and Mamba [ 54 ] alike. Whereas in the original task, the model is given the graph in-context, here we make the model memorize the graph’s edges in its weights. Whereas in the original task, the models spectacularly fail to learn path-finding even on small graphs, in our in-weights task, our models (both the Transformer and Mamba) succeed even on massive graphs. Figure 1: Associative vs. geometric memory of models trained on various graphs. Parametric memory in deep sequence models is often abstracted as if co-occurrences from atomic facts are stored in a weight matrix, while the co-occurring entities themselves are embedded arbitrarily [ 161 , 46 , 70 , 183 , 23 , 111 , 22 , 17 , 160 ] . ( left ). This associative view is hard to reconcile with our observation that the Transformer learns implicit reasoning on an in-weights path-star graph. In practice, the learned embeddings ( middle ) reflect global structure inferred from the local co-occurrences, implying a geometric view of memory. Both these views are valid ways to fit the training data, but it is not straightforward why the latter prevails during Transformer optimization. When associative memory is explicitly prohibited, as in a Node2Vec model ( right ), a more elegant geometry materializes. This points to a clear headroom to improve the geometric nature of a Transformer’s memory. Details of the Transformer architecture used for this visualization are provided in § B.2.2 . Similar geometries for Mamba and neural networks are presented in § C.3 . The success in the in-weights path-star task, we argue, is hard to reconcile within the associative view of parametric memory, a convenient and highly effective abstraction of neural network memory, popular in literature [ 17 , 22 , 70 , 161 , 46 , 183 , 23 , 111 , 160 ] . In this abstraction, knowledge is stored in a matrix, say {\bf{W}}_{\tt assoc} such that if u and v co-occur in the same context during training, the quantity {\bf{\Phi}}(v)^{T}{\bf{W}}_{\tt assoc}{\bf{\Phi}}(u) is large, for some arbitrary embedding {\bf{\Phi}} . With such a data structure, the path-star task requires composing the aforementioned matrix operation \ell times (for path length \ell ). Unless there is step-wise supervision for each composition, learning the \ell -fold composition should intuitively be a daunting needle-in-the-haystack task demanding \exp(\ell) time. By the specific design of our task, every possible form of step-wise guidance is eliminated . Yet our model appears to find the needle. This apparent paradox begins to be resolved due to salient observations in Khona et al. 2024 , Ye et al. 2025 that the Transformer represents the nodes of their graphs with some notion of distance. We generalize this observation, pointing out that this has profound implications and is in fact non-trivial to explain. First, the existence of such node embeddings implies that atomic facts can be stored in an altogether distinct paradigm of parametric memory, a geometric one, which is in dramatic contrast to the associative one. In the simplest form of this view, co-occurrences are not simply stored in a matrix {\bf{W}} , but rather neatly encoded within the embeddings, say {\bf{\Phi}}_{\tt geom} (see Fig. 1 ). In fact, these embeddings are so carefully arranged that they encode global relationships without explicit supervision to do so: even if entities u and v never appeared in the same context, the dot-product {\bf{\Phi}}_{\tt geom}(u)^{T}{\bf{\Phi}}_{\tt geom}(v) captures the model’s own notion of multi-hop distance between the two. This has implications on reasoning since what seemed to be a hard-to-learn \ell -fold composition of local associations, now becomes an easy-to-learn 1 -step geometric task for the model. While helping make sense of our paradox, the observed geometry raises fundamental questions. First, there must be competition between the two parametric memories, both equally valid solutions to the training objective; why does the geometric prevail over the associative? To some readers, a