Memorization to Generalization: Emergence of Diffusion Models from Associative Memory
Bao Pham ∗, Gabriel Raya ∗, Matteo Negri, Mohammed J. Zaki, Luca Ambrogioni, Dmitry Krotov · 2025
Read Memorization to Generalization: Emergence of Diffusion Models from Associative Memory by Bao Pham ∗, Gabriel Raya ∗, Matteo Negri, Mohammed J. Zaki, Luca Ambrogioni, Dmitry Krotov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21777.txt.
Introduction
Bao Pham and five colleagues reinterpret image-generating diffusion models as memory machines that overflow, in a 2025 paper. Dense associative memories store training items at the bottoms of energy valleys; overload the memory and new valleys appear that match no stored item. In memory research those spurious states are defects. In generation, the authors argue, they are the whole point.
Viewed through that lens, a diffusion model's sampling is memory retrieval. With little data it builds one attractor per example, pure memorization. As data grows past a critical load, spurious attractors emerge and generation begins. The team characterizes these states' basins, the curvature around them, and their computational behavior. The pattern held across architectures and datasets. The work reframes the memorization-versus-generalization debate as a phase transition with a named middle phase, giving theorists a handle on when models start creating.
Catalog record
| CREATOR | Bao Pham ∗, Gabriel Raya ∗, Matteo Negri, Mohammed J. Zaki, Luca Ambrogioni, Dmitry Krotov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21777
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Memorization to Generalization: Emergence of Diffusion Models from Associative Memory Bao Pham ∗, Gabriel Raya ∗, Matteo Negri, Mohammed J. Zaki, Luca Ambrogioni, Dmitry Krotov Abstract Hopfield networks are associative memory (AM) systems, designed for storing and retrieving patterns as local minima of an energy landscape. In the classical Hopfield model, an interesting phenomenon occurs when the amount of training data reaches its critical memory load — spurious states , or unintended stable points, emerge at the end of the retrieval dynamics, leading to incorrect recall. In this work, we examine diffusion models, commonly used in generative modeling, from the perspective of AMs. The training phase of diffusion model is conceptualized as memory encoding (training data is stored in the memory). The generation phase is viewed as an attempt of memory retrieval. In the small data regime the diffusion model exhibits a strong memorization phase, where the network creates distinct basins of attraction around each sample in the training set, akin to the Hopfield model below the critical memory load. In the large data regime, a different phase appears where an increase in the size of the training set fosters the creation of new attractor states that correspond to manifolds of the generated samples. Spurious states appear at the boundary of this transition and correspond to emergent attractor states, which are absent in the training set, but, at the same time, have distinct basins of attraction around them. Our findings provide: a novel perspective on the memorization-generalization phenomenon in diffusion models via the lens of AMs, theoretical prediction of existence of spurious states, empirical validation of this prediction in commonly-used diffusion models. † † ∗ denotes equal contribution. 1 Introduction Hopfield networks are the simplest energy-based models that conceptualize memories as attractor states corresponding to local minima of their energy function, and the memory retrieval as dynamical convergence towards those attractor states [ 1 , 2 , 3 ] . Recently, they have seen a resurgence of interest due to advances in their memorization capacity. Notably, Dense Associative Memories (DenseAMs), which are extensions of Hopfield networks with super-linear memory capacity [ 4 , 5 ] have paved the way for more sophisticated associative memory systems [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] , driven in part by their connection to the attention mechanism in transformers [ 19 , 20 , 13 ] and to neurobiology [ 20 , 21 ] . Simultaneously, generative diffusion models [ 22 ] have gained considerable popularity, due to their flexibility and accuracy in modeling high-dimensional distributions for a variety of domains —such as image generation [ 23 , 24 , 25 ] , audio [ 26 , 27 , 28 ] , video synthesis [ 29 , 30 , 31 , 32 ] , and other scientific applications. However, despite their effectiveness, diffusion models pose challenges related to privacy and security, as concerns grow about their tendency to replicate training data [ 33 , 34 , 35 ] . Such matters consequently emphasize the need for further understanding of memorization and generalization behaviors in diffusion models. Recent works [ 36 , 37 , 38 ] have begun establishing theoretical connections between DenseAMs and generative diffusion models, offering a foundation for bridging the two fields. Specifically, it was shown that the logarithm of the probability of the generated samples in diffusion models can be interpreted as the energy function of a commonly used DenseAM model with the softmax activation function. This makes it possible to apply theoretical tools developed for associative memory to better understand the computational properties of diffusion models. A possible key that can further bridge these two systems and highlight the properties of the memorization-to-generalization transition in diffusion models is the phenomenon of spurious patterns . Historically considered as detrimental to pattern recall [ 39 , 40 , 41 ] , spurious patterns can be interpreted as combinations or interpolations of stored patterns, hinting at the network’s ability to synthesize new patterns from existing training data. This blending of fundamental memories resembles the generalization process in generative models, where learned representations are used to generate novel outputs. In this way, spurious patterns offer a fascinating framework for exploring the balance between memorization — where models store exact patterns from training data — and generalization, where they use underlying features to create genuinely new samples [ 42 ] . Studying the conditions under which spurious patterns emerge and how they contribute to the network’s behavior can thus shed light on generalization in both associative memory and contemporary generative models. Previous works have explored memorization in generative models through various approaches. For instance, [ 43 , 44 ] proposed general methods to measure memorization, while [ 33 , 34 , 45 , 46 ] examined the memorization capacity in diffusion models as a function of the size of the training data. Other works have focused on understanding generalization, such as the theoretical estimates of the generalization gap [ 47 ] , and the spectral analyses [ 48 , 49 , 50 ] . A recent study [ 51 ] proposed that generalization in diffusion models arises as a result of the combination of limited capacity and structural constraints neural network, which results in predictable patterns at the boundary between memorization and generalization. Other recent studies [ 52 , 50 , 49 ] explored memorization and generalization through the relationship between the dimensionalities of the ground-truth manifold and the manifold that is learned by the model. Meanwhile, [ 53 ] explored the evolution of the generated density as a function of the training time to provide an analysis on how mode collapse occurs and leads to the loss of diversity in the generated density. Finally, [ 54 ] and [ 55 ] characterized memorization in diffusion models as “condensation” phase transitions known in the statistical physics of spin glasses and random energy models. Although these studies shed some light on memorization and generalization, they do not fully explore the model’s behavior during the transition between these two phases. Figure 1: A simple illustration depicting the change in the energy landscape as the size of the training dataset is increased. In the small data regime, the model memorizes the training data points as local minima of the energy. When the amount of training data exceeds the memory capacity of the model, spurious patterns are formed and training data points are no longer energy minima. Subsequent increase of the training set size leads to the generalization phase, which is defined by the formation of continuous manifold of the low energy states. Contributions. Most of the mentioned works on memorization-to-generalization transition tackled it in a generalization-centric fashion. Namely, the memorization phenomenon is typically viewed as a “small side effect” of the diffusion models. In our work we adopt a complimentary approach. From the beginning, we cast the diffusion modeling pipeline into the associative memory framework. The training phase of the diffusion model is conceptualized as writing into the memory operation. The generation phase is viewed as an attempt of memory recall . This recall can be successful, resulting