MeMo: Towards Language Models with Associative Memory Mechanisms
Fabio Massimo Zanzotto Elena Sofia Ruzzetti Giancarlo A. Xompero, Leonardo Ranaldi Davide Venditti Federico Ranaldi, Cristina Giannone Andrea Favalli Raniero Romagnoli · 2025
Read MeMo: Towards Language Models with Associative Memory Mechanisms by Fabio Massimo Zanzotto Elena Sofia Ruzzetti Giancarlo A. Xompero, Leonardo Ranaldi Davide Venditti Federico Ranaldi, Cristina Giannone Andrea Favalli Raniero Romagnoli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12851.txt.
Introduction
Fabio Massimo Zanzotto and eight co-authors present a 2025 paper on a language model design that memorizes text directly instead of only learning from it. Normal transformers absorb facts slowly through training. The authors argue memory should come first, with learning built on top.
Their design, MeMo, stores token sequences in layered associative memories, like a web of linked records. By design, what is stored stays visible and can be edited, even deleted on request. Tests with one layer and many layers show how much the system can hold verbatim. The work opens a path to models whose knowledge can be audited and corrected by hand.
Catalog record
| CREATOR | Fabio Massimo Zanzotto Elena Sofia Ruzzetti Giancarlo A. Xompero, Leonardo Ranaldi Davide Venditti Federico Ranaldi, Cristina Giannone Andrea Favalli Raniero Romagnoli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12851
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MeMo: Towards Language Models with Associative Memory Mechanisms Fabio Massimo Zanzotto Elena Sofia Ruzzetti Giancarlo A. Xompero, Leonardo Ranaldi Davide Venditti Federico Ranaldi, Cristina Giannone Andrea Favalli Raniero Romagnoli Abstract Memorization is a fundamental ability of transformer-based Large Language Models, achieved through learning. In this paper, we propose a paradigm shift by designing an architecture to memorize text directly, bearing in mind the principle that memorization precedes learning. We introduce MeMo, a novel architecture for language modeling that explicitly memorizes sequences of tokens in layered associative memories. By design, MeMo offers transparency and the possibility of model editing, including forgetting texts. We experimented with the MeMo architecture, showing the memorization power of the one-layer and the multi-layer configurations. 1 Introduction Transformer-based Large Language Models achieve unrivaled performance in language modeling by learning to capture and represent complex sequential dependencies from statistical patterns through extensive training phases that iteratively refine their weights to best approximate natural language. This has triggered significant interest in gaining a better understanding of the inner workings of these models, focusing on how these models generalize and capture structure between similar samples in terms of syntactic dependencies Vig and Belinkov (2019) , compositional relations Hupkes et al. (2020) ; Zanzotto et al. (2015) concerning the quantity Reizinger et al. (2024) and quality Yang et al. (2024) of training data. Besides generalization, a key component of Transformers’ success is the ability to memorize data while learning Ranaldi et al. (2023b) ; Ranaldi et al. (2023a) . Indeed, earlier work investigated this other side of learning. While Carlini et al. (2023) ; Mahdavi et al. (2024) demonstrated evidence of memorization, Kharitonov et al. (2021) ; Mahdavi et al. (2024) studied how the internal components lead to memorization, and Kim et al. (2023) estimated the boundary between generalization and memorization, providing an estimation on their storage capacity. Memorization is not inherently a drawback in language models because it plays a crucial role in handling factual knowledge, which is important for question answering, summarization, or information retrieval. This factual recall relies on a delicate balance. While generalization helps capture patterns and unseen relationships in data, memorization ensures that models retain critical and exact information when required. Recent research has highlighted that memorization capability can be effectively harnessed using concepts rooted in associative memories Kohonen (1972) ; Anderson (1972) - a system designed to link inputs to specific outputs and offers a structured and transparent way to store and retrieve information. By leveraging associative memory mechanisms, researchers have proposed strategies to post-edit LLMs Meng et al. (2022) ; Meng et al. (2023a) , enabling control over what is memorized, how it is stored, and how it is accessed, enhancing their reliability in fact-based tasks. In this paper, we propose a paradigm shift by designing Language Models based on a different principle: memorization proceeds learning. By using associative memories, we build MeMo, a novel architecture for language modeling that explicitly memorizes sequences of tokens in layered associative memories. MeMo leverages correlation matrix memories Kohonen (1972) ; Anderson (1972) , the concept that tokens and sequences of tokens can be represented as random vectors Plate (1995) ; Sahlgren (2005) , and the Johnson-Lindestrauss Transform to embed larger vectors in smaller spaces by preserving their distances Johnson and Lindenstrauss (1984) . By design, MeMo offers transparency and the possibility of model editing, including forgetting texts. We experimented MeMo, showing the memorization power of single and multi-layer architecture. 2 Preliminaries and Background Representing words or tokens in small random vectors is the first important step in building language models with neural network architectures. Using random vectors is a standard technique. Indeed, random vectors are used in random indexing Sahlgren (2005) in information retrieval to reduce the document vector space and in distributed representations for neural networks as a convenient way to determine a set of vectors to represent sets of different tokens Plate (1995) or structures Zanzotto and Dell’Arciprete (2012) ; Zanzotto and Ferrone (2017) ; Zanzotto et al. (2020) . Moreover, random vectors are used to initialize weight matrices in any language-oriented application in neural networks, including the initialization of transformers Vaswani et al. (2017) to build large language models from scratch. Multivariate Gaussian random vectors have the important property of being able to generate sets E of nearly orthogonal unitary vectors that can form an approximate base of the space R^{n} in a smaller space R^{d} Johnson and Lindenstrauss (1984) . Each token t is then represented with a distinct vector in \mathbf{t}\in E , and the two following properties hold with a probability larger than 1-\delta : \begin{array}[]{cc}\|\mathbf{a}^{T}\mathbf{b}\|<\epsilon&\text{if }a\neq b\\ 1-\epsilon<\mathbf{a}^{T}\mathbf{b}<1+\epsilon&\text{if }a=b\end{array} where a and b are tokens and \mathbf{a} and \mathbf{b} are vectors representing those tokens in the reduced space R^{d} . By using the Johnson-Lindestrauss Lemma Johnson and Lindenstrauss (1984) , it is possible to find a lower bound of how large d should be in order to host n vectors given the approximation \epsilon and the probability factor \delta (see Appendix A). In less precise equations, the two properties can be rewritten as: \mathbf{a}^{T}\mathbf{b}\approx\left\{\begin{array}[]{cc}0&\text{if }a\neq b\\ 1&\text{if }a=b\end{array}\right. Using these vectors with their properties, it is possible to represent a bag-of-tokens B in a single vector \mathbf{t_{B}} offering the operation that approximately counts the number of times a token is in B . The vector \mathbf{t_{B}} is obtained by summing up vectors representing tokens in B and, then, the counting operation is: \mathbf{a}^{T}\mathbf{t_{B}}\approx k where k is the number of times a belongs to the bag B . Correlation matrix memories (CMMs) Kohonen (1972) ; Anderson (1972) are a powerful tool to store key-value (k_{i},v_{i}) pairs in distributed memories as the sum of outer products of the vectors representing the keys k_{i} and vectors representing the values v_{i} : C=\sum_{i=1}^{n}\mathbf{k}_{i}\mathbf{v}_{i}^{T} (1) These CMMs have been generally defined on one-hot representations Hobson (2011) and, eventually, reduced afterwards Kohonen (1972) . Then, to retrieve the value associated with a key, the matrix C should be multiplied with \mathbf{k}_{j}^{T} . As vectors \mathbf{k}_{i} are one-hot vectors, the following property holds: \mathbf{k}_{j}^{T}C=\mathbf{v}_{j} To optimize the construction of these CMM matrices, we use the correlated form: C=KV^{T}=\left[\begin{array}[]{cccc}|&|&&|\\ \mathbf{k}_{1}&\mathbf{k}_{2}&\ldots&\mathbf{k}_{n}\\ |&|&&|\\ \end{array}\right]\left[\begin{array}[]{cccc}-&\mathbf{v}^{T}_{1}&-\\ -&\mathbf{v}^{T}_{2}&-\\ &\vdots&\\ -&\mathbf{v}^{T}_{n}&-\\ \end{array}\right] To make CMMs practical, in MeMo, we use these memories along with the multivariate Gaussian vectors to represent keys and values. Hence, the generic property of this associative matrices is \mathbf{k}_{j}^{T}C\approx\mathbf{e}_{j}^{T}V=\mathbf{v}_{j} where \mathbf{e}_{j} is the onehot vector of the position j and \mathbf{k}_{j} and \mathbf{v}_{j} are multivariate Gaussian vectors to represent the key k_{j} and the value v_{j} . The idea behind correlation matrix memories has often been used to explain that feed-forward matrices are where Transformer architectures store most information Meng et al. (2023b) . In MeMo, CMMs become the cornerstone for defining a novel approach to building Language Models. Johnson-Lindestrauss Transform Dasgupta and Gupta (1999) , derived by using the Johnson-Lindestrauss Lemma (JLL) Johnson and Lindenstrauss (1984) , guarantees that it exists a linear transformation T_{d\times n} that transforms vectors in a bigger space R^{n} in vectors in a smaller space R^{d} by preserving their distance with an approximation \epsilon . Then, given two vectors \mathbf{a} and \mathbf{b} in R^{n} , the following property is guaranteed: \|\mathbf{a}-\mathbf{b}\|-\epsilon<\|T\mathbf{a}-T\mathbf{b}\|<\|\mathbf{a}-\mathbf{b}\|+\epsilon The JLL