On the Role of Hidden States of Modern Hopfield Network in Transformer

Tsubasa Masumura, Masato TakiGraduate School of Artificial Intelligence and Science, Rikkyo University, Japan {t_masumura,taki_m}@rikkyo.ac.jp · 2025

Read On the Role of Hidden States of Modern Hopfield Network in Transformer by Tsubasa Masumura, Masato TakiGraduate School of Artificial Intelligence and Science, Rikkyo University, Japan {t_masumura,taki_m}@rikkyo.ac.jp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20698.txt.

Introduction

Tsubasa Masumura and Masato Taki, of Rikkyo University, sharpen in 2025 a known link between two memory ideas. One is the modern Hopfield network, a model of associative memory borrowed from neuroscience. The other is self-attention, the core mechanism inside Transformer models. Earlier work showed the two coincide only under a simplifying approximation.

Masumura and Taki drop that approximation and find the match survives in fuller form. The price is one extra element: a hidden state taken from the Hopfield network itself, carried inside attention. Their mechanism, modern Hopfield attention, passes attention scores from the first layer through to the last. This attacks rank collapse, a failure where representations in deep Transformers blur into near-copies. In a plain GPT-2, token similarities pile up at maximum by layers 12 and 24, while the Hopfield version keeps them low. Accuracy improved on Vision Transformer and GPT tasks, with no new training parameters added.

Catalog record

CREATORTsubasa Masumura, Masato TakiGraduate School of Artificial Intelligence and Science, Rikkyo University, Japan {t_masumura,taki_m}@rikkyo.ac.jp
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-20698

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Role of Hidden States of Modern Hopfield Network in Transformer Tsubasa Masumura, Masato TakiGraduate School of Artificial Intelligence and Science, Rikkyo University, Japan {t_masumura,taki_m}@rikkyo.ac.jp Abstract Associative memory models based on Hopfield networks and self-attention based on key-value mechanisms have been popular approaches in the study of memory mechanisms in deep learning. It has been pointed out that the state update rule of the modern Hopfield network (MHN) in the adiabatic approximation is in agreement with the self-attention layer of Transformer. In this paper, we go beyond this approximation and investigate the relationship between MHN and self-attention. Our results show that the correspondence between Hopfield networks and Transformers can be established in a more generalized form by adding a new variable, the hidden state derived from the MHN, to self-attention. This new attention mechanism, modern Hopfield attention (MHA), allows the inheritance of attention scores from the input layer of the Transformer to the output layer, which greatly improves the nature of attention weights. In particular, we show both theoretically and empirically that MHA hidden states significantly improve serious problem of deep Transformers known as rank collapse and token uniformity. We also confirm that MHA can systematically improve accuracy without adding training parameters to the Vision Transformer or GPT. Our results provide a new case in which Hopfield networks can be a useful perspective for improving the Transformer architecture. * * footnotetext: These authors contributed equally to this work 1 Introduction The relationship between associative memory in Hopfield networks 25 ; 1 , which has attracted interest from neuroscientists, and Transformers 49 based on key-value memory that have been studied in machine learning has attracted interest from the research community 48 ; 45 ; 5 ; 51 ; 24 . One of the most interesting results is the finding in 43 ; 34 that translating modern Hopfield networks into neural networks yields the Transformer architecture that has been very successful in natural language processing 41 ; 11 and computer vision 14 . What, then, do more general modern Hopfield networks imply for deep learning? This paper gives a concrete answer to this question. Hopfield networks 25 ; 1 are a class of models for associative memory. Despite these interesting properties, classical Hopfield networks have the limitation of small storage capacity. Recently, 33 proposed Dense Associative Memory which can achieve storage capacity that scales exponentially or power-wise with respect to the number of neurons by introducing high nonlinearity 9 . These models with large storage capacity are collectively referred to as modern Hopfield networks 43 ; 34 . Figure 1: (a) The left figure shows the layer structure of Transformer architecture using modern Hopfield attention (MHA). As the hidden state \bm{H}_{n} propagates through each attention layer, information from the upper layer’s attention scores is reused in the lower layers. Attention score \bm{Q}_{n}\bm{K}_{n}^{\top} is accumulated in the hidden state of each layer, and this value is used for attention calculation. (b) A visualization of the token uniformity in layers 12 and 24 of GPT-2 (Medium) trained on the Wikitext103 dataset, showing a violin plot of the cosine similarity between the tokens. For GPT-2 in the left column, there is a strong peak at similarity 1, and both layers have a mode of 1. On the other hand, in the case of GPT-2 with MHA in the right column, the cosine similarity is kept low and the uniformity of the tokens is dramatically improved. Recent advances in Transformer architecture, including its application to language models, have led to significant advances in the study of self-attention mechanisms. These advances have also shed new light on Hopfield networks. In 43 , it was shown that the state update rules of modern continuous Hopfield networks (MCHNs) have a mathematical structure exactly equivalent to that of the self-attention mechanism. Furthermore, 34 developed this relationship theoretically, pointing out that the self-attention layer of Transformer coincides with the adiabatic limit of generic modern Hopfield networks. Thus, it is expected that there is a deep relationship between the Transformer’s architectural design and associative memory. Still the connection, however, lack fundamental understanding. Therefore, we consider the question: is it possible to interpret modern Hopfield networks without adiabatic approximation in terms of Transformers? The adiabatic limit approximation removes the hidden state dynamics from MCHN. In this paper, we show that maintaining this dynamics introduces a hidden state on the Transformer side, thereby creating a mechanism for the propagation of attention score information from the upper to the lower layers. By implementing this new attention mechanism into Transformer architectures, this paper introduces a new type of self-attention layer called modern Hopfield attention ( MHA ), as shown in Figure 1 (a). MHA does not require additional parameters, and the increase in computational complexity is very small. Nevertheless, simply using MHA instead of the usual attention layer, performance gains can be obtained in various natural language processing and image recognition tasks. Furthermore, we found that MHA effectively solves the problem known as rank collapse, or token uniformity, where Transformer’s tokens lose diversity as Figure 1 (b). These results indicate that ideas derived from the Hopfield network may provide a new perspective for Transformer research. In summary, our contributions are as follows: • By investigating the relationship between MCHN and Transformer beyond the adiabatic approximation, we showed that this correspondence can be further generalized. Based on the correspondence with MCHN, we proposed a new type of attention mechanism with hidden state, MHA. The MHA-based Transformer improves the nature of attention weights by sharing attention score information across layers. • By training Transformers with MHA, we experimentally showed that the MHA mechanism also contributes to the performance of the Transformers. In particular, we investigated image recognition with Vision Transformer and text generation tasks with GPT-2, and confirmed that MHA actually improves their performance. This method does not generate any additional parameters, and thus can lead to performance gains with only a small increase in computational complexity. • Theoretically and experimentally, we showed that the reason why MHA works so well as an alternative to self-attention is related to rank collapse. The hidden state of MHA is likely to enhance its performance by cleverly improving Transformer’s rank collapse as Figure 1 (b). Our results suggest that the Hopfield Networks can provide guidance for improving the Transformer architectures. 2 Related Works The relationship between the modern Hopfield network and the Transformer was investigated in 43 ; 34 , and various improvements and extensions have been made to the modern Hopfield network 37 ; 58 ; 29 ; 28 ; 44 ; 7 ; 24 ; 27 ; 26 ; 52 ; 22 ; 20 . In 43 , the authors demonstrate the fast convergence of the Modern Hopfield Network (MHN) and justify its use as a conventional module in Transformer-related architectures. On the other hand, in this study’s MHA, we propose a dynamic structure that maintains and updates hidden states across layers. More specifically, our MHA naturally incorporates Hopfield recursion into the Transformer layer structure, as “state accumulation and updating” are performed in each layer. Research on improving the design of Transformer architecture using this relationship has also been conducted in 23 . Unlike these studies, this paper

Read the complete text