Yet another exponential Hopfield model
Linda Albanese, Andrea Alessandrelli, Adriano Barra, Peter Sollich · 2025
Read Yet another exponential Hopfield model by Linda Albanese, Andrea Alessandrelli, Adriano Barra, Peter Sollich (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06905.txt.
Introduction
Memory networks that can store hugely many patterns get a new variant from Linda Albanese and three co-authors in a 2025 neural networks paper. Hopfield models are linked memories: store patterns, then fetch them back from partial cues. A recent exponential family raised storage capacity a lot, and this paper proposes another member built differently.
The cost function is built from exponentials of plain quadratic losses, which naturally favors setups that recall a stored pattern perfectly. The model is not a mean-field system, yet its dynamics and retrieval traits remain open to analysis and match the first exponential model. A signal-to-noise argument shows stored patterns stay stable fixed points of the zero-temperature dynamics up to hugely many patterns. Wider basins of attraction shrink the load somewhat, but capacity keeps its exponential scaling, and a separate derivation confirms the prefactors.
Catalog record
| CREATOR | Linda Albanese, Andrea Alessandrelli, Adriano Barra, Peter Sollich |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-06905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Yet another exponential Hopfield model Linda Albanese, Andrea Alessandrelli, Adriano Barra, Peter Sollich Abstract We propose and analyze a new variation of the so-called exponential Hopfield model , a recently introduced family of associative neural networks with unprecedented storage capacity. Our construction is based on a cost function defined through exponentials of standard quadratic loss functions, which naturally favors configurations corresponding to perfect recall. Despite not being a mean-field system, the model admits a tractable mathematical analysis of its dynamics and retrieval properties that agree with those for the original exponential model introduced by Ramsauer and coworkers. By means of a signal-to-noise approach, we demonstrate that stored patterns remain stable fixed points of the zero-temperature dynamics up to an exponentially large number of patterns in the system size. We further quantify the basins of attraction of the retrieved memories, showing that while enlarging their radius reduces the overall load, the storage capacity nonetheless retains its exponential scaling. An independent derivation within the perfect recall regime confirms these results and provides an estimate of the relevant prefactors. Our findings thus complement and extend previous studies on exponential Hopfield networks, establishing that even under robustness constraints these models preserve their exceptional storage capabilities. Beyond their theoretical interest, such networks point towards principled mechanisms for massively scalable associative memory, with potential implications for both neuroscience-inspired computation and high-capacity machine learning architectures. 1 Introduction Within the statistical mechanics community, in the first formalization of the spontaneous pattern recognition and associative memory capabilities shown by large assemblies of neurons interacting à la Hebb , huge efforts were required in the 1980s by Amit, Gutfreund and Sompolinsky (AGS) to prove that the celebrated Hopfield model (effectively the harmonic oscillator for neural networks performing these tasks Hopfield ) was able to handle a maximal number of patterns P growing linearly in the number of neurons N in the neural network (i.e. P=\alpha N for some \alpha\in\mathbb{R}^{+} ) AGS1 ; AGS2 . After the AGS milestone, based on the recognition that the Hopfield model can be viewed as a pair-wise spin glass (with the N McCulloch \& Pitts neurons then playing the role of Ising spins), it was soon pointed out – by Baldi and Venkatesh Baldi from a computer science perspective and by Elizabeth Gardner Gardner from the point of view of statistical mechanics – that p -spin glasses equipped with Hebbian couplings are natural generalizations of the Hopfield model to many-body interactions. The authors of Refs. Baldi ; Gardner discovered that the p -spin maximal storage capacity was far larger than for the pair-wise Hopfield model, scaling as P=\gamma N^{p-1} for some \gamma\in\mathbb{R}^{+} , as a consequence of the abundance of synaptic connections in these dense networks. For the Hopfield case p=2 , this of course recovers the linear scaling of P with N . In the decades since then, countless techniques have been produced in the literature to deal with and formalize the emergent properties of these dense neural networks and, nowadays, we have a satisfactory picture of them, both from a statistical mechanics Diego ; FraDenso ; Albert ; LindaRSB ; LindaSuper ; LindaUnsup ; Lenka ; DanielinoDenso ; DaniDue ; Malatesta and from a computer science Krotov1 ; Krotov2 ; transformer ; DenseCapacity perspective. Indeed, mainly due to the efforts of Krotov and Hopfield, dense networks – referred to by them as dense associative memories – have attracted renewed interest (see e.g. KrotovNature ; KrotovNew2 ; KrotovNew3 ). Thus, it is not entirely surprising that in recent years the generalization towards an exponential storage capacity has been achieved. If we think of these exponential models as the sum of all the dense contributions of Hebbian p -spin glasses, then reaching an exponential N -dependence for the pattern storage capacity, while impressive, has a natural interpretation. There have been two pioneering contributions regarding such exponential models, one dealing with binary neurons ExpHop1 and the other with real-valued ones ExpHop2 . Focusing on the latter, Lucibello and Mézard recently studied various properties of the exponential Hopfield family by relying again on tools from glassy statistical mechanics LucibelloMezard (and in particular a systematic comparison with Derrida’s Random Energy Model Derrida ), see also ExpHop3 . To complement the work of Ref. LucibelloMezard we focus here on networks with binary neurons. In particular, we introduce and study a variation on the theme of an exponential Hopfield model equipped with such binary neurons. While not mean-field in nature, this network allows for a very simple mathematical analysis of its storage capacity if one focuses on the perfect recall regime . To estimate the maximal storage capacity within a signal-to-noise approach, we choose as initial condition for the noiseless neural dynamics – which we show is identical to the one that would be obtained from the model of Ref. ExpHop2 – precisely one of the patterns themselves. In such zero temperature dynamics, neural activities are effectively updated by steepest descent toward a local minimum of the network’s cost function, or Hamiltonian in statistical mechanics language. By adding more and more patterns to the network, we check for the stability of the input pattern to find out that the latter is preserved as long as P does not become too (exponentially) large (Sec. 2.1 ). To assess the size of the basins of attraction of the stored patterns, we provide as input for the neural dynamics a corrupted version of a pattern. The result shows that broad basins for pattern storage have a cost, namely, the larger the size of the basins of attractions, the smaller the number of patterns that can be stored. Crucially, however, even for finite radius of the basins of attraction, the exponential scaling of the storage capacity is preserved (Sec. 2.2 ). Finally, by considering the equilibrium conditions at zero temperature and relying upon the perfect retrieval constraint, we provide an independent proof of the exponential storage capacity, including an estimate of the prefactor (Sec. 2.3 ). Overall, while our analysis remains far from being an exhaustive picture of the function of the exponential Hopfield model we introduce, it helps to shed light on the exponential storage phenomenon of modern Hebbian neural networks. 2 Exponential Hopfield model: signal-to-noise approach Let us consider a set of N Ising neurons \bm{\sigma}=\{\sigma_{i}\}_{i=1,\ldots,N} where \sigma_{i}\in\{+1,-1\} and a set of P binary patterns \bm{\xi}=\{\xi_{i}^{\mu}\}_{i=1,\ldots,N}^{\mu=1,\ldots,P} , distributed as independent Rademacher variables, i.e. \displaystyle\mathbb{P}(\xi_{i}^{\mu})=\dfrac{1}{2}\delta_{\xi_{i}^{\mu},-1}+\dfrac{1}{2}\delta_{\xi_{i}^{\mu},+1}. (1) Before introducing cost and loss functions, we define control and order parameters that help us to quantify the information processing capabilities shown by this model. The control parameters are the noise T=1/\beta in the network (which we always take as T=0 ) and the storage \gamma of the network 1 1 1 Note that the parameter \gamma fine tunes the storage within the maximum storage scaling, which will be exponential in N in this context, vide infra , e.g. ( 18 ). while the order parameters are the P Mattis magnetizations m_{\mu},\ \mu\in\{1,...,P\} , defined as m_{\mu}:=\frac{1}{N}\sum_{i=1}^{N}\xi_{i}^{\mu}\sigma_{i}. (2) The cost function (or Hamiltonian in physics terminology) \mathcal{H}_{N}(\bm{\sigma}|\bm{\xi}) of the Exponential Hopfield Model that we study reads as \displaystyle\mathcal{H}_{N}(\bm{\sigma}|\bm{\xi}):=-N\displaystyle\sum\limits_{\mu=1}^{P}e^{-N\mathcal{L}^{\mu}_{N}(\bm{\sigma}|\bm{\xi})}=-N\displaystyle\sum\limits_{\mu=1}^{P}e^{-\frac{1}{2}||\xi^{\mu}-\sigma||^{2}}=-N\displaystyle\sum\limits_{\mu=1}^{P}e^{N\left(m_{\mu}-1\right)}\ . (3) Here we have also introduced the P standard loss function \mathcal{L}^{\mu}_{N}(\bm{\sigma}|\bm{\xi}):=\frac{1}{2N}||\bm{\xi}^{\mu}-\bm{\sigma}||^{2} for \mu\in\{1,...,P\} , to highlight that