Exponential Dynamic Energy Network for High Capacity Sequence Memory

Arjun Karuvally, Pichsinee Lertsaroj, Terrence J. Sejnowski, Hava T. Siegelmann · 2025

Read Exponential Dynamic Energy Network for High Capacity Sequence Memory by Arjun Karuvally, Pichsinee Lertsaroj, Terrence J. Sejnowski, Hava T. Siegelmann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24965.txt.

Introduction

Arjun Karuvally, Pichsinee Lertsaroj, Terrence J Sejnowski, and Hava T Siegelmann stretch the energy view of memory nets to hold chains of memories, not just still ones. Old Hopfield nets store still maps well, but a chain needs moves from one memory to the next. Their net, EDEN, pairs a high store energy net with a slow side group that shifts the land over time.

From the fast land forms, they work out how long a memory holds before it slips. A sharp shift shows between a still phase and a move phase. Store size is mapped as a task of space size and slip rate. The frame aims at sequence memory with rules, not just recall.

Catalog record

CREATORArjun Karuvally, Pichsinee Lertsaroj, Terrence J. Sejnowski, Hava T. Siegelmann
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24965

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exponential Dynamic Energy Network for High Capacity Sequence Memory Arjun Karuvally, Pichsinee Lertsaroj, Terrence J. Sejnowski, Hava T. Siegelmann Abstract The energy paradigm, exemplified by Hopfield networks, offers a principled framework for memory in neural systems by interpreting dynamics as descent on an energy surface. While powerful for static associative memories, it falls short in modeling sequential memory, where transitions between memories are essential. We introduce the Exponential Dynamic Energy Network (EDEN), a novel architecture that extends the energy paradigm to temporal domains by evolving the energy function over multiple timescales. EDEN combines a static high-capacity energy network with a slow, asymmetrically interacting modulatory population, enabling robust and controlled memory transitions. We formally derive short-timescale energy functions that govern local dynamics and use them to analytically compute memory escape times, revealing a phase transition between static and dynamic regimes. The analysis of capacity, defined as the number of memories that can be stored with minimal error rate as a function of the dimensions of the state space (number of feature neurons), for EDEN shows that it achieves exponential sequence memory capacity 𝒪 ⁡ ( γ N ) \mathcal{O}(\gamma^{N}) , outperforming the linear capacity 𝒪 ⁡ ( N ) \mathcal{O}(N) of conventional models. Furthermore, EDEN’s dynamics resemble the activity of time and ramping cells observed in the human brain during episodic memory tasks, grounding its biological relevance. By unifying static and sequential memory within a dynamic energy framework, EDEN offers a scalable and interpretable model for high-capacity temporal memory in both artificial and biological systems. 1 Introduction Memory is a crucial element of cognition that is essential for learning, reasoning, and decision-making. Understanding and replicating the human ability to store and recall information is a long-term challenge in both biological and artificial intelligence (AI). The energy paradigm, introduced by Hopfield and Amari 40 years ago, revolutionized memory modeling by characterizing the dynamical behavior of neural networks using an energy landscape Hopfield1982NeuralNA ; Amari2004NeuralTO . According to the energy paradigm, a stimulus instantiates a network state on an energy landscape. The neurons then interact with each other such that the state travels down the landscape until a minimum is reached. This minimum state is defined as the memory of the network. The energy approach to memory modeling represented a significant advancement of our scientific understanding of memory by offering an intuitive understanding of network dynamics, with added theoretical guarantees of stability. The disadvantage was that the number of memories that can be reliably stored was only a small fraction of the number of neurons McEliece1987TheCO ; Folli2016OnTM ; Amit1985SpinglassMO and scaled linearly with the increase in neurons. This limited the applicability of energy networks despite their theoretical advantages. Further, the dynamics on the energy surface guaranteed a single final memory, precluding any temporal behavior in the memories. Further research in improving these networks proceeded in two independent directions. In one direction, researchers sought to develop techniques to improve the limited memory capacity of the original neural network by proposing modifications to how neurons interact in the network. In the second direction, researchers sought to create alternative formulations to energy function such that sequences and temporal aspects of memory can also be modeled with similar theoretical guarantees. Figure 1: Schematic Model and Energy Landscape Behavior of Dynamic Energy Networks : A. The dynamic energy network, EDEN, has asymmetrically interacting slow neurons providing information about the next memory in the sequence to the two fast neural populations. B. Static energy-based networks are used as models of human associative memory where a single memory associated with a provided stimulus is recalled. EDEN, without the slow population, is a static energy network that retrieves a single memory from a collection of stored memories. The system’s state ( v_{i} ), represented by the blue ball, descends the energy surface until a stable memory (energy minimum at state “2") is reached. After retrieval, the state of the system does not change and stays at “2". C. Dynamic energy networks enable associative sequence memory, where the associated memory along with its sequential neighbors are recalled. In EDEN, the energy surface changes in response to the state of the system, causing the minima of the energy surface to change over time (from “2" to “3"), resulting in transitions between memories. Improving capacity has been central to the development of memory models. In the context of Hopfield networks, capacity is defined as the maximum number of memories that can be stored with minimal errors as a function of the number of dimensions in its state space (the number of visible neurons). Earlier studies revealed that the limited capacity of the classic Hopfield network was due to significant crosstalk between the memories resulting in energy functions with many spurious minima. A major breakthrough in capacity came with the introduction of higher order terms in the energy function that separated the contribution of each memory to the energy minimum Baldi1987NumberOS ; Chen1987HighOC ; Kanter1987AssociativeRO ; Gardner1987MulticonnectedNN ; Abbott1987StorageCO ; Horn1988CapacitiesOM ; Krotov2016DenseAM resulting in polynomial capacity scaling and dense networks - networks that store more memories than the number of neurons Krotov2021LargeAM . Further studies introduced exponential terms, greatly increasing memory capacity and enabling practical applications Widrich2020ModernHN ; Krotov2021LargeAM . Currently, energy networks are used in AI with applications in large-scale natural language processing Radford2018ImprovingLU ; Devlin2019BERTPO , computer vision Carion2020EndtoEndOD , and lifelong-learning systems Hayes2021ReplayID ; Acharya2020RODEORF as reliable external memory storage. Further, the self-attention mechanism in transformer architectures have been shown to be functionally equivalent to the exponential memory capacity network providing insights into their mysterious capabilities Vaswani2017AttentionIA ; Ramsauer2021HopfieldNI . The success story of high-capacity static energy networks demonstrates how utilizing the energy paradigm benefits advancement and practical applications. Despite these advancements in the energy paradigm, state-of-the-art networks are still restricted to retrieving single memories from a collection of stored memories. Reconciling the single stable memory states in the energy paradigm with the dynamic states required for modeling sequences remains a significant challenge Rabinovich2008TransientCD ; Durstewitz2008ComputationalSO ; BenShushan2017StabilizingPI ; Camera2019CorticalCV . Over the years, there have been several solutions proposed for the challenge. One proposal introduced networks combining symmetric interactions, asymmetric interactions, and delay signals to produce temporal behavior Kleinfeld1986SequentialSG ; Sompolinsky1986TemporalAI . These proposals succeeded in creating networks that exhibited sequential state transitions, but the energy paradigm could not be applied to these cases as occasionally the network traveled up the energy surface. Another proposal introduced noise into the dynamics for enabling transitions out of a memory basin of the energy surface Miller2010StochasticTB ; Jones2007NaturalSE ; Miller2016ItinerancyBA ; Braun2010AttractorsAN ; Asllani2018StructureAD ; Orhan2020ImprovedMI . The energy paradigm applied to these models revealed a lowering of the energy barrier between states as more noise is added to the system. Without theoretical insights obtained from the application of the energy paradigm, the modifications needed to improve sequence capacity could not be found. As a result, extant sequence networks have capacity much lower than the number of neurons. Developing an energy principle for temporal memory networks will enable memory researchers to develop networks that are capable and aligned with experimental data. It will also enable artificial intelligence researchers to develop capable external memory stores. Our work extends the energy paradigm to temporal

Read the complete text