Physics-inspired Energy Transition Neural Network for Sequence Learning
Zhou Wu, Junyi An, Baile Xu, Furao Shen, Jian Zhao · 2025
Read Physics-inspired Energy Transition Neural Network for Sequence Learning by Zhou Wu, Junyi An, Baile Xu, Furao Shen, Jian Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03281.txt.
Introduction
Zhou Wu and co-authors revisit recurrent neural networks for sequence learning in a 2025 paper. Transformers now lead the field, but they gain much of their strength from comparing every pair of steps at once. The authors ask whether a well-built recurrent model, which reads a sequence step by step, can remember distant context just as well.
Their design borrows from physics models of how energy moves between states over time. The resulting network stores long-range information in its memory more effectively. Across several sequence tasks it beat transformer-based rivals, while using a simpler, cheaper structure. The study argues that recurrent designs still hold promise in areas now dominated by transformers. The authors present their model as a strong base for future recurrent work.
Catalog record
| CREATOR | Zhou Wu, Junyi An, Baile Xu, Furao Shen, Jian Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03281
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Physics-inspired Energy Transition Neural Network for Sequence Learning Zhou Wu, Junyi An, Baile Xu, Furao Shen, Jian Zhao Abstract Recently, the superior performance of Transformers has made them a more robust and scalable solution for sequence modeling than traditional recurrent neural networks (RNNs). However, the effectiveness of Transformer in capturing long-term dependencies is primarily attributed to their comprehensive pair-modeling process rather than inherent inductive biases toward sequence semantics. In this study, we explore the capabilities of pure RNNs and reassess their long-term learning mechanisms. Inspired by the physics energy transition models that track energy changes over time, we propose a effective recurrent structure called the “Physics-inspired Energy Transition Neural Network” (PETNN) . We demonstrate that PETNN’s memory mechanism effectively stores information over long-term dependencies. Experimental results indicate that PETNN outperforms transformer-based methods across various sequence tasks. Furthermore, owing to its recurrent nature, PETNN exhibits significantly lower complexity. Our study presents an optimal foundational recurrent architecture and highlights the potential for developing effective recurrent neural networks in fields currently dominated by Transformer. 1 Introduction Recurrent Neural Networks (RNNs), a sophisticated architecture for modeling sequences, have successfully addressed numerous challenges in time series prediction ( Connor et al., 1994 ) , machine translation ( Cho et al., 2014 ) , and other areas. Although Transformer models have outperformed RNNs in many areas over the past few years, RNNs remain a valuable research direction due to their inherent inductive biases, which align well with the structure of many real-world sequence data. For modeling sequences, RNNs can leverage preceding data to influence succeeding data, making them highly suitable for sequence data tasks. In addition, the model uses the same parameters at all time steps, which greatly reduces the complexity and computational cost. However, this structure also faces challenges: when dealing with long sequences, RNNs suffer from gradient vanishing problems, making it difficult to deal with long-term dependency problem ( Bengio et al., 1994 ) . To address it, researchers have proposed variants of RNN, such as LSTM ( Hochreiter & Schmidhuber, 1997 ) , and GRU ( Chung et al., 2014 ) , which alleviate the long-term dependency problem by gating mechanism. Nevertheless, these methods are still limited in complex and long sequence tasks. Later, Transformer with self-attention ( Vaswani et al., 2017 ) was introduced, enabling the model to simultaneously attend to all positions in the sequence. This effectively addresses the long-term dependency problem but also introduces higher complexity and computational burden. The core challenge in sequence modeling lies in effectively identifying the dependencies between tokens and retaining the essential information from the antecedent sequence. Currently, sequence models capture long-term dependencies in two primary ways: through direct modeling of dependencies, which is common in RNN-based models, and incorporating memory mechanisms, as seen in models like LSTM and GRU. While RNN-based models, due to their simpler structure, have lower time complexity, they rely heavily on memory, leading to the well-known issue of forgetting past information. LSTM and GRU address this by introducing dedicated memory and forget mechanisms, improving their ability to capture long-term dependencies. However, both models still rely on sigmoid-based gates for controlling memory, which can lead to information leakage over long sequences. This limitation makes them less effective compared to Transformer models, which capture long-range dependencies by modeling all pairwise interactions within the sequence. In this paper, we propose Physical-inspired Energy Transition Neural Network (PETNN) , which is a novel recurrent structure for effective sequence modeling. Initially, we observed that the energy transition model closely resembles the memory cell, since the energy change is a time function. This observation inspired the core idea of PETNN, i.e., using energy as the cell state for updates. To address this, we propose a novel memory mechanism in PETNN. Unlike LSTM, which relies on predefined memory and forget gates, PETNN empowers the neurons to autonomously determine what information to learn and update based on the energy state, a process we refer to as the self-selective information mixing method . This approach offers two key advantages: first, it allows neurons to dynamically control the proportion of information updates; second, it enables neurons to decide the storage duration of relevant information. Empirical evaluations of the PETNN model have been conducted across diverse domains, for example, sequence tasks, which have been monopoly by Transformer-based models. The results show that PETNN can compete with Transformer in a range of tasks, which demonstrates PETNN has great potential to be a strong alternative or complement to Transformer. Moreover, in image classification task, the results show that PETNN can be used not only for recurrent structures, but can also be easily embedded in general neural networks to solve non-sequential problems, laying a solid foundation for our future work. Our contributions are as follows: • We propose PETNN, a recurrent neural networks that incorporates wisdom from physics-based spatiotemporal model (energy transition model). • We analyze PETNN’s structure and demonstrate its advantages in handling long-term dependencies. • Results show that PETNN outperforms Transformers on several sequence tasks. Ablation experiments confirm the effectiveness of PETNN’s memory mechanism, providing new insights for sequence learning research. 2 Preliminary In this section, we provide a brief review of the energy transition mechanism ( Khan et al., 2019 ) , which is the prototype for PETNN. Following this, we introduce how to integrate the physical model into the machine learning model. In physics, atoms are characterized by discrete energy levels. When an external energy source interacts with an atom in its ground state, the atom absorbs energy and enters an excited state. However, this excited state is unstable, and the atom eventually releases the absorbed energy after a period of time ( Loudon, 2000 ) , returning to a lower energy state. This process, as shown in Figure 1 , widely applied in quantum physics, is exemplified by the photoelectric effect ( Einstein, 1905 ) . More details on the physics of energy transition can be found in Appendix A. Moreover, we refer interested readers to ( Griffiths & Schroeter, 2018 ) . Figure 1: Quantum theory explains the energy transition model through the processes of absorption and release. Inspired by this theory, our model captures and retains informative features in a way that resembles energy transition model theory. Just as excited states in energy transition models are deterministic, depending on the system’s properties and external input, each basic unit in our model adjusts its lifetime based on specific needs. During this retention period, information is accumulated in the mixed memory, similar to how energy input triggers a transition to an excited state in quantum systems, resulting in an elevated information state. The information updating process resembles the collapse of the quantum wave function ( Penrose, 1996 ) , where the updating function deterministically converts the mixed state into a specific state when the lifetime ends. After this period, part of information is discarded, akin to entropy reduction ( Wehrl, 1978 ) , in which redundant or irrelevant information is removed to maintain system efficiency and low entropy. Specifically, unlike the discrete energy levels in quantum systems, our model uses thresholds to distinguish between two states while processing continuous inputs, enabling dynamic and flexible adaptation. This design ensures that our