Never Reset Again: A Mathematical Framework for Continual Inference in Recurrent Neural Networks

Bojian Yin, Federico Corradi · 2024

Read Never Reset Again: A Mathematical Framework for Continual Inference in Recurrent Neural Networks by Bojian Yin, Federico Corradi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15983.txt.

Introduction

Bojian Yin and Federico Corradi free recurrent neural networks from their reset button, in a 2024 paper. Recurrent networks read streams step by step, but their inner state saturates, so engineers periodically wipe it, a reset that demands knowing where inputs begin and end and wastes computation.

Their fix is a training loss, not a hardware change: cross-entropy blended with a divergence term, weighted by how informative each input is. The network learns to hold meaningful signals and let noise wash through, keeping its state stable across very long sequences. Across several recurrent architectures the reset-free models beat their reset-based counterparts, especially on continual tasks. Streaming systems, from sensors to subtitles, could run without waiting for boundaries that may never come.

Catalog record

CREATORBojian Yin, Federico Corradi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15983

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Never Reset Again: A Mathematical Framework for Continual Inference in Recurrent Neural Networks Thanks: The authors gratefully acknowledge the helpful discussions with Sander Bohte and Simeon Kanya. This work has been funded by the Dutch Organization for Scientific Research (NWO) with Grant KICH1.ST04.22.021 Bojian Yin, Federico Corradi Abstract Recurrent Neural Networks (RNNs) are widely used for sequential processing but face fundamental limitations with continual inference due to state saturation, requiring disruptive hidden state resets. However, reset-based methods impose synchronization requirements with input boundaries and increase computational costs at inference. To address this, we propose an adaptive loss function that eliminates the need for resets during inference while preserving high accuracy over extended sequences. By combining cross-entropy and Kullback-Leibler divergence, the loss dynamically modulates the gradient based on input informativeness, allowing the network to differentiate meaningful data from noise and maintain stable representations over time. t our reset-free approach outperforms traditional reset-based methods when applied to a variety of RNNs, particularly in continual tasks, enhancing both the theoretical and practical capabilities of RNNs for streaming applications. I Introduction Recurrent neural networks (RNNs) constitute the foundation for processing sequential data in deep learning and are the natural choice for time series modelling [ 1 , 2 ] . Over time, these architectures have evolved into specialized variants designed to address a variety of computational tasks. Notably, Gated Recurrent Units (GRUs) [ 3 ] optimize gradient flow, Linear Recurrent Networks including State Space Models (SSMs) [ 4 , 5 , 6 ] enable efficient and parallel training of large datasets, and Spiking Neural Networks (SNNs) implement biologically plausible, energy-efficient computation for edge devices [ 7 , 8 ] . Currently, RNN principles are applied to modern Large Language Models (LLMs), which, although based on transformer architectures [ 9 ] , act as RNNs during inference by processing tokens sequentially with cached states for computational efficiency [ 10 , 11 ] . Despite these architectural advancements in RNNs, fundamental challenges remain when continuously processing input data streams. In these scenarios, networks are required to generate stable representations and accurate predictions while processing uninterrupted data streams that substantially exceed typical training sequence lengths. This must be achieved without compromising inference performance. A major challenge in continual inference settings is the state saturation problem, which significantly limits the practical deployment of RNNs [ 12 , 13 ] . This challenge originates despite their theoretical Turing completeness, which highlights their potential to solve a diverse range of tasks [ 14 , 15 ] . This phenomenon manifests as progressive accuracy degradation during prolonged input exposure, affecting all RNN variants: traditional architectures, SNNs, and even LLM models [ 12 , 13 ] . The degradation is particularly apparent in continuous speech recognition and streaming tasks [ 16 , 17 ] , where performance declines gradually with exposure to long input speech sequences. The mechanism that leads to state saturation is the temporal accumulation of information within the hidden states. During continual operation, this accumulation generates interference between historical and present information, progressively degrading the network’s capacity to process new inputs [ 18 , 12 , 13 ] . Recent theoretical advances have provided insights into this limitation. Merrill [ 12 ] used language theory to formally analyze the behavior of saturated networks, while Panahi et al. [ 19 ] explained the implicit additive properties of RNNs by incorporating biologically-inspired saturation bounds. The significance is further highlighted by Paassen et al.’s [ 20 ] that have demonstrated that RNNs can emulate any finite state machine by mapping neurons to states. Current methods to address state saturation involve state collapse prevention via continual pre-training on longer sequences [ 13 ] and dynamic state resets using an extra action selection circuit [ 17 ] . Other methods use periodic or dynamic hidden state resets but struggle with the need for synchronization with input boundaries, often missing in real-world scenarios [ 21 ] . While distinct sample boundaries are manageable in controlled settings, they’re typically absent in practical streaming and edge deployments [ 22 , 17 ] . Despite these efforts, state saturation still challenges RNNs during long inferences, impacting their ability to process continuous data streams reliably. These observations motivate our central research question: “Can we develop a method for training RNNs that eliminates the need for resetting during continual inference?” This study offers a mathematical approach to the state saturation problem, eliminating the need for hidden state resets. We present an adaptive loss function that adjusts learning dynamically, using categorical cross-entropy to assess the target patterns and Kullback-Leibler (KL) divergence [ 23 ] to build a uniform distribution for noisy or irrelevant input. This approach enables autonomous output probability adjustment while preserving hidden state continuity. The main contributions of this work are: • a formal characterization of how prolonged input sequence affects RNNs dynamics in continual inference, including vanilla RNNs, GRU, SSMs, SNNs. • a mathematical analysis demonstrating saturation prevention and gradient flow maintenance in the absence of hidden state resets. • the development of a loss function that combines cross-entropy and KL divergence terms and enables stable and accurate continual inference without state reset. Fig. 1: Computational graphs of different recurrent architectures. II Background The sequential modelling performance of recurrent architectures is fundamentally constrained by and also benefits from their hidden dynamics [ 20 , 12 , 13 ] . This section examines the theoretical foundations and use of state reset mechanisms, contextualizing their role in mitigating saturation phenomena. Sequential data processing in deep learning is described by the general recurrent framework \mathbf{h}_{t}=f(\mathbf{h}_{t-1},\mathbf{x}_{t}) , where \mathbf{h}_{t} represents the hidden state at time t , and f(\cdot) denotes a learnable state transition function operating on the previous state \mathbf{h}_{t-1} and current input \mathbf{x}_{t} . This framework is commonly used in RNNs and for applications in many computational tasks. For example, vanilla RNNs implement this framework through \mathbf{h}_{t}=\tanh(\mathbf{W}_{h}\mathbf{x}_{t}+\mathbf{U}_{h}\mathbf{h}_{t-1}+\mathbf{b}_{h}) , where \{\mathbf{W}_{h},\mathbf{U}_{h},\mathbf{b}_{h}\} constitute the learnable parameters. To solve the vanishing gradient problem inherent in vanilla RNNs, GRU units have been introduced with adaptive gating mechanisms: \mathbf{r}_{t}=\sigma(\mathbf{W}_{r}\mathbf{x}_{t}+\mathbf{U}_{r}\mathbf{h}_{t-1}+\mathbf{b}_{r}) and \mathbf{z}_{t}=\sigma(\mathbf{W}_{z}\mathbf{x}_{t}+\mathbf{U}_{z}\mathbf{h}_{t-1}+\mathbf{b}_{z}) , facilitating controlled information flow through \mathbf{h}_{t}=(1-\mathbf{z}_{t})\odot\mathbf{h}_{t-1}+\mathbf{z}_{t}\odot\tanh(\mathbf{W}_{h}\mathbf{x}_{t}+\mathbf{U}_{h}(\mathbf{r}_{t}\odot\mathbf{h}_{t-1})+\mathbf{b}_{h}) . Additionally, SSMs introduce a continuous-time perspective because they formulate the time dynamics as \dot{\mathbf{h}}(t)=\mathbf{A}\mathbf{h}(t)+\mathbf{B}\mathbf{x}(t) , \mathbf{y}(t)=\mathbf{C}\mathbf{h}(t)+\mathbf{D}\mathbf{x}(t) , where \{\mathbf{A},\mathbf{B},\mathbf{C},\mathbf{D}\} represent learnable state-space parameters for long-range dependency. In addition, SNNs introduce bio-inspired principles and temporal processing by relying on the neuron’s membrane potential dynamics \mathbf{u}_{t}=\alpha\mathbf{u}_{t-1}+\sum_{i}w_{i}\mathbf{s}_{t-1}^{i} , coupled with a threshold-based spike generation mechanism \mathbf{s}_{t}=H(\mathbf{u}_{t}-\theta) , where H denotes the Heaviside function and \theta represents the firing threshold. Each architectural variant in this unified framework offers unique characteristics in modeling capacity, biological plausibility, and computational efficiency, allowing for tailored solutions in sequential data processing. II-A Formalization of Reset Mechanisms In general, Reset techniques in RNNs can be formalized by state transformation operations. The reset function \Phi:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} operates on hidden states h_{t}\in\mathbb{R}^{d} according to: h_{t}=\Phi(h_{t-1},r_{t})=r_{t}\odot h_{t-1} (1) where r_{t}\in\{0,1\}^{d} denotes a binary reset mask, and \odot denotes the Hadamard product. This formulation requires a priori knowledge of sequence boundaries—an assumption that proves problematic in continuous processing paradigms [ 24 , 25 ] . This reset operation breaks the recurrent dynamic by enforcing \mathbf{h}_{t}\leftarrow\mathbf{h}_{0} , where \mathbf{h}_{0} typically follows either zero initialization ( \mathbf{h}_{0}=\mathbf{0} ) or random initialization ( \mathbf{h}_{0}\sim\mathcal{N}(0,\sigma^{2}\mathbf{I})

Read the complete text