Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modeling

Bojian Yin, Shurong Wang, Haoyu Tan, Sander Bohte, Federico Corradi, Guoqi Li · 2026

Read Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modeling by Bojian Yin, Shurong Wang, Haoyu Tan, Sander Bohte, Federico Corradi, Guoqi Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02226.txt.

Introduction

Bojian Yin, Shurong Wang, Haoyu Tan, and colleagues rethink how recurrent networks handle long signals in this 2026 paper. Audio and video hold brief key moments inside long stretches of quiet or noise. Standard recurrent networks update their memory at every step, even when nothing changes. That habit erases old memory and blocks learning from distant events.

Their network, the suRNN, gives each neuron a switch that opens only for useful input. During dull stretches, memory stays exactly as it was, untouched. Learning signals can then flow straight back across long gaps. On long-range tests and text data, suRNNs matched or beat far larger models. They did so while staying much cheaper for long-term storage of past events.

Catalog record

CREATORBojian Yin, Shurong Wang, Haoyu Tan, Sander Bohte, Federico Corradi, Guoqi Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61229
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.02226

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modeling Bojian Yin, Shurong Wang, Haoyu Tan, Sander Bohte, Federico Corradi, Guoqi Li Abstract Real-world sequential signals, such as audio or video, contain critical information that is often embedded within long periods of silence or noise. While recurrent neural networks (RNNs) are designed to process such data efficiently, they often suffer from “memory decay” due to a rigid update schedule: they typically update their internal state at every time step, even when the input is static. This constant activity forces the model to overwrite its own memory and makes it hard for the learning signal to reach back to distant past events. Here we show that we can overcome this limitation using Selective-Update RNNs (suRNNs), a non-linear architecture that learns to preserve its memory when the input is redundant. By using a neuron-level binary switch that only opens for informative events, suRNNs decouple the recurrent updates from the raw sequence length. This mechanism allows the model to maintain an exact, unchanged memory of the past during low-information intervals, creating a direct path for gradients to flow across time. Our experiments on the Long Range Arena, WikiText, and other synthetic benchmarks show that suRNNs match or exceed the accuracy of much more complex models such as Transformers, while remaining significantly more efficient for long-term storage. By allowing each neuron to learn its own update timescale, our approach resolves the mismatch between how long a sequence is and how much information it actually contains. By providing a principled approach to managing temporal information density, this work establishes a new direction for achieving Transformer-level performance within the highly efficient framework of recurrent modeling. Figure 1 : suRNN architecture and the selective update mechanism : (a) Transition of suRNN from time step t-1 to t : unlike conventional RNNs that apply a uniform, time-agnostic transition, suRNN adopts a per-neuron, time-dependent binary gate \mathbf{g}_{t}\in\{0,1\}^{H} . This allows each neuron to dynamically choose whether to preserve or update its state at each step. (b) Selective-update mechanism for H=3 neurons: the per-neuron gating logic functions as an RC circuit with a switch. When the switch is off ( \mathbf{g}_{t,i}=0 , bottom path), neuron \mathbf{h}_{t,i} bypasses the update and remains unchanged; when the switch is on ( \mathbf{g}_{t,i}=1 , top path), neuron \mathbf{h}_{t,i} undergoes a standard non-linear update. 1 Introduction Recurrent Neural Networks (RNNs) are a classical and foundational framework for sequential modeling ( 7 ; 20 ; 28 ) . Theoretically, RNNs are capable of representing arbitrary state-dependent computations through Turing-complete recurrence ( 23 ; 22 ) . Practically, RNNs are attractive for \mathcal{O}(1) memory complexity during autoregressive inference and low-latency decoding, making them natural candidates for streaming, on-device, and ultra-long sequence deployment ( 10 ; 27 ) . Despite this elegant formulation and favorable inference efficiency, RNNs have been largely surpassed by Transformers ( 39 ) and modern State Space Models (SSMs) ( 14 ; 38 ) , particularly in tasks where capturing long-range dependencies is essential. Real-world sequential data (e.g., text, audio, or video) exhibit highly non-uniform information density: significant events are typically sparse and separated by long intervals of redundancy or noise. Ideally, a sequence model should allocate computation effort in proportion to information density rather than raw sequence length. However, many widely used models remain information-agnostic in this regard. For instance, Transformers compute pairwise interactions across time steps via \mathcal{O}(L^{2}) attention map, while SSMs integrate history through long-range convolutions. Yet, these models have access to distant context but they process each time step uniformly, expending equal effort on informative and redundant content. RNNs face an even worse situation where they apply a dense, uniform, and time-agnostic transformation at every time step, forcing the valuable historical information to be constantly overwritten and re-encoded. This synchronization of computation with sequence length prevents the model from distinguishing meaningful transitions from non-informative ones, resulting in unnecessary state updates that are wasteful and can actively degrade long-term retention. To bridge this mismatch, we propose Selective-Update RNNs (suRNNs) (Fig. 1 a), an architecture designed to alleviate the long-term dependency problem through temporal sparsity at the neuron level. Our core insight is that stable long-sequence retention requires two properties: (i) an identity map to preserve states during non-informative intervals, and (ii) a mechanism to perform informative updates selectively. suRNNs instantiate these by replacing conventional continuous gating with a time-dependent, neuron-wise binary gate \mathbf{g}_{t,i}\in\{0,1\} , thereby decoupling effective recurrent updates from raw sequence. This modification directly overcomes the time-agnostic update schedule of conventional RNNs, and injects the temporal awareness to the model through diverse gating schedules. To handle the non-differentiability of the discrete gates, we adopt the Straight-Through Estimator (STE) ( 1 ) , drawing a direct parallel to surrogate gradient methods in Spiking Neural Networks ( 30 ) . As depicted in Fig. 1 b, selective update induces a dual-mode dynamics. When the gate is switched off ( \mathbf{g}_{t,i}=0 ), the i -th neuron acts as an ideal memory cell, preserving the exact same state from the previous time step; when the gate is switched on ( \mathbf{g}_{t,i}=1 ) for the i -th neuron, a standard nonlinear update is applied. Consequently, this mechanism induces a sparse credit-assignment effect, in which the number of non-identity transitions along a gradient path scales with the number of informative events (gate activations) rather than the raw sequence length. Unlike prior adaptive computation approaches — such as the global halting logic of Adaptive Computation Time ( 11 ) , the discrete step-skipping of Skip RNNs ( 3 ) , or the rigid schedules of Clockwork RNNs ( 24 ) — suRNNs implement sparsity at the finest possible resolution: the individual neuron. These earlier methods often struggle to scale to larger models and longer sequences, as they are hindered by the optimization instability of auxiliary ponder losses or the sequential bottlenecks of global control flow ( 3 ) . By contrast, suRNNs integrate sparsity directly into the recurrent updates, enabling heterogeneous effective timescales to emerge naturally across the hidden state. This per-neuron selectivity avoids global contextual synchronization on all neurons, ensures stable training on large-scale datasets, and concentrates representational capacity exclusively on salient transitions. Our main contributions are as follows: • Selective Update Mechanism : We introduce suRNN, an architecture that replaces continuous gating with binary selection to enable exact state preservation during intervals of informational stasis. • Sparse Credit Assignment : We apply the STE to recurrent gating, inducing a functional gradient path that scales with the number of salient events rather than raw sequence length, effectively bypassing the vanishing/explosion gradient problem. • Strong Empirical Performance : We show that suRNN outperforms existing RNNs in streaming tasks such as psMNIST, sCIFAR, and selective copy. Furthermore, suRNN matches or exceeds the accuracy of modern Transformers on the Long Range Arena (LRA) and WikiText benchmarks while maintaining \mathcal{O}(1) inference efficiency. 2 Background 2.1 Recurrent Neural Network Conceptually, recurrent neural networks (RNNs) map an input sequence \{\mathbf{x}_{t}\}_{t=1}^{T} to hidden states \{\mathbf{h}_{t}\}_{t=1}^{T} , where \mathbf{h}_{t}\in\mathbb{R}^{H} summarizes the history up to time t . A standard recurrent layer evolves according to: \mathbf{h}_{t}=f_{\theta}(\mathbf{h}_{t-1},\mathbf{x}_{t}),\quad\text{for}\;1\leq t\leq T, (1) where f_{\theta}(\cdot,\cdot) is parameterized by \theta and generates the next state \mathbf{h}_{t} from the previous state \mathbf{h}_{t-1} and

Read the complete text