mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling
Tristan Torchet, Christian Metzner, Laura Kriener, Melika Payvand · 2025
Read mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling by Tristan Torchet, Christian Metzner, Laura Kriener, Melika Payvand (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01829.txt.
Introduction
Tristan Torchet and co-authors design a lean sequence model that mixes delay based convolutions with a tiny gated memory, in a paper from 2025. Small edge devices must process streams that mix fast local wiggles with slow global trends, all inside a tight memory budget. Heavyweight models with fixed memory tend to sacrifice either quick precision or long range choice.
Their design spaces a convolution's taps by learnable delays, which theory shows acts like a delay embedding that rebuilds fast dynamics from partial views. A minimal gated recurrent unit rides alongside to carry the slow context. The split of labor aims to keep both timescales sharp without growing the footprint. The authors test the hybrid on sequence benchmarks against current compact models.
Catalog record
| CREATOR | Tristan Torchet, Christian Metzner, Laura Kriener, Melika Payvand |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01829
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling Tristan Torchet, Christian Metzner, Laura Kriener, Melika Payvand Abstract Edge devices for temporal processing demand models that capture both short- and long-range dynamics under tight memory constraints. While Transformers excel at sequence modeling, their quadratic memory scaling with sequence length makes them impractical for such settings. Recurrent Neural Networks (RNNs) offer constant memory but train sequentially, and Temporal Convolutional Networks (TCNs), though efficient, scale memory with kernel size. To address this, we propose mGRADE ( m inimally G ated R ecurrent A rchitecture with D elay E mbedding), a hybrid-memory system that integrates a temporal 1D-convolution with learnable spacings followed by a minimal gated recurrent unit (minGRU). This design allows the convolutional layer to realize a flexible delay embedding that captures rapid temporal variations, while the recurrent module efficiently maintains global context with minimal memory overhead. We validate our approach on two synthetic tasks, demonstrating that mGRADE effectively separates and preserves multi-scale temporal features. Furthermore, on challenging pixel-by-pixel image classification benchmarks, mGRADE consistently outperforms both pure convolutional and pure recurrent counterparts using ∼ \small\sim 20 % 20\% less memory footprint, highlighting its suitability for memory-constrained temporal processing at the edge. This highlights mGRADE’s promise as an efficient solution for memory-constrained multi-scale temporal processing at the edge. 1 Introduction Embedded systems hold immense potential for advanced temporal processing at the edge. In particular, embedded sequence models promise to enable low-latency, energy-efficient inference for real-time tasks like sensor data processing and autonomous control. However, the constrained memory capacity of embedded systems poses a significant challenge, especially for modeling data with long-range dependencies that require storing information over large temporal contexts. While State-of-the-Art sequence models, such as Transformers ( Vaswani et al. 2017 ) and Temporal Convolutional Networks ( Waibel et al. 1989 , TCN;) can capture multi-timescale information, they are ill-suited to embedded systems. The main reason is that their memory footprint grows with the sequence length, impeding real-time processing on long sequences. In contrast, RNNs, particularly gated variants like the Long Short-Term Memory ( Hochreiter and Schmidhuber 1997 ; Gers et al. 2000 , LSTM;) and the Gated Recurrent Unit ( Cho et al. 2014 ; Chung et al. 2014 , GRU;) , offer constant inference-time memory over input sequences of arbitrary length. However, since they are not parallelizable, they are inefficient to train on long sequences, which limits their practical utility for modeling long-range temporal dependencies. Given the gap between the tight memory requirements of embedded systems and current designs of advanced sequence models, we ask: Can we design a memory-efficient sequence model that is able to capture both long- and short-term dependencies for advanced temporal signal processing in embedded systems? To achieve this goal, we investigate a hybrid-memory model that synergistically combines the strengths of fixed-length context models, particularly TCNs, and gated parallelizable RNNs. We fuse a causal temporal convolution with learnable spacings, equivalent to a delay embedding, and a gated parallelizable recurrent unit into a single stackable layer. This combined model is called the m inimal G ated R ecurrent A rchitecture with D elay E mbeddings ( mGRADE ). mGRADE is effectively a hybrid-memory model where the temporal convolution acts as a fixed-size cache for short-term dependencies and the gated recurrence provides a compression of the signal’s long-term history. Its design satisfies the requirements of advanced processing on embedded systems since it (1) maintains constant memory complexity with respect to the sequence length at inference time, (2) is not limited to a fixed-length context window, and (3) can be efficiently trained in parallel regardless of the sequence length. We employ two types of synthetic tasks to theoretically demonstrate the functional contributions of the two components of mGRADE and how they complement each other. The first task shows how the temporal convolution caches delayed input copies, enabling the recurrent hidden state representation to generalize better to unobserved input-coupled dynamics on short timescales ( Section 4.1 ). With the second task, we show how the gated recurrent component remembers long-range dependencies by maintaining recursive activations enabling associative recall over arbitrarily long sequence lengths ( Section 4.2 ). Finally, we demonstrate that these theoretical capabilities translate to real-world performance, with mGRADE achieving competitive results on sequential image modeling benchmarks, sequential MNIST ( LeCun and Cortes 2010 , sMNIST;) and sequential CIFAR (sCIFAR), with a \small\sim 20\% lower total memory footprint than comparable single-memory counterparts ( Section 5 ). Our theoretical arguments offer insights into the modeling capabilities of the combination of gated recurrence and temporal convolutions. The demonstrated empirical performance of our hybrid-memory approach versus purely single-memory models positions mGRADE as a compelling solution for high-performance sequence modeling in resource-constrained embedded systems. 2 Related Works This section surveys prior work that informs our hybrid-memory architecture. We proceed as follows: first we review gated recurrent networks, then convolution-based sequence models, and finally recent efforts to understand the functional contributions of the building blocks of larger architectures. Gated Recurrent Models . For many years, RNNs were virtually synonymous with sequence modeling. Gated RNNs, notably the LSTMs and GRUs, alleviate vanishing-gradient effects ( Bengio et al. 1994 ; Hochreiter and Schmidhuber 1997 ) through learned gating mechanisms that selectively control the flow of information and neuron state updates. Although effective, these models train sequentially and are therefore inefficient for training on very long sequences. Bradbury et al. 2017 and Martin and Cundy 2018 mitigate this bottleneck by removing the hidden-state dependence of the gates, which enables parallel training through a prefix scan ( Blelloch 1990 ) . Hierarchically Gated Recurrent Networks (HGRN; Qin et al. 2023 ) additionally introduce a hierarchical gating bias, initially closing gates more tightly in deeper layers, and thereby encouraging a hierarchical processing of time scales. A parallel line of inquiry, which considers parallelizable training a fundamental requirement, entirely decouples gating from recurrence. Adding a gated layer, such as a Gated Linear Unit (GLU; Shazeer 2020 ), allows the recurrent core to be linearized and parallelized. Early investigations from Mozer 1993 ; Mikolov and Zweig 2012 ; Pachitariu and Sahani 2013 have recently converged in State-Space Models (SSMs; Gu et al. 2022 ). SSMs, related architectures such as Linear Recurrent Units (LRU; Orvieto et al. 2023 ) and HGRN utilize complex-valued parameters in addition to highly specific and precise initialization schemes, thus reducing their hardware compatibility and ease of use. Convolutional Models for Sequence Modeling . TCNs were originally proposed in ( Waibel et al. 1989 ) and popularized by Wavenet ( van den Oord et al. 2016 ) . They rely on the discrete convolution operation y[n]=(x*k)[n] ubiquitous in signal processing. The key component of this operation is the so-called kernel k ( Eq. 1 ). It is parametrized by two finite sets \Omega=\{w_{0},w_{1},...,w_{K}\ |\text{ }w_{i}\in\mathbb{R}\} and \Psi=\{p_{0},p_{1},...,p_{K}\ |\text{ }p_{i}\in\mathbb{N}\} of K elements each, representing the weights and positions of the non-zero elements in the kernel, respectively. Intuitively, this convolution of a signal can also be understood as a delay embedding, where the positions \Psi correspond to the applied delays. The maximum position value, \Gamma=max_{p}(\Psi) defines the highest applied delay and with that the local receptive field , representing the temporal span of historical inputs that influence