R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networks
Nicholas H. Barbara, Ruigang Wang, Ian R. Manchester · 2025
Read R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networks by Nicholas H. Barbara, Ruigang Wang, Ian R. Manchester (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01250.txt.
Introduction
Nicholas Barbara, Ruigang Wang, and Ian Manchester design a family of recurrent neural networks that are stable and calm by construction, in a 2025 paper. Recurrent networks process sequences step by step, and for control tasks they must not amplify noise or drift unstable. Their design wires a linear system together with a deep network whose response is capped, then tunes the weights directly inside that safe form.
Unlike a close rival design, nothing needs solving step by step at run time, so training and use run up to ten times faster on graphics hardware. Tests on system modeling, observer design, control, and image sequences matched the rival's quality while scaling to bigger models and longer inputs.
Catalog record
| CREATOR | Nicholas H. Barbara, Ruigang Wang, Ian R. Manchester |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-01250
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networks Nicholas H. Barbara, Ruigang Wang, Ian R. Manchester Abstract This paper presents the Robust Recurrent Deep Network (R2DN), a scalable parameterization of stable and robust recurrent neural networks for machine learning and data-driven control. We construct R2DNs as the feedback interconnection of a linear time-invariant system and a 1-Lipschitz deep feedforward network, and directly parameterize the weights so that our models are stable (contracting) and robust to input perturbations (Lipschitz) by design. Our parameterization uses a structure similar to the recurrent equilibrium network (REN), but without having to iteratively solve an equilibrium layer at each time-step. This speeds up model inference and training on GPUs, and makes it computationally feasible to scale up the network size and input sequence length in comparison to RENs. We compare R2DNs to RENs on representative problems in nonlinear system identification, observer design, learning-based feedback control, and sequential image classification. We find that training and inference are up to an order of magnitude faster with similar performance, and that they scale more favorably with respect to model expressivity. I Introduction Built on the power of deep neural networks (DNNs), modern machine learning has achieved significant progress in a wide range of fields across science and engineering [ 1 , 2 ] . However, despite their expressive function approximation capabilities, it is widely known that DNNs can be very sensitive to input perturbations, leading to brittle behavior and unexpected failures [ 3 , 4 , 5 ] . This is particularly true for applications involving dynamical systems, where it is natural to consider DNNs with internal states, such as recurrent neural networks (RNNs) [ 6 ] . In these cases, a fundamental challenge is to regulate not only the input sensitivity, but also the stability of an RNN’s internal states over time [ 7 ] . Several neural architectures have been proposed to address the limitations of RNNs by directly imposing constraints on their internal stability and input-output robustness [ 8 , 7 , 9 , 10 , 11 ] . One particular architecture is the recurrent equilibrium network , or REN [ 7 ] , which is structured as the feedback interconnection of a linear time-invariant (LTI) system and a scalar activation function, as shown in Figure 1(a) . The REN model class contains many common network architectures as special cases, including multi-layer perceptrons (MLPs), convolutional neural networks (CNNs), and residual networks (ResNets). RENs are designed to satisfy strong internal stability and input-output robustness properties via contraction [ 12 ] and integral quadratic constraints (IQCs) [ 13 ] , respectively. Most importantly, these stability properties are guaranteed by construction via a direct parameterization – a surjective mapping from a vector of learnable parameters \theta\in\mathbb{R}^{N} to the network weights and biases – which makes RENs compatible with standard neural network training algorithms and has enabled their use in a range of tasks including system identification [ 7 , 14 ] , observer design [ 15 ] , stability-certified reinforcement learning [ 16 , 17 ] , imitation learning [ 18 ] , and power electronics [ 19 ] . (a) REN architecture [ 7 ] . (b) R2DN architecture (ours). Fig. 1 : Block diagrams for the REN and the proposed R2DN architectures. We replace the scalar activation function \sigma with a 1-Lipschitz feedforward network, and modify the LTI system \bm{G} to remove direct feedthrough from w\rightarrow v . However, a key limitation of RENs is that calling the model requires iteratively solving an implicit equation for an equilibrium layer [ 20 , 21 ] , which can be computationally costly both during training and in applications that demand frequent model inference, such as control. Moreover, directly parameterizing RENs with a given sparsity structure (e.g., convolutions) is not straightforward [ 22 ] , which further limits the scalability and design flexibility of these models. We seek to address these limitations while retaining the internal stability and input-output robustness properties of RENs. In this paper, we introduce the robust recurrent deep network (R2DN) as a scalable, computationally efficient alternative to RENs [ 7 ] . Our approach shares a similar structure and direct parameterization to RENs, but with two small tweaks to the model architecture (Figure 1(b) ) which lead to dramatic improvements in scalability and design flexibility: (a) we eliminate the equilibrium layer by removing the feedthrough term from w to v in the LTI system \bm{G} ; and (b) we replace the scalar activation \sigma with a scalable 1-Lipschitz DNN. Our numerical experiments show that R2DNs are up to an order of magnitude faster in training and inference than RENs with similar test performance on tasks in system identification, estimation, control, and sequential image classification. Related work. Recent years have seen a boom in the development of scalable RNN architectures, particularly in the large language model (LLM) community, where they have been proposed as a scalable alternative to transformers for long-horizon sequence modeling. In particular, the focus has been on large-scale, structured state-space models (SSMs) [ 23 ] , which are linear (often parameter-varying) systems that can be efficiently evaluated on GPUs with parallel associative scans [ 24 ] . The LLM community has noted that directly parameterizing these SSMs to be internally stable is critical in preventing model divergence over long sequences [ 25 ] . Works from the controls community such as [ 11 , 26 ] have also proposed parameterizations of stable SSMs which additionally certify \ell_{2} -gain bounds on the input-output response, allowing them to be used as controllers with closed-loop stability guarantees, but they are yet to be tested at scale. At the same time, there is continued focus on improving the scalability of nonlinear RNNs [ 27 , 28 , 29 ] , since the expressivity of nonlinear interactions permits smaller nonlinear models to solve state-tracking and recall tasks that large linear models cannot [ 29 ] . In the nonlinear world, scalable parameterizations of stable RNNs have received comparatively little attention, with notable exceptions including [ 9 , 30 ] . The present paper addresses this gap by focusing on scalable parameterizations of not only stable, but also robust nonlinear RNNs. We choose the incremental notions of contraction for internal stability [ 12 ] and Lipschitzness (incremental \ell_{2} -gain) for input-output robustness. These incremental properties ensure both smooth and (under mild assumptions [ 12 ] ) bounded responses to perturbations, facilitating smooth generalization to unseen data [ 22 ] . II Problem Setup Given a dataset \mathcal{D} , we consider the problem of learning a nonlinear state-space model of the form x_{t+1}=f_{\theta}(x_{t},u_{t}),\quad y_{t}=h_{\theta}(x_{t},u_{t}), (1) where x_{t}\in\mathbb{R}^{n},u_{t}\in\mathbb{R}^{m},y_{t}\in\mathbb{R}^{p} are the states, inputs, and outputs of the system at time t\in\mathbb{N} , respectively. Here f_{\theta}:\mathbb{R}^{n}\times\mathbb{R}^{m}\rightarrow\mathbb{R}^{n} and h_{\theta}:\mathbb{R}^{n}\times\mathbb{R}^{m}\rightarrow\mathbb{R}^{p} are parameterized by some learnable parameter vector \theta\in\Theta\subseteq\mathbb{R}^{N} (e.g., the weights and biases of a deep neural network). The learning problem can be formulated as an optimization problem \min_{\theta\in\Theta}\quad\mathcal{L}(f_{\theta},h_{\theta};\mathcal{D}) (2) for some loss function \mathcal{L} . If \Theta=\mathbb{R}^{N} , the learning problem ( 2 ) has no constraints and can be solved via unconstrained optimization tools such as gradient descent, which is particularly useful when working with large models. If, however, \Theta\subset\mathbb{R}^{N} , the