Weight-Space Linear Recurrent Neural Networks
Roussel Desmond Nzoyem, Nawid Keshtmand, Idriss Tsayem, David A.W. Barton, Tom Deakin · 2025
Read Weight-Space Linear Recurrent Neural Networks by Roussel Desmond Nzoyem, Nawid Keshtmand, Idriss Tsayem, David A.W. Barton, Tom Deakin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01153.txt.
Introduction
Roussel Desmond Nzoyem and colleagues introduced a new sequence model in 2025 called WARP. Standard recurrent networks squeeze the past into a fixed set of hidden numbers, which can blur detail. WARP instead keeps its memory as the full weights of a small helper network, updated step by step from changes in the input.
Because memory is a working network, the model can adapt at test time without gradient steps and can absorb known physics rules. Across classification, image completion, and forecasting tests it matched or beat strong rivals. A physics-guided version won one task by a very wide margin. The design rethinks what a memory state can be.
Catalog record
| CREATOR | Roussel Desmond Nzoyem, Nawid Keshtmand, Idriss Tsayem, David A.W. Barton, Tom Deakin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01153
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Weight-Space Linear Recurrent Neural Networks Roussel Desmond Nzoyem, Nawid Keshtmand, Idriss Tsayem, David A.W. Barton, Tom Deakin Abstract We introduce WARP ( W eight-space A daptive R ecurrent P rediction), a simple yet powerful framework that unifies weight-space learning with linear recurrence to redefine sequence modeling. Unlike conventional recurrent neural networks (RNNs) which collapse temporal dynamics into fixed-dimensional hidden states, WARP explicitly parametrizes the hidden state as the weights of a distinct root neural network. This formulation promotes higher-resolution memory, gradient-free adaptation at test-time, and seamless integration of domain-specific physical priors. Empirical validation shows that WARP matches or surpasses state-of-the-art baselines on diverse classification tasks, spanning synthetic benchmarks to real-world datasets. Furthermore, extensive experiments across sequential image completion, dynamical system reconstruction, and multivariate time series forecasting demonstrate its expressiveness and generalization capabilities. Critically, WARP’s weight trajectories offer valuable insights into the model’s inner workings. Ablation studies confirm the architectural necessity of key components, solidifying weight-space linear RNNs as a transformative paradigm for adaptive machine intelligence. 1 Introduction Deep sequence models consistently demonstrate limitations in generating trajectories that diverge from their training distribution Alijani et al. 2024 ; Hataya et al. 2024 ; Graham et al. 2023 . For instance, subsets of neural ODE Chen et al. 2018 parameters necessitate adaptation via gradient descent to maintain performance on out-of-distribution (OoD) sequences Kassaï Koupaï et al. 2024 ; Nzoyem et al. 2025a . While powerful, their inherent gradient calculation cost has recently catalyzed research into gradient-free alternatives Serrano et al. 2024 ; Morel et al. 2025 ; Hemmer and Durstewitz 2025 , essentially trading accuracy for computational efficiency. In an effort to preserve both while achieving true zero-shot adaptation, we investigate two powerful emerging paradigms: weight-space learning and linear recurrence Rumelhart et al. 1986 . Weight-space learning — the paradigm that treats the weights and biases of a neural network as data points for another learning system Schürholt et al. 2024 --- offers unprecedented potential for extracting properties of a trained model solely from its ‘‘weights’’ 1 1 1 Following the convention from Zhou et al. 2023 , we refer to the learnable parameters of the processed network as ‘weights’ (or ‘weight-space’ to indicate the space they belong to) and those of the higher-level learning system (the neural functional) as simply ‘parameters’. . Applications span from predicting generalization error Unterthiner et al. 2020 and recovering training data Dupont et al. 2022 to classifying and editing implicit neural representations De Luigi et al. 2023 . With the proliferation of model repositories such as HuggingFace and CivitAI, developing methods that effectively learn directly from weights has become increasingly vital Kahana et al. 2024 . To date, the literature has predominantly focused on utilizing these weights as inputs and outputs to secondary models, leaving their potential as intermediate representations (e.g. latent vectors, hidden states) in end-to-end training systems unexplored. Concurrently, linear Recurrent Neural Networks (RNNs) have experienced a significant resurgence, driven by their hardware-friendly nature and the resulting ease of training Dao and Gu 2024 . Linearity facilitates advanced parallelization capabilities Movahedi et al. 2025 , and has enabled exceptional performance on long-sequence tasks Gu et al. 2021 ; Orvieto et al. 2023 . That said, recent findings question the information content of their compressed state representations Merrill et al. 2024 . Additionally, an important body of work establishes how linear Transformers Vaswani et al. 2017 and a particular instantiation of linear RNNs — State-Space Models (SSMs) — are fundamentally less powerful than standard RNNs Beck et al. 2024 ; Deletang et al. 2023 ; Merrill and Sabharwal 2023 . These strongly suggest that non-linearities are essential for the expressivity of deep sequence models. The preceding analyses motivate our examination of gradient-free adaptation through the lens of weight-space linear RNNs. To harness the strengths of the aforementioned paradigms, we formulate several research questions: \bullet Can the weights of an auxiliary (root) neural network serve as high-resolution hidden states for linear RNNs ? \bullet Can the resulting architecture adapt its dynamics during inference without requiring gradient computation ? \bullet Can the activation functions in the root network provide sufficient non-linearity to significantly enhance the expressive capacity of such models ? We answer all these questions in the affirmative. Specifically, our technical contributions are summarized as follows: (1) We propose a novel framework (WARP) for learning on sequential data, integrating weight-space features, sequence mixing via linear recurrence, and nonlinear self-decoding which enables the embedding of domain-specific inductive biases. (2) Following the SSM paradigm, we introduce multiple training algorithms, including an auto-regressive mode well-suited for noisy sequence. WARP demonstrates competitive or state-of-the-art performance across diverse forecasting and classification benchmarks. (3) We establish a comprehensive benchmark suite to evaluate various capabilities of RNNs regarding reconstruction, adaptation and memory-retention in physical systems. WARP consistently outperforms traditional RNNs, SSMs, and Transformer architectures. 2 Weight-space Adaptive Recurrent Prediction (WARP) State Transition & Decoding in Recurrent Neural Networks Standard RNNs \mathbf{h}_{t}=f_{\Phi}(\mathbf{h}_{t-1},\mathbf{x}_{t}) \mathbf{y}_{t}=g_{\Psi}(\mathbf{h}_{t}) Linear RNNs \mathbf{h}_{t}=A\mathbf{h}_{t-1}+B\mathbf{x}_{t} \mathbf{y}_{t}=C\mathbf{h}_{t-1} Weight-Space Linear RNNs \theta_{t}=A\theta_{t-1}+B(\mathbf{x}_{t}-\mathbf{x}_{t-1}) \mathbf{y}_{t}=\theta_{t}(\tau) Figure 1 : Conceptual comparison between various RNN architectures. Standard RNNs (e.g. Hochreiter and Schmidhuber 1997 ) feature a non-linear transition function f_{\Phi} unlike their linear counterparts (e.g. Orvieto et al. 2023 ). Our proposed WARP is a Weight-Space Linear RNN, with hidden states represented as \theta_{t} to emphasize their view as weights of a self-decoding function (in addition to their view as tensors). In the decoding phase, \tau=t/(T-1) with T the training sequence length. This section introduces WARP, a novel framework for deep sequence models that operate by directly modulating the weights of a function approximator in response to sequential inputs. We begin by establishing the problem setting, followed by WARP’s architectural and training details. 2.1 Problem Setting Our framework addresses the canonical sequence modeling problem, wherein a computational model must establish a mapping from an input \mathbf{x}_{t}\in\mathbb{R}^{D_{x}} to a corresponding output \mathbf{y}_{t}\in\mathbb{R}^{D_{y}} , with t\in\{0,\ldots,T-1\} denoting the time step index. The integer T>0 represents the training sequence length, which remains invariant across all sequences within a training batch 2 2 2 We note that T may be different for testing sequences. D_{\bullet} is the dimensionality of the sub-scripted quantity. . We assume that all input sequences are sampled at the same uniform intervals. With B representing the batch dimension, our models establish a mapping from \mathbb{R}^{B\times T\times D_{x}} to \mathbb{R}^{B\times T\times D_{y}} . In the regression setting of time series forecasting , we have \mathbf{y}_{t}\triangleq\mathbf{x}_{t+1} , as our objective is to predict future tokens conditioned on a preceding sequence of tokens, designated as the “context” \mathbf{x}_{<L}\triangleq\{\mathbf{x}_{t}\}_{t\in\{0,\ldots,L-1\}} , where L denotes the context length. Consistent with the sequence modeling literature, we desire the ability to perform auto-regressive rollouts during inference. For classification tasks, only the final token \mathbf{y}_{T-1} is treated as a softmax -activated logit to assign a label to the sequence. 2.2 Architecture Unlike traditional recurrent networks that update hidden states \mathbf{h}_{t},\forall\,t\in\{0,\ldots,T-1\} , weight-space linear RNNs such as WARP update the flattened parameters of a root neural network \theta_{t} , effectively learning a dynamics model in weight-space (see Figure 1 ). Specifically, we define the recurrence relation and the subsequent decoding: \theta_{t}=A\theta_{t-1}+B\Delta\mathbf{x}_{t},\qquad\text{and}\qquad\mathbf{y}_{t}=\theta_{t}(\tau), (1) where \theta_{t}\in\mathbb{R}^{D_{\theta}} represents the weights of the root neural network at time step t , and \Delta\mathbf{x}_{t}=\mathbf{x}_{t}-\mathbf{x}_{t-1}