DREAMSTATE: Diffusing States and Parameters for Recurrent Large Language Models
Liu Xiao [email protected] · 2026
Read DREAMSTATE: Diffusing States and Parameters for Recurrent Large Language Models by Liu Xiao [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19221.txt.
Introduction
Liu Xiao rethinks the inner state of recurrent language models as something that can be edited. Modern recurrent nets like RWKV model nearby text well and keep a fixed size state, a core edge over transformers. Yet few studies treat that state as a store of knowledge that can be read or changed.
The work, DREAMSTATE, diffuses both states and parameters, borrowing the gradual noise and cleanup of image diffusion. If a recurrent model's state can be edited like a file, builders gain a new lever for fixing facts and behavior. The study opens that lever to its first close look.
Catalog record
| CREATOR | Liu Xiao [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-19221
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DREAMSTATE: Diffusing States and Parameters for Recurrent Large Language Models Liu Xiao [email protected] Abstract Modern Recurrent Neural Networks (RNNs), such as RWKV, are distinguished by their powerful short-range modeling capabilities and efficient fixed-size states, which constitute a core advantage over standard Transformers. However, there is a significant lack of research into their internal state as an editable knowledge representation. To fill this gap, we first explore the representational properties of the RWKV state by proposing the DREAMSTATE framework. This framework utilizes a conditional Diffusion Transformer (DiT) to directly model the probability manifold of the state, enabling its generation and editing. The structural nature of this representation is validated through t-SNE visualizations and controlled generation experiments. After successfully uncovering and modeling the state’s representational potential, we further propose a novel hybrid architecture that combines the local advantages of RNNs with global context adaptability. This architecture features a parallel DiT that processes a variable-length global context to dynamically generate and adjust the core recurrent module’s WKV parameters, transforming the fixed recurrence mechanism into a context-aware dynamic function. Experiments demonstrate that this hybrid model can be trained stably via a multi-objective loss, validating its design feasibility. Our work not only opens a new research direction for RNN state representation but also provides a concrete architectural reference for future model design. The code is publicly available at: https://huggingface.co/2dgx41s/DreamState . 1 INTRODUCTION The landscape of sequence modeling has been dominated by the Transformer architecture ( Vaswani et al. 2017 ) , but its quadratic complexity poses challenges for long sequences. This has renewed interest in Recurrent Neural Network (RNN) architectures ( Hochreiter and Schmidhuber 1997 ; Cho et al. 2014 ; Chung et al. 2014 ) like RWKV, which blend the linear-time inference of RNNs with the parallelizable training of Transformers ( Peng et al. 2023 ) . Foundational works in deep learning have paved the way for these modern architectures ( Bengio and LeCun 2007 ; Hinton et al. 2006 ; Goodfellow et al. 2016 ) . However, a critical limitation shared by both Transformers and many early Linear RNNs is their inability to perform fundamental state-tracking tasks Merrill et al. 2024 , such as solving parity, which impairs performance in areas like code evaluation and logical reasoning. Other modern sequence models include State Space Models ( Gu et al. 2022 ; Gu and Dao 2023 ) . Recent work by Grazzi et al. 2024 concludes that this failure stems from constraining the state-transition matrix eigenvalues to the [0, 1] range. Their key finding is that extending this range to [-1, 1] provably unlocks these state-tracking capabilities, significantly enhancing model expressivity without computational overhead. The recent RWKV-7 ”Goose” model directly leverages this principle through its generalized delta rule, which facilitates such dynamic state transitions ( Peng et al. 2025 ) . This adoption positions modern RNNs like RWKV-7 as not just efficient Hu et al. 2025 , but fundamentally more expressive than standard Transformers for tasks requiring robust state maintenance . This newfound richness of the internal state motivates our core question: if the state is such a powerful, structured representation of context, can we model and manipulate it directly? We argue that how we model the State is the key for the future of Large Language Models ( Brown et al. 2020 ; OpenAI 2023 ; Touvron et al. 2023 ) . This investigation leads to a more profound inquiry. Recent work ( Devlin et al. 2019 ; Raffel et al. 2020 ; Dai et al. 2019 ; Radford et al. 2021 ; Dosovitskiy et al. 2021 ; Kolesnikov et al. 2021 ; Liu et al. 2021 ; Carion et al. 2020 ) has shown that standard Transformers can suffer from ”attention noise,” where the model overallocates attention to irrelevant context. The RWKV’s state, being a recurrent variant of attention, faces an analogous, yet deeper, issue. Its knowledge compression mechanism is governed by a set of static ’WKV’ parameters, which are fixed after training. This static nature acts as a form of ”structural noise” or ”weighting bias,” as a single, fixed mechanism is suboptimal for the diverse contexts a model might encounter. We question this static assumption, drawing inspiration from the noise-canceling paradigm. Just as differential attention aims to cancel noise in a-scores Ye et al. 2024 , we propose to mitigate this structural noise by making the parameters themselves dynamic. We leverage the unique properties of diffusion models. Because the diffusion component is parallel, it can be conditioned on a variable-length context, allowing it to track global features to guide generation. We hypothesize that this capability can be repurposed to dynamically generate the WKV parameters. This would allow a globally-informed parallel module (the DiT) to dynamically configure the operational ”laws” of a local, efficient serial module (the RWKV block) for the immediate task. To formalize this, we ground our approach in the mathematics of generative modeling. This hypothesis leads to a two-stage experimental design. First, as a foundational step, we learn the distribution of states p(S|c) and validate it through visualization and controlled generation. Second, we design and implement our novel hybrid architecture, where a DiT generates ’WKV’ parameters based on global context, and validate it through stable joint optimization. Our contributions are thus: • We propose DREAMSTATE, a framework for generatively modeling the dynamic state of an RWKV model, validated through t-SNE visualization and controlled inference experiments. • We introduce a novel architecture where the static recurrence parameters (’WKV’) are themselves dynamically generated by a diffusion model that tracks global input features from a variable-length context, as a principled method to counteract the ”structural noise” of fixed recurrence. • We define a multi-objective training paradigm that jointly optimizes for parameter generation and next-token prediction, and show its viability through empirical results. • Our work opens new avenues for building more controllable and adaptive world models, grounded in a probabilistic treatment of both state and parameter spaces. • Our work provides a proof-of-concept that a hybrid of traditional token-wise autoregression and a novel concept-wise autoregression (via state generation) is a viable and promising direction for future models. 2 Methodology: From Static Recurrence to Dynamic Synthesis Our methodology is built upon the premise that the limitations of a static recurrence can be overcome by treating its core components—first its state, then its parameters—as variables that can be generatively modeled. We use the mathematical framework of diffusion models to first learn the manifold of valid states, and then to dynamically generate the parameters that control the evolution on this manifold. 3 Recurrence as Attention and the Problem of Structural Noise The RWKV state update can be unrolled to show that the state at time t is an aggregation of all past key-value products, making it a recurrent analogue to the attention mechanism in Transformers. The specific update rule for the Weighted Key Value (WKV) state in RWKV-7, wkv_{t} , is defined by the recurrence relation: wkv_{t}=wkv_{t-1}(\text{diag}(w_{t})-\hat{\kappa}_{t}^{T}(a_{t}\odot\hat{\kappa}_{t}))+v_{t}^{T}\cdot\tilde{k}_{t} (1) When unrolled, this shows the state wkv_{t} as an explicit weighted sum: wkv_{t}=\sumop\displaylimits_{i=1}^{t}\left(v_{i}^{T}\tilde{k}_{i}\prodop\displaylimits_{j=i+1}^{t}(\text{diag}(w_{j})-\hat{\kappa}_{j}^{T}(a_{j}\odot\hat{\kappa}_{j}))\right) (2) This formulation shows how the influence of past key-value products (v_{i}^{T}\tilde{k}_{i}) decays over time through the product of transition matrices. A key limitation, however, is that the weighting scheme is governed by a