StateSpaceDiffuser: Bringing Long Context to Diffusion World Models

Nedko Savov, Naser Kazemi, Deheng Zhang, Danda Pani Paudel, Xi Wang, Luc Van Gool · 2025

Read StateSpaceDiffuser: Bringing Long Context to Diffusion World Models by Nedko Savov, Naser Kazemi, Deheng Zhang, Danda Pani Paudel, Xi Wang, Luc Van Gool (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22246.txt.

Introduction

Video world models predict how a scene unfolds, yet they forget: after a few steps their scenes drift away from what was seen before. Nedko Savov and five co-authors fix the memory gap in 2025 by grafting a state space module, which holds the whole history, onto a diffusion model. The pair keeps diffusion's sharp images while remembering the past.

A new test checks whether a model can bring back content it saw earlier in a long rollout. The hybrid stayed coherent ten times longer than a strong diffusion only rival. It held together in both a flat maze and a rich 3D world. The authors say lasting state, not bigger sampling, is what world models were missing.

Catalog record

CREATORNedko Savov, Naser Kazemi, Deheng Zhang, Danda Pani Paudel, Xi Wang, Luc Van Gool
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22246

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

StateSpaceDiffuser: Bringing Long Context to Diffusion World Models Nedko Savov, Naser Kazemi, Deheng Zhang, Danda Pani Paudel, Xi Wang, Luc Van Gool Abstract World models have recently gained prominence for action-conditioned visual prediction in complex environments. However, relying on only a few recent observations causes them to lose long-term context. Consequently, within a few steps, the generated scenes drift from what was previously observed, undermining temporal coherence. This limitation, common in state-of-the-art world models, which are diffusion-based, stems from the lack of a lasting environment state. To address this problem, we introduce StateSpaceDiffuser, where a diffusion model is enabled to perform long-context tasks by integrating features from a state-space model, representing the entire interaction history. This design restores long-term memory while preserving the high-fidelity synthesis of diffusion models. To rigorously measure temporal consistency, we develop an evaluation protocol that probes a model’s ability to reinstantiate seen content in extended rollouts. Comprehensive experiments show that StateSpaceDiffuser significantly outperforms a strong diffusion-only baseline, maintaining a coherent visual context for an order of magnitude more steps. It delivers consistent views in both a 2D maze navigation and a complex 3D environment. These results establish that bringing state-space representations into diffusion models is highly effective in demonstrating both visual details and long-term memory. Project page: https://insait-institute.github.io/StateSpaceDiffuser/ . † † footnotetext: Corresponding author: [email protected] Figure 1: Recalling content long in the past. Given a history of images I_{1},...,I_{T} and accompanying actions, we navigate all the way back to the beginning - I_{1} . The task is to generate frames along the way, consistent to what is seen in the history, given the actions. As an example, we show the predictions of the first frame - \hat{I_{1}} . Can a generative model recall the content of I_{1} long back in the sequence? Diffusion models fall short (✗), our model correctly recalls the content of I_{1} (✓). Figure 2: Our Approach. While diffusion models are limited to a short sequence input, our approach enables long-context processing for diffusion models with a state-space representation. 1 Introduction World models have gained popularity for the production of visual consequences of given past observations and actions. These models can learn to generate environment observations entirely by training on many interactions with the environment. Simply by observation, they are capable of handling complex environments, such as car driving [ 27 , 29 , 41 , 66 ] , 3D virtual environments [ 77 , 1 , 19 ] , platformer games [ 8 , 68 ] , ego-centric action videos [ 82 ] , or navigation [ 2 , 84 ] . They enable interactivity without the burden of hand-coding complex environments, but also offer feature representations for robotics and reinforcement learning agents for planning. For long interaction with world models, it is essential that the generated video remains consistent with previously observed or generated content. Revisited areas should preserve their appearance, and objects observed again should keep their properties. However, as shown in Fig. 1 , current high-fidelity world models, which are mostly based on diffusion, cannot preserve context outside of a short time window, most often directly limited by their input window size [ 82 , 1 , 19 , 8 ] . This leads to an increasing drift in content over time, where earlier information is forgotten or overwritten. The inability to retain persistent memory of the environment poses a major challenge, especially for real-world applications such as agent planning and virtual interaction, where coherent, temporally consistent environments are essential. Therefore, in this work our task is to stay consistent with a long history of past inputs, even if generating a single future frame. This is in contrast to long generation which focuses on producing extended realistic sequences, even without prior context. To improve content consistency in diffusion-based world models, we make use of a persistent long-context representation. Specifically, we leverage features from a discrete state-space model (Mamba), which has been shown to be very effective at capturing long-term context in prior work [ 20 , 31 ] . We summarize our system in Fig. 2 . Although these models were previously applied to language and visually simple environments [ 12 ] , our goal is to preserve the long-term context in modern diffusion-based world models, targeting environments with higher visual complexity such as CSGO [ 60 ] . In contrast, other state-based models, such as those using LSTMs or GRUs [ 38 , 39 , 40 ] , have limited generative capacity and are mainly used for agent planning. Our approach combines the strong generative power of diffusion models with the long-term context tracking of state-space representations. Importantly, the state-space model (SSM) is computationally efficient, which allows it to process arbitrarily long sequences. This is achieved by maintaining a compact state that is updated at every sequence step. During training, SSMs have linear complexity in sequence length [ 31 ] , further improved by parallelization. Unlike them, CNNs have a fixed receptive depth, and transformers characterize with a heavy quadratic complexity. At inference, in a streaming fashion, the SSM can be executed with constant per-step latency and constant memory footprint, whereas transformers and CNNs, at best, still grow linearly per step. As we show, in test time, our proposed model scales far beyond its trained horizon, while the SSM contributes less than 2% of the total inference compute of the full model. Our proposed model, StateSpaceDiffuser, summarized in Fig. 2 , consists of a state-space model that operates over the long sequence, and a diffusion model - conditioned on both a short window of observations and state-space model features. The latter enables the diffusion branch to generate the content of the next frame conditioned on a long context rather than the last few frames. To evaluate the consistency of the generated long context of StateSpaceDiffuser, we design and develop an evaluation framework that involves navigating environments to then return back to the initial position. We evaluate on two environments. (1) A simpler 2D maze environment (MiniGrid), in which we establish the presence or absence of memory ability by remembering the maze layout given partial observations. And (2) a 3D first-person shooter game (CSGO), which serves to show the performance of our method on a visually challenging interactive environment with many factors at play. Our quantitative and qualitative evaluation results show that StateSpaceDiffuser produces content significantly more consistent with a long history than a diffusion-only method. Evaluation in the maze environment yields 51.9% PSNR improvement over the baseline on average (56.3% improvement on the most memory challenging cases). A user study confirms that our method produces images closer to previously observed content in the CSGO dataset compared to baselines. More details are shown in Sec. 5 . Our contributions are as follows. • We propose StateSpaceDiffuser, which integrates a state-space model with a diffusion model for visual world modeling. It is capable of generating consistent content in long-horizon generation, with almost no extra computational cost. • We develop an evaluation protocol to test the content preservation abilities of a world model and perform extensive evaluations of world models on long-horizon generation tasks. • Our evaluation shows a significant quantitative improvement and a strong user preference over the baseline in

Read the complete text