Render, Don’t Decode: Weight-Space World Models with Latent Structural Disentanglement
Roussel Desmond Nzoyem, Mauro Comi · 2026
Read Render, Don’t Decode: Weight-Space World Models with Latent Structural Disentanglement by Roussel Desmond Nzoyem, Mauro Comi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06298.txt.
Introduction
Roussel Desmond Nzoyem and Mauro Comi rethink world models by storing the state of a scene as neural network weights. Models trained on unlabeled video usually encode pixels into opaque latent spaces and lean on heavy decoders to rebuild frames, which is costly and hard to interpret. Their framework, NOVA, instead represents each state as the weights and biases of a small coordinate based network that renders the scene analytically.
Removing the decoder brings compactness, portability, and free super resolution, and the model runs at about forty million parameters on one consumer graphics card. Like other latent action models, it can be distilled into a video generator. Without extra losses, NOVA separates background, foreground, and motion on its own. Users can then edit content or dynamics independently, a step toward customizable virtual experience.
Catalog record
| CREATOR | Roussel Desmond Nzoyem, Mauro Comi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 93 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06298
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Render, Don’t Decode: Weight-Space World Models with Latent Structural Disentanglement Roussel Desmond Nzoyem, Mauro Comi Abstract Training world models on vast quantities of unlabelled videos is a critical step toward fully autonomous intelligence. However, the prevailing paradigm of encoding raw pixels into opaque latent spaces and relying on heavy decoders for reconstruction leaves these models computationally expensive and uninterpretable. We address this problem by introducing NOVA , a world modelling framework that represents the system state as the weights and biases of an auxiliary coordinate-based implicit neural representation (INR). This structured representation is analytically rendered, which eliminates the decoder bottleneck while conferring compactness, portability, and zero-shot super-resolution. Furthermore, like most latent action models, NOVA can be distilled into a context-dependent video generator via an action-matching objective. Surprisingly, without resorting to auxiliary losses or adversarial objectives, NOVA can disentangle structural scene components such as background, foreground, and inter-frame motion, enabling users to edit either content or dynamics without compromising the other. We validate our framework on several challenging datasets, achieving strong controllable forecasting while operating on a single consumer GPU at ∼ \sim 40M parameters. Ultimately, structured representations like INRs not only enhance our understanding of latent dynamics but also pave the way for immersive and customisable virtual experiences. 1 Introduction The capacity to predict plausible future states within an environment is a cornerstone of physical intelligence. World models 21 achieve this by learning transition dynamics conditioned on actions, enabling embodied agents to plan and act through internal simulation (e.g. robotic manipulation and autonomous driving). Recently, the abundance of unlabelled, action-free online videos has driven the emergence of latent action (world) models 18 , 7 , 52 . Devoid of ground-truth action signals during training, they must discover underlying control mechanisms purely from observation, bringing them closer to video generation models, which learn to produce coherent sequences of frames from noise or context. The predominant paradigm for training world models and video generation models involves projecting and reconstructing frame sequences through a latent space, for example, using a video tokeniser 7 . A prominent class of autoregressive models encode individual frames into abstract latent vectors via a variational autoencoder 33 , then map resulting predictions back to pixels via a learned spatial decoder. This paradigm has been immensely productive 22 , 42 , 7 , but carries intrinsic limitations. First, the decoder consumes more storage, adding a number of hardly interpretable parameters that could limit usability in safety-critical scenarios 51 . As a case in point, 7 find that as they scale, a much larger decoder is essential, accounting for up to 87\% of their video tokeniser. Second, any change in output resolution requires expensive retraining or post-hoc upsampling workarounds 8 . Third, the latents it consumes lack portability, as they are tethered to that specific decoder. Furthermore, they have no semantic or geometric interpretations. Current attempts to disentangle semantic components such as background, identity, and motion from an abstract latent space require auxiliary losses 20 , adversarial objectives 13 , or architectural decompositions 27 , all of which add training complexity without structural guarantees. Coordinate-based implicit neural representations (INRs) 54 , 14 represent an image signal not as a pixel array but as the weights of a neural network, thus encoding a continuous function \mathbb{R}^{2}\to\mathbb{R}^{C} that maps any spatial coordinate to its RGB ( C=3 ) or greyscale ( C=1 ) pixel value. They gained prominence by powering novel view synthesis frameworks such as NeRF 43 , and have recently demonstrated compression capabilities that surpass JPEG at low bitrates 44 . Crucially, further advantages arise from an analytical rendering process that circumvents the need for a spatial decoder. Despite inherent spectral aliasing challenges 5 , 38 , properly constrained parameter-free rendering intrinsically supports continuous, arbitrary-resolution querying at test time, a capability that is critical in domains such as extreme weather events forecasting, where a single pixel can encapsulate a vast area of the globe (e.g., over 784 km 2 in state-of-the-art global AI models 34 ); in such contexts, unlocking granular resolution can be the difference between life and death for massive populations. Furthermore, unlike abstract encodings, INRs are highly portable, as their analytical formulation makes them easily sharable across the scientific community. Given these fundamental strengths, structured representations like INRs and Gaussian splats 30 are increasingly recognised as the most promising technologies to power next-generation scientific modelling, visual perception, and computer graphics. In this paper, we argue that INRs are a potent representation upon which world models and video generation models can be built. To realise this, we leverage the paradigm of weight-space learning (WSL), which views the weights and biases of neural networks such as INRs as data points for a higher-level learning system 53 . This paradigm was recently utilised by 35 and 46 to achieve remarkable performance on time series modelling benchmarks. These WSL approaches, however, remain untested on video modalities. Crucially, the latter proved unstable, requiring weight, gradient, and value clipping to prevent numerical explosion. Any viable weight-space world model must first address these acute stability issues. Figure 1: Video editing facilitated by object-motion disentanglement . Conditioned on the same initial frame, all models must generate future states; at t=6 , this state is abruptly replaced with an alien frame’s encoding. (a) The standard world model (WM), trained in an abstract latent space with NOVA ’s decomposition strategy, manages to separate content from motion. (b) LAPO 52 generates clean frames, but allows the foreign frame to completely alter its dynamics; furthermore, its generation quality degrades over extended horizons (starting at t=9 ). (c) NOVA generates visually striking digits over long horizons and swaps their identities without altering the underlying dynamics (e.g., a 5 seamlessly inherits the trajectory of a 1 ). To bridge this gap, we introduce NOVA ( N eural O ntology for V isual A bstraction), a stable and physically consistent world model framework that evolves dynamics entirely in the weight space of an implicit neural representation. Its straightforward architecture induces a three-level semantic ontology that mirrors human intuition about object-centric video structure: ( i ) the background ( \bar{\bm{z}} ) is captured by a single set of base weights shared across the entire dataset, learned to encode everything common to all videos; ( ii ) the foreground content ( \bm{z}_{t} ) is defined by per-frame weight offsets encoding what is present in the current video frame but not shared globally; and ( iii ) the motion ( \bm{u}_{t} ) is an inter-frame action signal that dictates how a latent state transitions to the next. To achieve such visual abstraction , our framework leverages an additive formulation that cleanly isolates these properties during both forward dynamics and base composition prior to rendering. Specifically, our key contributions are as follows: 1. We build NOVA , the first world model defined by a weight-space representation. We demonstrate that the inclusion of a base representation \bar{\bm{z}} addresses the characteristic instability of weight-space model training. Similar to recent literature 52 , we show that this framework can be tuned to produce actions autoregressively during inference, enabling seamless transition from world modelling to context-conditioned video generation. 2. We show that the (\bar{\bm{z}},\bm{z}_{t},\bm{u}_{t}) parametrisation induces a semantically meaningful three-level