Walk through Paintings: Egocentric World Models from Internet Priors
Anurag Bagchi, Zhipeng Bao, Homanga Bharadhwaj, Yu-Xiong Wang, Pavel Tokmakov, Martial Hebert · 2026
Read Walk through Paintings: Egocentric World Models from Internet Priors by Anurag Bagchi, Zhipeng Bao, Homanga Bharadhwaj, Yu-Xiong Wang, Pavel Tokmakov, Martial Hebert (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15284.txt.
Introduction
Anurag Bagchi, Zhipeng Bao, and co-authors teach a video model to step inside paintings. Their 2026 arXiv paper deals with world models, systems that predict how a scene changes with each action. A good model should foresee the correct future, not just a plausible one.
The team presents EgoWM, the Egocentric World Model. The method takes any pre-trained video diffusion model and makes it action-conditioned. No new architecture is needed. Fed with internet priors, the model can then walk through paintings from a first-person view, frame by carefully predicted future frame.
Catalog record
| CREATOR | Anurag Bagchi, Zhipeng Bao, Homanga Bharadhwaj, Yu-Xiong Wang, Pavel Tokmakov, Martial Hebert |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15284
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Walk through Paintings: Egocentric World Models from Internet Priors Anurag Bagchi, Zhipeng Bao, Homanga Bharadhwaj, Yu-Xiong Wang, Pavel Tokmakov, Martial Hebert Abstract What if a video generation model could not only imagine a plausible future, but the correct one — accurately reflecting how the world changes with each action? We answer this by presenting Egocentric World Model (EgoWM), a simple, architecture-agnostic method that transforms any pre-trained video diffusion model into an action-conditioned world model, enabling controllable prediction of the future. Rather than training from scratch, we re-purpose the rich world priors of Internet-scale video models, injecting motor commands through lightweight conditioning layers. This allows our model to follow actions faithfully, while preserving generalization and realism. Our approach scales naturally across embodiments and action spaces — from 3-DoF mobile robots to 25-DoF humanoids, where predicting egocentric joint-angle–driven dynamics is substantially more challenging. The model produces coherent rollouts for both navigation and manipulation, requiring only modest fine-tuning. To evaluate physical correctness independent of appearance, we introduce the Structural Consistency Score (SCS), which measures whether stable scene elements evolve consistently with the provided actions. Our method improves SCS by up to 80% over prior state-of-the-art (Navigation World Models) while exhibiting up to 6 × \times lower latency and generalizing robustly to unseen environments — including navigation inside paintings. Figure 1 : Our framework generates future frame predictions (shown in blue) that accurately follow the provided robot actions (ground-truth frames shown in green). Notably, it effortlessly generalizes to vastly different embodiments, tasks and domains, including non-realistic ones, such as navigating within paintings (top right). This generalization is enabled by our simple, universal architecture. * * footnotetext: Equal advising. 1 Introduction Modeling how the future visual state of the world evolves in response to an agent’s actions – often referred to as world modeling 15 – is a fundamental capability for organisms with vision. It enables planning in navigation and manipulation tasks and even creative problem-solving 28 ; 33 . Animals acquire this ability through a lifetime of interaction and observation of others, allowing us to anticipate outcomes even in novel or physically impossible scenarios. For artificial agents, however, action-conditioned videos in diverse real-world settings are limited and expensive to acquire – an agent must interact with the world to get this data. As a result, many existing works train world models on narrow, simulator-defined datasets 41 ; 3 or with bespoke designs 7 ; 53 ; 37 ; 45 , which constrains their scalability. This leads to a crucial question: can we re-purpose off-the-shelf video models trained on large-scale passive data, and convert them into world models using only a small amount of paired action–observation data? Several prior works have explored learning action-conditioned video prediction models 7 ; 54 ; 41 ; 17 ; 14 . For example, the recent Navigation World Models (NWM) approach trains a 1-billion-parameter diffusion model on egocentric videos of robots and humans to model navigation behavior. While NWM demonstrates strong cross-environment generalization, collecting large datasets for every new embodiment or domain is impractical. More generally, prior world models in robotics or games remain domain-specific, learned for a single robot or a single environment. In contrast, large generative video models trained on diverse datasets have shown the ability to visually emulate real-world interactions when given suitable prompts 2 ; 49 . Our insight is to harness these pre-trained models as a starting point. By fine-tuning them with lightweight conditioning on action inputs, we can enable strong out-of-domain generalization: for instance, our model trained on real videos can simulate plausible navigation in entirely unseen environments, even those that do not exist in reality, thanks to the rich prior learned from internet-scale data. Specifically, we build on the observation that modern video diffusion architectures use a denoising timestep embedding to control the generation process. We introduce an action embedding module that piggybacks on this timestep conditioning. In practice, the action at each (latent) frame is encoded and added to the model’s time-conditioning pathway via learned scale-and-shift transformations. This design is architecture-agnostic: it does not alter the original model’s layers and works across different backbone architectures. We demonstrate this with both a UNet-based 36 diffusion model and a DiT-based (Diffusion Transformer) 32 model. Importantly, our conditioning operates directly in the joint-angle space of the robot or humanoid, enabling seamless scaling to complex embodiments with up to 25-DoF action spaces . As shown in Figure 1 , this makes our method applicable to high-dimensional humanoid navigation and manipulation — a capability not demonstrated by any prior open-source world model. Predicting how a humanoid’s body moves and how the entire visual scene evolves accordingly is significantly harder than predicting SE(3) camera motion. Furthermore, compared to NWM, our model achieves higher spatial resolution and lower latency, making it more practical for real-time, closed-loop deployment. To quantitatively evaluate our approach, we also propose a new metric: Structural Consistency Score (SCS) . Prior metrics such as LPIPS 51 or FVD 40 focus on perceptual similarity or realism 26 ; 48 , which can conflate visual fidelity with physical correctness. A model could produce sharp, realistic frames that nevertheless depict physically inconsistent outcomes. SCS instead measures whether the structure of the generated environment evolves correctly in response to the agent’s actions, independent of texture. Concretely, we automatically identify stable scene structures ( e.g . , walls, furniture, landmarks) in the first frame and track them across both real and generated videos using a segmentation tracker. Comparing the resulting trajectories allows us to assess whether the model’s predicted world evolves coherently with the true environment. A higher SCS indicates stronger structural and physical consistency with the provided action sequence. In summary, our contributions are as follows: • We propose a simple yet powerful recipe to convert pre-trained video diffusion models into action-conditioned world models . • We demonstrate that our method works across architectures and extends seamlessly to egocentric 25-DoF humanoids, enabling high-resolution video prediction for navigation and manipulation. • We introduce the Structural Consistency Score (SCS) , a metric for evaluating the structural and physical plausibility of generated videos independent of appearance. By learning from large-scale passive datasets and a small amount of action-labeled data, our framework offers a path toward scalable, general-purpose visual dynamics models. 2 Related Work World models and video prediction have long been studied in reinforcement learning and robotics. For example, the classic World of 15 trained a variational RNN to simulate simple games, allowing an agent to plan within its own latent dream. Follow-up works demonstrated video prediction for control in robotics – e.g . , visual foresight model 11 , which learned to predict future camera frames given robot actions and used model-predictive control to push objects to the target location, or SV2P 4 , which introduced stochastic latent variables for multi-modal future predictions. These pioneering approaches, however, were limited to relatively constrained environments (toy simulations or single-task robot setups). In the gaming domain, several works have proposed to visually emulate an entire game engine from screen pixels and keyboard actions 22 ; 29 ; 46 . More recently, there is a trend of utilizing high-capacity generative models for world modeling. The DIAMOND agent 3 uses a diffusion-based world model to better preserve