Physical Object Understanding with a Physically Controllable World Model
Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen, Wanhee Lee, Gia Ancone, Seungwoo Kim, Luca Thomas Wheeler, Jared Watrous, Honglin Chen, Daniel Bear, Stefan Stojanov, Daniel LK Yamins, Stanford University, OpenAI, Noetik Inc, Google · 2026
Read Physical Object Understanding with a Physically Controllable World Model by Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen, Wanhee Lee, Gia Ancone, Seungwoo Kim, Luca Thomas Wheeler, Jared Watrous, Honglin Chen, Daniel Bear, Stefan Stojanov, Daniel LK Yamins, Stanford University, OpenAI, Noetik Inc, Google (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00439.txt.
Introduction
Venkatesh and co-authors build a world model that can be asked odds on any part of a scene. To grasp things, a model must learn how parts form wholes and push each other. Past forms can not state odds for one trait given the rest. Their class of chance-based world models can.
Such models train well through plain line-by-line fits, they show. From raw clips, rich grasp of things springs up in the model. It rolls out more than one sound future for a scene. Shared move traits across those futures bare things and their joint parts. Known things can then be steered in 3D space in the model. Links twixt things can be read off for tasks like a block-stack game.
Catalog record
| CREATOR | Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen, Wanhee Lee, Gia Ancone, Seungwoo Kim, Luca Thomas Wheeler, Jared Watrous, Honglin Chen, Daniel Bear, Stefan Stojanov, Daniel LK Yamins, Stanford University, OpenAI, Noetik Inc, Google |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00439
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Physical Object Understanding with a Physically Controllable World Model Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen, Wanhee Lee, Gia Ancone, Seungwoo Kim, Luca Thomas Wheeler, Jared Watrous, Honglin Chen, Daniel Bear, Stefan Stojanov, Daniel LK Yamins, Stanford University, OpenAI, Noetik Inc, Google Abstract A central challenge in visual intelligence is learning the physical structure of scenes from raw videos: how regions form objects and the laws that govern their interactions. Solving these tasks requires world models capable of inferring distributional states of the world from partial observations – capabilities that current architectures do not provide. We introduce a new class of probabilistic world models that support estimation of the probability of any visual variable, such as appearance and dynamics, conditioned on any other variables. Here, we identify that these models can be trained efficiently with autoregressive sequence modeling, yielding world models from which rich object understanding emerges. First, we demonstrate that our model captures the physical laws governing how objects move by generating multiple plausible future states of the world through sequential inference. Then, by analyzing motion correlations across these futures, we extract objects and articulated object subparts. Having discovered these objects, we show that our world model can manipulate them in 3D. Finally, we demonstrate how physical relationships between objects can be computed from the world model, enabling applications such as Visual Jenga. Our project page and code is available at: https://neuroailab.github.io/psi-website/blog.html 1 1 footnotetext: equal contribution 2 2 footnotetext: work done while at Stanford 1 Introduction A central goal in visual intelligence is to build models that understand the physical structure of scenes – how things move if interacted with, how regions correspond to objects, how objects move in 3D, and the physical relationships among them, such as articulation, support, etc. Achieving this requires probabilistic reasoning that captures how one variable in the scene depends on another – e.g., how motion at one location influences motion elsewhere, or how forces applied to an object propagate through its structure. Bringing all these capabilities together within a unified architecture remains a major challenge. Many existing world-modeling approaches operate on prompts such as text 38 , actions 5 , or global scene embeddings 35 ; 1 – which do not provide a way to isolate or query how local scene variables (e.g., appearance, depth, motion) influence one another. Understanding these relationships is essential for reasoning about objects and their physical interactions. Here, we solve this problem by introducing a new class of world models, where we represent the state of the world with local variables and predict the probability distribution of each such variable, given any other variables. We show that this model concept can be formulated as a GPT-style 32 next token prediction sequence model, allowing efficient and scalable model training. Under this framework, we build a Physically controllable World Model, with tokens describing visual scenes such as RGB tokens that encode appearance, flow tokens that encode dynamics, and camera tokens that encode viewpoint changes between frames. Our world model supports a wide range of inference pathways by treating different scene variables as probabilistic structures that can either be inferred from observations or used to condition future predictions. For example, optical flow can be generated as an intermediate representation to model plausible scene dynamics conditioned on sparse pokes, or supplied as a conditioning signal to render future appearance states of the world. More generally, our framework can be used to integrate multiple structured representations of a scene—such as appearance, motion, geometry, and actions—within a unified probabilistic framework. We therefore refer to this framework as a Probabilistic Structure Integrator ( PSI ), emphasizing its ability to represent, predict, and compose diverse scene structures through probabilistic inference. We then use PSI to extract a wide range of physical object understanding. First, its predicted distributions provide motion statistics, highlighting which regions are likely to move and by how much. Next, sequential inference over flow and RGB tokens produces multiple plausible futures, capturing uncertainty and dynamics in the visual world. From these imagined futures, we derive object-like entities by using motion-correlation analysis to group pixels that move together, achieving state-of-the-art object segmentation scores on SpelkeBench 27 . Further, we show how articulated subparts can be found, outperforming prior methods on DragAMove 2 . These discovered objects can then be consistently manipulated in 3D, achieving state-of-the-art results on 3DEditBench 21 . Finally, we demonstrate how the world model reasons about physical relationships between objects to perform tasks like Visual Jenga 4 . 2 Related Works Visual world models: Recent world models provide strong generative and perceptual abilities – they excel at producing long-form video 5 ; 1 , performing instruction-conditioned generations 24 ; 1 , and offering rich multimodal reasoning 37 . Yet these systems fall short of the flexible conditional inference that deeper physical scene understanding requires 44 . They do not expose a mechanism for physically grounded queries – such as how local scene variables change under specific physical interventions. PSI overcomes this by modeling the visual world as a probabilistic model over local variables, enabling fine-grained causal reasoning for a range of scene-understanding tasks. Further, while classical world models are typically defined as action-conditioned predictors 16 , PSI is best viewed as a “poor man’s world model” 20 , where expensive-to-obtain true action data is proxied by cheap visual data patches like optical flow and camera conditioning that encode approximations to agent actions. Specialized object understanding models: Several prior works target aspects of object and scene understanding, typically in specialized settings. For object discovery, supervised segmentation models such as SAM2 33 rely on costly annotations, but capture appearance-based groupings rather than physically coherent entities 48 ; 17 . Self-supervised methods like CutLER 41 and ProMerge 22 combine attention maps from pretrained encoders 28 to identify object regions, but remain brittle in complex natural scenes. Object editing systems – including drag-based manipulation 45 ; 13 ; 7 ; 2 and depth-conditioned diffusion models 29 ; 15 – understand how objects move in 3D, but their performance degrades in cluttered scenes 7 . Methods like Visual Jenga 4 infer physical relationships between objects using a combination of off-the-shelf models. In contrast, PSI is a unified, physically controllable world model that discovers objects, understands how they move in 3D, and infers physical relationships in a zero-shot way without requiring specialized architectures. 3 PSI architecture and training details In this section, we first introduce the formulation of world models as probabilistic graphical models 20 . We then describe how these models can be realized using a sequence prediction architecture. Finally, we present a physically controllable visual world model built under this framework. Probabilistic World Models. Building models of the visual world involves learning distributions of future states of the world, conditioned on partial observations. To this end, we construct a probabilistic world model whose basic concept is to (1) partition the state of the world into a set of local variables and then (2) learn to predict the conditional probability distribution of each such variable, conditioned on any subset of these variables. Formally, we start with a data format that is described by a finite set of spatiotemporal pointer locations \mathcal{P} and a finite vocabulary \mathcal{V}