WorldBench : Disambiguating Physics for Diagnostic Evaluation of World Models
Rishi Upadhyay, Howard Zhang, Jim Solomon, Ayush Agrawal, Pranay Boreddy, Shruti Satya Narayana, Yunhao Ba, Alex Wong, Celso M de Melo, Achuta Kadambi · 2026
Read WorldBench : Disambiguating Physics for Diagnostic Evaluation of World Models by Rishi Upadhyay, Howard Zhang, Jim Solomon, Ayush Agrawal, Pranay Boreddy, Shruti Satya Narayana, Yunhao Ba, Alex Wong, Celso M de Melo, Achuta Kadambi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21282.txt.
Introduction
Rishi Upadhyay and nine co-authors build a test for whether AI world models truly grasp physics. These generative models are pitched for hard jobs like robot planning and training self-driving systems. For such work, a model must mimic real dynamics with high fidelity, not just look plausible.
Their benchmark, WorldBench, is designed to separate look-alike cases that obey different physical rules. A model that confuses them will fail when the hidden rule matters. The tests work as a diagnostic, showing which parts of physics a model has learned. That helps builders fix weak spots before trusting a model with real tasks.
Catalog record
| CREATOR | Rishi Upadhyay, Howard Zhang, Jim Solomon, Ayush Agrawal, Pranay Boreddy, Shruti Satya Narayana, Yunhao Ba, Alex Wong, Celso M de Melo, Achuta Kadambi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21282
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WorldBench : Disambiguating Physics for Diagnostic Evaluation of World Models Rishi Upadhyay, Howard Zhang, Jim Solomon, Ayush Agrawal, Pranay Boreddy, Shruti Satya Narayana, Yunhao Ba, Alex Wong, Celso M de Melo, Achuta Kadambi Abstract Recent advances in generative foundational models, often termed ”world models,” have propelled interest in applying them to critical tasks like robotic planning and autonomous system training. For reliable deployment, these models must exhibit high physical fidelity, accurately simulating real-world dynamics. Existing physics-based video benchmarks, however, suffer from entanglement, where a single test simultaneously evaluates multiple physical laws and concepts, fundamentally limiting their diagnostic capability. We introduce WorldBench, a novel video-based benchmark specifically designed for concept-specific, disentangled evaluation, allowing us to rigorously isolate and assess understanding of a single physical concept or law at a time. To make WorldBench comprehensive, we design benchmarks at two different levels: 1) an evaluation of intuitive physical understanding with concepts such as object permanence or scale/perspective, and 2) an evaluation of low-level physical constants and material properties such as friction coefficients or fluid viscosity. When SOTA video-based world models are evaluated on WorldBench, we find specific patterns of failure in particular physics concepts, with all tested models lacking the physical consistency required to generate reliable real-world interactions. Through its concept-specific evaluation, WorldBench offers a more nuanced and scalable framework for rigorously evaluating the physical reasoning capabilities of video generation and world models, paving the way for more robust and generalizable world-model-driven learning. Figure 1 : We introduce WorldBench, a video-based benchmark to evaluate world foundation model performance on specific physical concepts/constants and material properties. Prior benchmarks typically either entangle multiple concepts (making it difficult to uniquely identify model failures) or use coarse-grained metrics like binary selection (making it difficult to disambiguate between visually realistic and physically accurate trajectories). Our benchmark leverages both an intuitive physics and physical parameter estimation subset to provide greater insight into world model performance. Figure 2 : Overview of our generation and evaluation process . For generation (top), we use Kubric, which uses PyBullet and Blender under the hood. During evaluation (bottom), we first pass the initial frames of the generated video to the world foundation model which completes the video. The completed video is passed to SAM2 along with bounding boxes based on ground truth masks. The segmentations outputted by SAM2 are compared to ground truth segmentations to obtain the final metrics. 1 Introduction Imagine watching a tower of blocks teeter and fall, or a ball rolling its way down a staircase. As humans, we effortlessly predict its motion. However, this intuitive grasp of physical dynamics remains a core challenge for AI. Recent world foundation models, most notably NVIDIA’s Cosmos 2 , promise to learn such skills at scale, suggesting that these models can be used as synthetic data generators for the real world. Rigorously evaluating these claims requires benchmarks that are designed and focused on probing physical understanding at a concept-specific level. Existing benchmarks for physical reasoning tasks often provide coarse-grained or binary metrics on scenes that entangle multiple physics concepts, limiting their diagnostic ability. For example, benchmarks such as PHYRE 6 or CLEVRER 60 contain scenes which require a core understanding of multiple concepts (perspective accuracy, collision dynamics, and support relations). While some benchmarks like Physion 7 and IntPhys2 41 ; 11 do provide some level of granularity in testing physical concepts, they do not contain experiment-level tests of specific physical parameters or material constants. Furthermore, a reliance on coarse-grained or binary metrics like object contact prediction limit their ability to capture nuanced physical phenomena, such as object dynamics (velocity, acceleration, rotation, etc.), deformation, or occlusion. In this paper, we introduce WorldBench , a novel benchmark designed to rigorously evaluate the disentangled, concept-specific, physical reasoning capabilities of WFMs through video prediction. Our testing framework uses fine-grained categories of evaluation to fill in a critical gap in in the current research landscape. The video-based output requires models to accurately forecast the physically plausible evolution of full visual scenes over time, providing a more robust signal than previous binary metrics. To achieve this nuanced assessment while ensuring repeatable, interpretable outcomes, we design simplified, yet physically rich and visually realistic scenes. Our benchmark is split into two subsets. The intuitive physical understanding subset targets four critical principles: motion physics, object permanence, support relations, and scale/perspective. This subset is meant to benchmark a model’s ability to generate plausible dynamics governed mostly by their respective core principles (e.g. ball rolling behind pillars, object moving towards the camera, etc.). WorldBench also includes a physical parameter estimation subset. This feature requires the video output to accurately adhere to specific, known physical parameters that govern the scene’s dynamics, such as gravitational acceleration, fluid viscosity, and friction coefficients. By enforcing adherence to these measurable parameters, we provide a definitive, quantifiable metric for assessing a model’s true grasp of each underlying physical law. Our proposed task, which we term ”constrained video prediction,” allows for a significantly more nuanced and detailed assessment of physical laws than previous work, including a more fine-grained diagnosis of which concept a model underperforms on, and an accurate representation of object dynamics (velocity, acceleration, rotation, deformation, and occlusion). The first subset’s focus on specific, real-world physics properties, such as object permanence and scale/perspective—which are absent in prior benchmarks like Physion—ensures that WorldBench provides a richer signal of how close these models are to truly learning and understanding real-world dynamics. The second subset’s focus on real-world physical parameters also enables us to directly measure the physical accuracy of world models, which is particularly critical if these models aim to be synthetic data generators. For future work in the evaluation of these world models, our benchmark also leads to a wider array of downstream tasks, such as object tracking, anomaly detection, action planning, etc. Using WorldBench , we extensively test state-of-the-art world models, including the Cosmos architecture, revealing substantial gaps in physical consistency and generalization when compared to both real-world captured and simulated, physically accurate expectations. Importantly, we find that models tend to generate visually realistic scene evolutions (e.g. ball follows parabolic trajectory), but fail to adhere to physical parameters (e.g. ball accelerates downward at 9.8\frac{m}{s}^{2} ). By leveraging video outputs rather than binary selection tasks, our benchmark provides a way to quantitatively disambiguate between visually plausible and physically accurate model outputs. Our results highlight the critical limitations of current world model architectures and motivate further research into physically grounded learning. Below is a summary of our key contributions: • We introduce a video-based benchmark comprised of real and synthetic videos to evaluate WFM performance on specific physical concepts and parameters. • Our first subset provides scenes designed to test a model’s qualitative understanding of foundational concepts such as object permanence, scale/perspective, support relations, and motion physics. • Our second subset provides carefully designed experiments which directly measure a models ability to reproduce specific well-defined physics constants and behaviors such as gravitational acceleration, fluid viscosity, and friction coefficients. • We perform an empirical analysis of SOTA WFMs and Image-to-Video models to identify concept-specific shortcomings and gaps in physical understanding. Figure 3 : Overview of our physical parameter estimation pipeline. Given an input video, we first use checkerboard detection and SAM2 to extract