Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation

Atin Pothiraj, Jaemin Cho, Yue Zhang, Elias Stengel-Eskin, Mohit Bansal · 2026

Read Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation by Atin Pothiraj, Jaemin Cho, Yue Zhang, Elias Stengel-Eskin, Mohit Bansal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25306.txt.

Introduction

Generated videos can look sharp while breaking basic physics, like a paper towel dissolving instead of soaking up liquid. Atin Pothiraj and colleagues introduce a checking pipeline in 2026 that grades physical plausibility in fine detail. A vision language model turns a prompt into a graph of linked questions about objects, actions, and physical interactions, where later questions depend on earlier answers. Scores then pinpoint exactly which category of realism failed and where.

Human ratings on a new dataset, covering videos from three leading generators, matched the pipeline's fine grained scores better than earlier evaluation methods did. Closed source models ranked above an open model on physical realism. Follow up tests found models can write human like checking questions but still answer them worse than people do.

Catalog record

CREATORAtin Pothiraj, Jaemin Cho, Yue Zhang, Elias Stengel-Eskin, Mohit Bansal
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25306

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation Atin Pothiraj, Jaemin Cho, Yue Zhang, Elias Stengel-Eskin, Mohit Bansal Abstract Video generation models are increasingly capable of producing realistic videos, but they still struggle to generate videos that follow basic physical laws. Compounding this is a lack of reliable granular evaluation methods for localizing and specifying physical law violations in videos. We address this by introducing Physics Question Scene Graph ( PQSG ), a hierarchical question-based evaluation pipeline. PQSG evaluates generated videos by checking their faithfulness to a prompt across objects, actions, and adherence to physical laws using a graph-based hierarchy of questions generated by a vision-language model (VLM), guided by high-quality in-context examples. By representing questions as a graph, PQSG introduces logical dependencies within questions, ensuring that each query is contextually valid. Moreover, PQSG provides granular assessments of which qualities of the video violate physical plausibility constraints. We validate PQSG by creating FinePhyEval , a dataset with physics-based prompts and corresponding generated videos from diverse state-of-the-art video generation models (Sora 2, Veo 3, and Wan 2.1), with each video annotated across multiple categories by humans. Using FinePhyEval , we measure the correlation between PQSG ’s fine-grained scores and human judgments, showing higher overall correlations than prior work. We also find that PQSG ranks closed-source models higher than Wan 2.1 on physical realism. Lastly, we show that the annotations we provide in FinePhyEval can also be used for subtask evaluation: we benchmark two strong VLMs on generating and answering questions, finding that while models can create human-like questions, they still fall short of human performance in answering them. 1 Introduction Figure 1 : An example prompt and its corresponding generated video from Wan 2.1, with sample PQSG nodes and edges (not all are included) for each category. While the video contains the correct objects (a grabber, paper towel, bowl, etc.) it does not show all the right actions, and the physical interactions shown are implausible, with the paper towel dissolving into the liquid rather than absorbing it. PQSG specifies in which category the video is unrealistic (in this case, in its physics) and, within that category, which physical interactions are implausible (here, that the liquid fails to absorb into the paper towel, reflected in Physics Q2). An intuitive understanding of physical laws has been posited as a core component of human cognition [ 25 ] , allowing us to reason accurately about the world and the consequences of our actions in it. This capacity, often called “world modeling,” has long been a goal of video generation models [ 37 , 40 ] : to reason about future world states by rendering future video frames. Indeed, recent video generation models have become increasingly adept at generating photorealistic videos of objects and people in various environments [ 5 ] . However, while some understanding of basic physical laws has been documented even in infants [ 45 , 44 ] , current video generation models still struggle with physical realism, producing outputs that violate fundamental physical laws, such as solid mechanics, fluid dynamics, and optics [ 35 ] . Adherence to these laws is key for developing true world models that can serve as practical simulators. For example, a simulation of dish-washing must accurately represent water flow and the contact between a sponge and a dish. Such physical accuracy is critical for downstream applications in robotics [ 10 ] , embodied agent training [ 43 ] , and synthetic data generation [ 7 ] . Compounding current models’ shortcomings in producing physically-plausible videos is a lack of reliable, fine-grained, and automated methods for evaluating physics adherence. Existing methods provide only high-level, coarse-grained scores, failing to capture the inherent structure of physics evaluation. We argue that a robust physics evaluation must capture the logical dependencies within a scene. For example, to evaluate whether an object’s fall is physically plausible, one must verify that the object is present (object-level correctness) and that it is falling (action-level correctness). Only then can the physics of the fall (e.g., acceleration) be assessed. Existing methods (e.g., [ 35 , 18 , 24 ] ) lump these factors into a single, aggregate score, making it difficult to determine why a video failed. Moreover, these combined metrics are often fooled by videos that are visually realistic and temporally consistent, but may still be physically implausible [ 56 ] . For example, in Fig. 1 , baseline methods may assign a high score to a video where human annotators identify a clear physics violation (e.g., a paper towel dissolving instead of absorbing liquid). As models become more powerful, the errors they display will also become subtler, motivating the need for a fine-grained evaluation metric that can provide feedback on specific physical inaccuracies, both for evaluation and for fine-grained video repair [ 27 ] . To address this gap, we introduce the P hysics Q uestion S cene G raph ( PQSG ), a novel, structured evaluation framework for video generation models that measures fine-grained details of physical dynamics. PQSG is designed with two key properties: (1) PQSG is granular : rather than assessing adherence to physical laws in one, overarching score, it decomposes the assessment into multiple sub-queries, allowing for granular feedback on exactly which aspect of the video violates a physical law. (2) PQSG follows a dependency structure , ensuring that questions are asked in a sensible way: if an object is not present in a video, we do not subsequently ask about actions or physics interactions involving it. We illustrate this in Fig. 1 , where the sub-queries are divided into three hierarchical categories: object , action , and physics . Note that PQSG evaluates the semantics mentioned in text prompts — it assesses whether a generated video realizes the objects, actions, and physical interactions specified by its prompt. We formulate PQSG as a two-stage process: first, in the question-generation stage (QG), we task vision-language models (VLMs) with generating hierarchical physics-aware graphs of questions from text prompts. In the second stage – question-answering (QA) – these physics scene graphs are then passed with the generated video to a VLM to answer questions about the video, producing interpretable object, action, and physics scores, as seen in Fig. 1 (as well as Fig. 2 ). In the case of Fig. 1 , PQSG ’s QG stage generates targeted questions (e.g., questions about fluid dynamics) which are then answered, leading to appropriately low physics scores. To evaluate the components of PQSG (QG, QA) and subsequently evaluate diverse video generation models using PQSG , we collect FinePhyEval , a dataset of 195 human-annotated prompt-video pairs, with prompts sourced from Physics-IQ [ 35 ] and videos generated from three diverse state-of-the-art video generation models: Sora 2 [ 39 ] , Veo 3 [ 13 ] , and Wan 2.1 [ 47 ] . FinePhyEval provides comprehensive human annotations for: (1) video quality across four Likert-scale categories (object, action, physics, and overall quality), where we find strong inter-annotator agreement; and (2) PQSG ’s constituent subtasks of question-generation (QG) and question-answering (QA). On FinePhyEval , we first show that PQSG results in better correlations to overall human ratings than prior metrics, and that –

Read the complete text