PAI-Bench : A Comprehensive Benchmark For Physical AI
Fengzhe Zhou 1,* Jiannan Huang 1,* Jialuo Li 1,* Deva Ramanan 2 Humphrey Shi 1 1 Georgia Tech 2 CMU · 2025
Read PAI-Bench : A Comprehensive Benchmark For Physical AI by Fengzhe Zhou 1,* Jiannan Huang 1,* Jialuo Li 1,* Deva Ramanan 2 Humphrey Shi 1 1 Georgia Tech 2 CMU (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01989.txt.
Introduction
Physical AI aims at models that can perceive the real world and predict how it changes, a skill robots and vehicles depend on. Fengzhe Zhou and co-authors introduce PAI-Bench in 2025, a broad benchmark for measuring how well today's multimodal language models and video generators support that aim.
How far current models really grasp physical dynamics was poorly understood, because most benchmarks test words and pictures, not motion and cause. PAI-Bench evaluates models on their ability to perceive and forecast real-world behavior. The results give the field a shared yardstick for a capability that matters more as AI moves into physical settings.
Catalog record
| CREATOR | Fengzhe Zhou 1,* Jiannan Huang 1,* Jialuo Li 1,* Deva Ramanan 2 Humphrey Shi 1 1 Georgia Tech 2 CMU |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01989
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PAI-Bench : A Comprehensive Benchmark For Physical AI Fengzhe Zhou 1,* Jiannan Huang 1,* Jialuo Li 1,* Deva Ramanan 2 Humphrey Shi 1 1 Georgia Tech 2 CMU Abstract Physical AI aims to develop models that can perceive and predict real-world dynamics; yet, the extent to which current multi-modal large language models and video generative models support these abilities is insufficiently understood. We introduce Physical AI Bench (PAI-Bench), a unified and comprehensive benchmark that evaluates perception and prediction capabilities across video generation, conditional video generation, and video understanding, comprising 2,808 real-world cases with task-aligned metrics designed to capture physical plausibility and domain-specific reasoning. Our study provides a systematic assessment of recent models and shows that video generative models, despite strong visual fidelity, often struggle to maintain physically coherent dynamics, while multi-modal large language models exhibit limited performance in forecasting and causal interpretation. These observations suggest that current systems are still at an early stage in handling the perceptual and predictive demands of Physical AI. In summary, PAI-Bench establishes a realistic foundation for evaluating Physical AI and highlights key gaps that future systems must address. \projectlinks Code SHI-Labs/physical-ai-bench Leaderboard shi-labs/physical-ai-bench-leaderboard PAI-Bench-G Dataset shi-labs/physical-ai-bench-generation PAI-Bench-C Dataset shi-labs/physical-ai-bench-conditional-generation PAI-Bench-U Dataset shi-labs/physical-ai-bench-understanding 1 1 footnotetext: Equal contribution 1 Introduction “Human intelligence was not forged in a vacuum. It was sculpted by the relentless demands of the physical world." We learned to think because we had to move, grasp, and survive. AI systems stand at a similar threshold. To mature from a mere information processor into a truly capable partner, they must leave the clean, predictable confines of digital simulation and enter the messy, unforgiving reality. This is the core mission of Physical AI : the embodiment of intelligent algorithms into autonomous systems that must perceive, predict, and act directly within the physical world. In this work, we focus on the perception and prediction components of this mission through visual signals, specifically video streams. Regarding perception , while recent multi-modal large language models (MLLMs) have demonstrated escalating capabilities with increased scale [ 51 , 8 , 76 ] , their advancements are predominantly validated using benchmarks for abstract reasoning (e.g., OCR [ 30 ] , mathematical problems [ 72 ] ) and simple daily-life perception [ 102 , 29 , 70 ] . Their true efficacy within specialized Physical AI applications remains largely uncharted. Similarly, in the domain of prediction , a Physical AI agent must rely on an internal world model to forecast the consequences of actions and complex physical dynamics. Video generative models (VGMs) [ 62 , 16 ] represent a powerful new paradigm for learning such dynamics, as they are implicitly trained to understand physics rules by predicting future video frames. However, existing benchmarks [ 38 , 39 , 101 , 53 ] for these models have largely focused on assessing aesthetic appeal and temporal consistency, with far less attention paid to evaluating their fundamental understanding of real-world rules. It thus remains unclear whether they can model complex interactions, adhere to physical laws, or make reasonable predictions in practical applications. To address these deficiencies and advance Physical AI, we introduce Physical AI Bench ( PAI-Bench ), a comprehensive benchmark designed to systematically evaluate model performance on tasks grounded in Physical AI. PAI-Bench is structured into three distinct tracks: • PAI-Bench-G (Generation): Targets prediction by evaluating visual quality and physical plausibility of VGMs. • PAI-Bench-C (Conditional Generation): Further probes prediction by evaluating the fidelity of conditional VGMs, specifically their conformance to input control signals (e.g., depth maps). • PAI-Bench-U (Understanding): Targets the perception component by evaluating MLLM capabilities on video understanding tasks specific to the Physical AI domain. Figure 1 : Overview of PAI-Bench framework. PAI-Bench is a comprehensive bench designed for diverse topics in Physical AI, including evaluation for text and condition to physical World Generation, and physical world understanding. All three tracks adhere to a unified design principle: grounding evaluation in physically meaningful tasks and real-world data. Specifically, all videos are sourced from real-world captures, such as dashcam recordings. Furthermore, PAI-Bench covers several sub-domains, such as autonomous vehicles, robotics, ego-view scenes, etc. , across three tracks, comprising 2,808 high-quality cases. Based on PAI-Bench, we conducted a comprehensive evaluation of leading VGMs and MLLMs. Extensive experiments reveal that while SOTA VGMs like Veo3 [ 21 ] achieve high visual fidelity, their prediction abilities are limited, as they often fail to adhere to basic physical laws or to model complex real-world dynamics. Concurrently, the perception capabilities of powerful MLLMs, such as GPT-5 [ 61 ] , lag substantially behind human performance. These findings suggest that the core prediction and perception capabilities required for Physical AI remain in a nascent stage. In summary, our contributions are three-fold: • We introduce PAI-Bench, the first benchmark to provide a unified and comprehensive review of video generation, conditional video generation, and video understanding. • As the first high-quality benchmark grounded in Physical AI, PAI-Bench spans diverse scenarios (e.g., industrial environments, everyday activities, physical common sense) and incorporates a suite of task-aligned metrics. • We conduct a large-scale evaluation of 15 VGMs, 4 conditional VGMs with 5 control settings, and 16 MLLMs, establishing the current landscape of Physical AI and highlighting key challenges to guide future research. 2 Related Work Prediction and Perception are foundational capabilities for Physical AI applications. However, a significant gap persists in current benchmarking: the absence of a comprehensive framework specifically for the Physical AI domain. Furthermore, existing benchmarks are largely fragmented, typically assessing prediction or perception in isolation. In the realm of prediction , world models [ 34 , 12 , 4 , 16 ] are increasingly vital for applications such as robotics and autonomous driving. A prominent paradigm utilizes VGMs to forecast high-fidelity future frames [ 4 , 62 , 21 , 1 , 17 , 93 , 97 , 79 , 36 , 21 , 88 , 62 , 60 , 97 ] . This progress has spurred the proposal of numerous benchmarks [ 26 , 41 , 53 , 54 , 57 , 98 ] . However, existing evaluation methodologies are often narrow in scope. Some benchmarks [ 38 , 101 , 39 ] primarily assess visual quality and temporal consistency, while others [ 57 , 14 , 11 , 22 ] concentrate exclusively on physical plausibility. We introduce PAI-Bench-G , which distinguishes itself by integrating both aspects via a domain score and a quality score, with a strong emphasis on Physical AI applications. Furthermore, a significant evaluation gap has emerged as VGMs [ 3 , 4 , 79 , 21 , 36 , 97 , 62 ] increasingly leverage multimodal control signals (e.g., depth maps) for guided synthesis. No existing benchmark systematically tests this controllability. PAI-Bench-C is therefore introduced as the first benchmark to systematically evaluate this condition guided generation capability. Similarly, in the perception domain, while numerous benchmarks exist for general video understanding (e.g., action recognition) [ 70 , 102 , 29 , 90 , 72 , 49 , 37 ] , evaluations for physically grounded reasoning remain notably limited, particularly for video-based analysis [ 27 , 73 , 48 , 20 , 59 ] . PAI-Bench-U bridges this gap by unifying tasks for general video comprehension with