NewtPhys: Do Foundation Models Understand Newtonian Physics?

Sebastian Cavada Soumava Paul Tuan-Hung Vu Andrei Bursuc Raoul de Charette · 2026

Read NewtPhys: Do Foundation Models Understand Newtonian Physics? by Sebastian Cavada Soumava Paul Tuan-Hung Vu Andrei Bursuc Raoul de Charette (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03986.txt.

Introduction

Cavada and co-authors build NewtPhys, a data set for low-level physics sense. Past tests used fake or half-fake scenes with question answering. Those tests stressed high-level events, not fine physical detail. NewtPhys uses multi-view images of real scenes. Physics-based simulation adds dense labels over time.

Labels include 3D forces and per-pixel measures of motion and form. They also cover tracking, semantics, and geometry at each step. The team tests 56 vision-language models on the set. Ten vision foundation models were tested as well. Results reveal clear limits in low-level physics reasoning. The set also aims to guide future physics-aware tests and models.

Catalog record

CREATORSebastian Cavada Soumava Paul Tuan-Hung Vu Andrei Bursuc Raoul de Charette
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03986

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NewtPhys: Do Foundation Models Understand Newtonian Physics? Sebastian Cavada Soumava Paul Tuan-Hung Vu Andrei Bursuc Raoul de Charette Abstract Previous work has evaluated physics reasoning in foundation models using synthetic or semi-synthetic scenes and visual question-answering tasks. However, these benchmarks emphasize high-level events and lack the visual fidelity required to assess true low-level Newtonian understanding. We introduce NewtPhys , a 4D physically annotated dataset built from multiview images of real-world scenes with physics-grounded simulations. The dataset provides dense, fine-grained annotations across timesteps — including 3D forces and amodal per-pixel quantities covering physics, tracking, semantics and geometry — bridging the gap between simplistic synthetic setups and realistic visual complexity. Using NewtPhys, we systematically evaluate 56 VLMs, including 54 open-weight models and 2 closed-source frontier models, and 10 VFMs and reveal limitations in low-level physics reasoning. Beyond benchmarking, our dataset enables future research in physics-grounded vision and the development of next-generation physics-aware evaluations. Code and datasets are available at: https://astra-vision.github.io/NewtPhys . 1 Introduction Humans develop intuitive Newtonian models early in life 1 , enabling prediction of motion, contact, and gravity, which allow them to navigate the physical world and solve complex visual tasks 36 ; 7 . Building vision systems with comparable physical grounding remains a key goal, especially for embodied agents. Recent Vision-Language Models (VLMs) and Vision Foundation Models (VFMs) excel at recognition and open-ended reasoning, raising the question of whether they truly understand the causal structure of physics or merely rely on correlations. However, common evaluations based on spatial awareness or 3D reasoning do not directly test physics awareness, and whether they understand low-level quantities such as forces, collisions, gravity, or deformation 63 ; 16 is yet unanswered. Existing benchmarks face a trade-off: realistic datasets rely on human judgments or high-level proxy labels and therefore do not provide physics labels, while benchmarks with explicit Newtonian annotations use simplified synthetic worlds 58 ; 44 ; 4 or depict simplistic real-world dynamics in lab conditions 64 . To bridge this gap, we introduce NewtPhys, a 4D benchmark for low and high-level Newtonian understanding in realistic settings. While physical annotation is impractical outside lab conditions, NewtPhys uses 3D Gaussian Splatting (3DGS) 26 to combine real-world scenes and objects as simulatable particles of a Newtonian simulator. Our benchmark not only captures the visual appearance as videos, but also pixel-aligned physical labels and events over time, including forces, deformation fields, material and instance maps, and scene flow, illustrated in . We use NewtPhys to conduct a large-scale study on 56 Vision Language Models (VLMs) and 10 Vision Foundation Models (VFMs). For VLMs, we create a physics-grounded Visual Question Answering (VQA) dataset, covering 141K question-answer pairs, across six physics understanding categories. Our extensive study shows that although advanced VLMs continue to grow in scale and capability alongside the evolution of LLMs, progress in physics understanding are surprisingly limited. While frontier models continue to dominate the benchmark, large-scale open-source models are narrowing the performance gap. While these models can often identify materials, they struggle to estimate granular Newtonian properties like density or mass, suggesting that VLMs are not yet capturing the causal mechanics required for true physical world modeling. Our study leads to two key observations. First, VLMs tend to ignore visual cues and rely on easier shortcuts derived from priors learned by LLMs. Second, most popular commonsense benchmarks correlate weakly with physics understanding, further hindering progress in physical reasoning. Benefiting from NewtPhys pixel-wise physics annotation, we also benchmark 10 vision-only VFMs, introducing the task of Physics Probing to evaluate whether Newtonian signals are accessible in the learned representations. While stronger visual representations generally improve Physics Probing performance, our results suggest that VFMs are not inherently physics-grounded and may fail at reliably capturing physically meaningful signals. Our contributions are fourfold: • NewtPhys, a benchmark combining real 3DGS scenes with Newtonian simulation to produce rendered videos with dense pixel-aligned physical annotations. • A point-based simulation pipeline supporting multi-object interactions and soft-body dynamics in realistic scenes. • A large-scale dataset includes 141K VQA spanning six categories and 730K frames with 11 ground-truth maps, enabling statistically meaningful evaluation. • Comprehensive evaluations of 66 VFMs/VLMs revealing current limitations in low-level Newtonian physics understanding. 2 Related works The idea that humans rely on an internal, approximate model of physics to understand and predict the visual world has long been studied in cognitive science under the umbrella of intuitive physics 36 ; 1 ; 7 . This perspective has strongly influenced machine learning and vision, where early and foundational work framed physical reasoning as inference over latent variables and object-centric world models 48 ; 3 ; 2 . In parallel, others explored how visual representations can capture causal structure and physical regularities from data 24 ; 66 . A substantial body of work aims to learn physical laws or latent dynamics from visual input, typically in simplified environments 56 ; 55 . These works demonstrate that structured physical representations can be learned, but they largely operate in toy 2D or simple 3D worlds with limited visual and physical complexity. Some benchmarks study physical reasoning in richer scenes, including event prediction, counterfactual reasoning, and causal queries 58 ; 44 ; 4 . However, physics annotations in these datasets is still mostly symbolic or event-level. High-level understanding IntPhys, 2021 IntPhys2, 2025 PhysBench, 2025 PhysicsIQ, 2025 Low-level understanding Physics 101, 2016 Physion, 2021 CompPhy, 2022 CRIPPVQA, 2022 ContPhy, 2024 PhysBench*, 2025 Physion++, 2023 PhyWorld, 2025 5 objects colliding (crop) 4 soft objects Flexible pencil cases (crop) 5 objects interacting 10 objects in free fall Multi-way interactions NewtPhys (high- and low-level understanding) Figure 1 : Benchmarks for physical understanding. High-level physical understanding benchmarks ( e.g . , event ordering, general physics, frame reconstruction) are typically more realistic than low-level ones, which rely on toy simulators. In contrast, NewtPhys provides a realistic benchmark for both high- and low-level physical understanding in real-world scenes. Recent benchmarks move towards more realistic data and foundation model evaluation. Physics-IQ evaluates generative video models based on physical plausibility judged by learned or human critics 38 while PhysBench evaluates VLMs on image–video–text questions about physical situations in real-world content 12 . NewtonGen and related datasets test qualitative or counterfactual physical reasoning 60 . While these benchmarks reveal important limitations of current models, they do not provide ground-truth Newtonian quantities and therefore cannot determine whether models represent forces, stresses, or contact patterns internally, rather than relying on high-level visual or statistical cues. Some benchmarks target more diagnostic or application-oriented settings, such as PhysToolBench 65 , PISA 28 , or Morpheus 64 . There also exist works on deformable or soft-body simulation environments 51 . To our knowledge, no benchmark provides dense, pixel-aligned supervision of forces and deformations in visually realistic scenes. NewtPhys is designed to fill this gap: it provides per-pixel physical fields (forces, collisions, deformations, scene flow) aligned with rendered videos, enabling direct evaluation of whether vision models encode low-level Newtonian signals rather than only high-level physical plausibility. Figure 1 visualizes existing benchmarks 1 1 1 Note PhysBench shows large variability in visual realism, from high-level (photographic) to low-level (toy-simulation) physics. (top) and NewtPhys (bottom) which, in contrast, has realistic renderings and rich physical interactions for low- and high- physics understanding. 3 The NewtPhys

Read the complete text