Watching Physics: the Generative Science of Matter and Motion

Hagen Holthusen, Kevin Linka, Ellen Kuhl · 2026

Read Watching Physics: the Generative Science of Matter and Motion by Hagen Holthusen, Kevin Linka, Ellen Kuhl (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16843.txt.

Introduction

Holthusen, Linka and Kuhl ask whether physics of matter in motion can be learned from images and video. Images and video encode structure, dynamics and cause that equations alone cannot fully capture. Generative video models gain scientific value coupling visual data with experiments and simulations. Across rubber compression, can crushing and cardiac motion, the authors map where visual learning succeeds or fails. Visible kinematics yield measurable surface strain, while dominant internal variables break the plausibility-admissibility link. The authors name this convergence the Generative Sciences of Matter and Motion.

Catalog record

CREATORHagen Holthusen, Kevin Linka, Ellen Kuhl
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33261
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16843

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Watching Physics: the Generative Science of Matter and Motion Hagen Holthusen, Kevin Linka, Ellen Kuhl Abstract Can we learn the physics of matter in motion directly from images and video—and trust it? Answering this question requires integrating experiments, physics-based simulation, and data across traditionally separate disciplines. Much of this knowledge is visual and temporal rather than textual: images and videos encode structure, dynamics, and causality that equations alone cannot fully capture. Recent generative models produce compelling visual content, yet they rely on observational data and often lack physical validity. Here we show that generative video models gain scientific value when they couple visual data with experiments and high-fidelity simulations. Using deformation mechanics as a testbed, we study three systems of increasing complexity—rubber compression, can crushing, and cardiac motion—and identify regimes in which visual learning succeeds, fails, and requires mechanistic supervision. When physics manifests in visible kinematics, generative models recover measurable quantities such as surface strain; when internal state variables dominate, visual plausibility no longer ensures physical admissibility. We propose that this convergence defines a new frontier, the Generative Sciences of Matter and Motion , which unifies Simulogenics, Physiogenics, and Materiogenics. These physics-grounded foundation models can turn visual generation into a scientific instrument for inference, prediction, and design of matter in motion. keywords generative artificial intelligence, video generation, experiment, simulation, physics-informed machine learning Motivation For centuries, science has sought to explain the world through equations of matter and motion. From Hooke’s law of elasticity hooke1678 to Newton’s laws of dynamics newton1687 and the Navier–Stokes equations of fluids navier1822 ; stokes1845 , physics has advanced by abstracting nature into mathematical form. These abstractions have enabled predictive simulation and transformed computers into laboratories for virtual experimentation vonneumann1945 . In parallel, artificial intelligence has evolved from symbolic reasoning turing1950 ; mccarthy1955 to deep neural networks that generate text, images, and video lecun2015 ; goodfellow2014 . Information theory has formalized this transition from description to representation shannon1948 , and modern generative models now learn the latent structure of the data themselves ho2020 ; brown2020 . Together, these developments mark a shift from explaining nature to generating it and unify physics and AI into the Generative Sciences of Matter and Motion . Figure 1: Generative Science of Matter and Motion. Conceptual workflow of the proposed generative-video-AI framework. On the input side, the model is trained on diverse video streams and learns deformation patterns of materials and structures under external loading from visual dynamics. To improve physical fidelity and quantitative consistency, this learning process is further reinforced with experimentally observed responses and video sequences from high-fidelity computer simulations. The resulting latent representation enables a material and structural understanding and supports inference of physically plausible generated simulations. On the output side, the framework supports the generation and inference of physically plausible motion and deformation across scales. This includes not only visually realistic video sequences, but also quantitatively meaningful predictions that can be evaluated, conditioned, or refined through physics-aware inference. The resulting paradigm bridges observation, simulation, and generation, and enables applications such as inverse material characterization, virtual experimentation, and predictive modeling of complex systems. Simulogenics–From equations to generation. Simulogenics establishes the generative science of physical simulation, where models learn to produce physically consistent trajectories of matter and motion. Classical mechanics has grounded this effort in variational principles and conservation laws hestenes1973 ; newton1687 , and continuum theories have formalized the dynamics of complex media navier1822 ; stokes1845 . The finite element method has translated these equations into computational frameworks for predictive simulation zienkiewicz1977 , and digital twins have extended this paradigm toward real-time predictive systems grieves2014 . Modern generative models now invert this process and learn to generate solutions directly from data sohl2015 ; ho2020 . This convergence suggests that simulation itself can be learned and bridge first-principles physics with data-driven generation to enable learning simulation without equations . Physiogenics–From observation to governing laws. Physiogenics defines the generative science of inferring physical laws from visual observations of matter in motion. Foundational theories of perception have framed vision as an inverse problem that reconstructs the world from images marr1982 ; gibson1979 , and statistical physics has demonstrated that complex dynamics follow structured stochastic laws kolmogorov1941 . Quantum mechanics has further recast physical behavior in probabilistic terms feynman1948 . Recent advances in generative video models now encode rich dynamical priors directly from visual data ho2022video ; ha2018world . This perspective suggests that physical laws need not be prescribed explicitly, but can instead emerge from observation which elevates video to a primary modality for scientific discovery, and enables learning physics from vision . Materiogenics–From structure to functional design. Materiogenics introduces the generative science of designing matter with prescribed properties and functions. Classical materials science has established structure–property relationships as the foundation for materials selection and engineering design ashby1996 . Advances in topology optimization have enabled systematic inverse design of high-performance structures bendsoe2003 . At the same time, foundational work in biophysics has connected physical laws to the organization of living matter and highlighted the interplay between physics, biology, and function schrodinger1944 . Recent advances in self-supervised representation learning demonstrate how we can capture structured physical dynamics from data, which results in predictive world models grounded in temporal coherence and interaction assran2025vjepa2 . Generative models now navigate high-dimensional design spaces and propose novel material configurations goodfellow2014 ; brown2020 . Yet, these models increasingly benefit from the encoding of simple physical simulations, such as rigid-body motions, which provide low-dimensional structure to ensure physically consistent generation liu2024physgen . This shift reframes materials design as a generative process and enables learning matter by design . In the context of mechanics and materials science, these advances prompt a fundamental question: Can models trained on unstructured video observations alone infer the underlying physics that govern matter and motion? Although large-scale datasets contain extensive real-world dynamics, they rarely encode constitutive response, state variables, loading paths, or boundary conditions explicitly. Computational simulations provide these quantities with precision; yet, current generative models rarely incorporate them. As a result, existing approaches achieve strong visual realism but often lack quantitative reliability in physically demanding scenarios. Outline. This perspective operationalizes Physiogenics by using deformation mechanics as a testbed to determine whether generative video AI can infer physics beyond visual realism. We establish that Physiogenics provides the missing link between data-driven observation and generative simulation; it directly connects to Simulogenics , and opens the path toward Materiogenics . The framework integrates visual observations, experiments, and high-fidelity simulations to learn a latent representation that encodes the governing mechanics of matter and motion and supports generation and quantitative inference (Figure 1 ). We instantiate this paradigm through a staged benchmark with three scenarios of increasing complexity: (i) uniaxial deformation, where the model recovers kinematics in the spirit of digital image correlation and compares them to finite element references; (ii) nonlinear thin-walled collapse in can crushing, validated against experiments and simulations; and (iii) cardiac deformation over one heartbeat as a high-dimensional biomechanical test case. These examples map input to latent representation to output, quantify capabilities, expose limitations, and define the requirements for learning physics from vision , learning simulation without equations , and learning matter by design . Integrating experiment, generation, and simulation In the following three exploratory

Read the complete text