Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulation
Chuhao Chen, Zhiyang Dou, Chen Wang, Yiming Huang, Anjun Chen, Qiao Feng, Jiatao Gu, Lingjie Liu, University of Pennsylvania · 2025
Read Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulation by Chuhao Chen, Zhiyang Dou, Chen Wang, Yiming Huang, Anjun Chen, Qiao Feng, Jiatao Gu, Lingjie Liu, University of Pennsylvania (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06440.txt.
Introduction
Chuhao Chen leads a team of eight authors in 2025 in building Vid2Sim, a system that watches a video and rebuilds the scene as a simulation. It recovers the object's look, its shape, and its physical traits, such as stiffness, so the scene can be replayed and pushed around in a physics engine.
Older methods tune a separate heavy pipeline for each new scene, which is slow and fiddly. Vid2Sim instead learns a general skill that carries across scenes, and it runs the simulation without a mesh, the usual scaffolding of triangles. A general tool like this could serve robotics, film, and design alike.
Catalog record
| CREATOR | Chuhao Chen, Zhiyang Dou, Chen Wang, Yiming Huang, Anjun Chen, Qiao Feng, Jiatao Gu, Lingjie Liu, University of Pennsylvania |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06440
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulation Chuhao Chen, Zhiyang Dou, Chen Wang, Yiming Huang, Anjun Chen, Qiao Feng, Jiatao Gu, Lingjie Liu, University of Pennsylvania Abstract Faithfully reconstructing textured shapes and physical properties from videos presents an intriguing yet challenging problem. Significant efforts have been dedicated to advancing such a system identification problem in this area. Previous methods often rely on heavy optimization pipelines with a differentiable simulator and renderer to estimate physical parameters. However, these approaches frequently necessitate extensive hyperparameter tuning for each scene and involve a costly optimization process, which limits both their practicality and generalizability. In this work, we propose a novel framework, Vid2Sim, a generalizable video-based approach for recovering geometry and physical properties through a mesh-free reduced simulation based on Linear Blend Skinning (LBS), offering high computational efficiency and versatile representation capability. Specifically, Vid2Sim first reconstructs the observed configuration of the physical system from video using a feed-forward neural network trained to capture physical world knowledge. A lightweight optimization pipeline then refines the estimated appearance, geometry, and physical properties to closely align with video observations within just a few minutes. Additionally, after the reconstruction, Vid2Sim enables high-quality, mesh-free simulation with high efficiency. Extensive experiments demonstrate that our method achieves superior accuracy and efficiency in reconstructing geometry and physical properties from video data. Figure 1 : Vid2Sim achieves high-quality reconstruction of appearance, geometry, and physics from multi-view videos effectively. The reconstruction results are simulation-ready, enabling high-fidelity and visually appealing animations via mesh-free simulation. Here, we present our method’s reconstruction and simulation results in the GSO Downs et al. 2022 dataset. 1 Introduction Understanding and reconstructing appearance, geometry, and physical properties from observations with high fidelity, a.k.a. system identification, is a fundamental yet challenging task in computer vision. Traditional methods Qiao et al. 2021 ; Rojas et al. 2021 ; Du et al. 2021 ; Geilinger et al. 2020 ; Heiden et al. 2021 ; Jatavallabhula et al. 2021 ; Ma et al. 2022 often rely on known shape information of given objects, which limits their practicality for broader applications. Recent advancements Li et al. 2023 ; Kaneko 2024 ; Zhong et al. 2025 ; Cai et al. 2024 leverage neural representations, such as NeRF Mildenhall et al. 2021 and Gaussian Splatting Kerbl et al. 2023 along with differentiable simulators Jiang et al. 2016 to create a unified framework that jointly learns 3D geometry, appearance, and physical parameters. That being said, none of the previous efforts have achieved accurate, generalizable, and efficient reconstruction of appearance, geometry, and physical properties from the input video, as they suffer from two main limitations. First, most existing methods Li et al. 2023 ; Kaneko 2024 ; Zhong et al. 2025 ; Cai et al. 2024 employ heavy per-scene optimization to identify physical parameters, making the understanding of various scenes computationally expensive. Second, these approaches struggle to accurately model complex, physics-driven deformations, as they typically use Material Point Methods (MPM) Jiang et al. 2016 for simulation. This method is limited by its grid-based representation and its typical dependence on symplectic time integration, which constrains expressiveness. Although alternative approaches, such as Spring-Gaus Zhong et al. 2025 , employ more efficient mass-spring models, they are limited to modeling elastic dynamics. In this paper, we propose a novel framework, named Vid2Sim, for the high-fidelity reconstruction of textured shapes and the estimation of physical properties directly from videos. We first train a feed-forward neural network that integrates general physical knowledge, utilizing a pre-trained video vision transformer Tong et al. 2022 to infer a range of physical attributes from the input video sequences. This component is coupled with an advanced 3D reconstruction pipeline Tang et al. 2025 that predicts both object geometry and appearance, encoded with 3D Gaussians to facilitate instant system identification. In contrast to prior methods, Vid2Sim incorporates an efficient simulation pipeline leveraging an implicit Euler solver as inspired by Modi et al. 2024 . This simulation approach is mesh-free and uses Linear Blend Skinning (LBS) to enable reduced-order, computationally efficient simulations that are highly adaptable to complex deformations and fully end-to-end trainable. Then, we perform a lightweight optimization with a novel Neural Jacobian module to efficiently refine estimates of appearance, geometry, and physical properties, aligning the reconstructed outputs precisely with observed video data. This post-prediction optimization completes in only a few minutes. Upon reconstruction, the system enables high-quality, mesh-free simulations via the implicit Euler solver, supporting accurate dynamic behavior modeling. We conduct extensive experiments to evaluate our method where Vid2Sim demonstrates remarkable accuracy and efficiency in recovering geometry, appearance, and physical properties from videos compared to existing methods. In summary, our contributions are three-fold: • We propose Vid2Sim, a novel framework for generalizable, video-based reconstruction of appearance, geometry, and physical properties for mesh-free, reduced-order simulation. • We introduce a generalizable feed-forward model with physical world knowledge to estimate the dynamics, followed by an efficient optimization step with Neural Jacobian to improve the reconstruction results further. • Vid2Sim demonstrates remarkable effectiveness and efficiency, achieving state-of-the-art performance in accuracy and speed compared to existing methods. 2 Related Work 2.1 Physics-aware Dynamic 3D reconstruction Dynamic 3D reconstruction is one of the critical tasks in computer vision and graphics. Recent advances in 3D representations like NeRF Mildenhall et al. 2021 and 3D Gaussian Splatting Kerbl et al. 2023 as well as template-based models Loper et al. 2023 ; Romero et al. 2022 ; Li et al. 2017 make it flexible to reconstruct complex 3D scenes from visual data. These methods are recently extended to a dynamic 3D reconstruction Pumarola et al. 2021 ; Yang et al. 2023 ; Wu et al. 2024a from either monocular videos Xian et al. 2021 ; Gao et al. 2021 ; Tretschk et al. 2021 ; Qiao et al. 2022 ; Yang et al. 2024 ; Wu et al. 2024b ; Shimada et al. 2023 or multi-view videos Park et al. 2021a ; Park et al. 2021b ; Luiten et al. 2023 . With the introduction of physics-informed learning Cuomo et al. 2022 ; Chiu et al. 2022 , approaches that incorporate physical priors to enhance the understanding and reconstruction of dynamic scenes have gained popularity. For instance, PAC-NeRF Li et al. 2023 first jointly reconstructed the dynamic scene and a simulatable model using the differentiable Material Point Method Hu et al. 2018 ; Hu et al. 2019 , and it was subsequently improved regarding the quality Cai et al. 2024 ; Kaneko 2024 and adaptability Zhong et al. 2025 . While these methods achieve physically complete reconstruction, none of them are generalizable. In contrast to all existing methods, we first propose a generalizable pipeline that achieves simulation-ready geometry and physical property recovery in a feed-forward manner, which is inspired by the recent achievements in large 3D reconstruction model Hong et al. 2023 ; Tang et al. 2025 ; Zhang et al. 2025 and 4D reconstruction model Ren et al. 2024 . A highly efficient optimization step is conducted to further enhance the reconstruction quality. 2.2 Vision-based Physical Simulation Mesh-free Physical Simulation Traditional physical elasticity simulation, such as the finite element method (FEM) Cutler et al. 2002 , often requires a mesh or tetrahedral representation. This complicates the