MatPhys: Learning Material-Aware Physics Parameters for Deformable Object Simulation from Videos
Yang Yang, Yiyan Wang, Zheming Liu, Naoya Iwamoto · 2026
Read MatPhys: Learning Material-Aware Physics Parameters for Deformable Object Simulation from Videos by Yang Yang, Yiyan Wang, Zheming Liu, Naoya Iwamoto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19386.txt.
Introduction
Yang and colleagues present MatPhys, a system that predicts spring-mass physics settings for bendable objects from a single video. Earlier methods assume one uniform material and refit each scene, giving clashing values for the same material. MatPhys splits objects into parts using image features, assigns each part its own material behavior, and ties values together through a shared material codebook. The result matches per-scene fitting on reconstruction and forecasting while generalizing better to new objects and interactions.
Catalog record
| CREATOR | Yang Yang, Yiyan Wang, Zheming Liu, Naoya Iwamoto |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34805 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19386
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MatPhys: Learning Material-Aware Physics Parameters for Deformable Object Simulation from Videos Yang Yang, Yiyan Wang, Zheming Liu, Naoya Iwamoto Abstract Reconstructing simulation-ready deformable objects is important for vision, graphics, and robotics. Existing physics-driven methods can recover physical digital twins from videos, but they suffer from two fundamental limitations: they typically assume a homogeneous material across the whole object, and their scene-specific inverse optimization, combined with the inherent ambiguity of monocular observation, yields inconsistent parameters for the same material across different scenes or interactions. We propose MatPhys, a material-aware feed-forward framework that predicts spring-mass parameters from a single-view video, addressing these two issues with two coupled designs. To relax the homogeneous material assumption, we use DINO features to decompose the object into semantically meaningful parts and to query a part-level material prior, assigning each part its own physical behavior. To enforce cross-scene consistency, we introduce a learned material codebook of shared material embeddings as the bridge between appearance and physics, and further use the part-level prior as a reference distribution that constrains the decoder so that the same material yields consistent parameters across scenes and interactions. Together, these designs turn an under-constrained monocular problem into feed-forward inference grounded on shared, reusable material concepts. Experiments show that our method matches per-scene optimization baselines in reconstruction and future prediction, while achieving stronger generalization to unseen interactions and objects with more consistent physical parameters. Figure 1 : Given a single-view video of a deformable object under interaction, MatPhysreconstructs a fully simulatable digital twin with complete geometry, high-fidelity appearance, and accurate physical parameters. Once reconstructed, the digital twin supports intuitive interactive editing: users can freely place control points on the object and manipulate them through drag-based interaction. 1 Introduction Creating interactive physical digital twins from visual observations is an important problem in computer vision, graphics, and robotics. A 3D asset for interaction and simulation should not only have high-quality geometry and appearance, but also physical properties that allow it to respond to external forces and manipulation. Such assets can support future prediction, interactive editing, and large-scale simulation data generation. However, recovering simulation-ready 3D assets from limited visual input remains challenging. Recent progress in NeRF [ 13 , 4 , 17 , 31 ] , 3D Gaussian Splatting [ 8 ] , and their dynamic extensions [ 33 , 12 , 24 , 15 , 16 , 22 , 29 , 6 ] has greatly improved dynamic scene reconstruction. These methods can recover high-quality geometry, appearance, and observed motion from images or videos. However, they usually model motion as visual deformation or temporal change, without explicitly recovering the physical parameters behind the object. To bridge rendering and simulation, recent works introduce physical models into 3D representations. Some learning-based methods [ 26 , 32 , 24 , 6 ] use neural networks to model object dynamics directly. They have strong fitting ability, but often require large amounts of training data and are limited to specific objects and actions. Other physics-driven methods couple 3D representations with explicit simulators and estimate physical parameters through differentiable inverse optimization. MPM-based methods [ 19 , 11 , 3 ] provide expressive continuum dynamics for complex deformation, while spring-mass systems [ 20 , 1 , 34 ] offer a lightweight and interpretable alternative with explicit stiffness, damping, topology, and contact parameters. Recent spring-mass Gaussian methods, including Spring-Gaus [ 34 ] , PhysTwin [ 7 ] , and NeuSpring [ 27 ] , demonstrate simulation-ready reconstruction from videos by combining 3D Gaussians with inverse physical fitting. However, their parameters are still optimized in a scene-specific manner, which can lead to inconsistent estimates for similar materials across scenes and interactions. Moreover, geometry-driven decomposition, such as KNN-based clustering, does not necessarily align physical regions with semantic material parts. This raises a natural question: can deformable-object physics be inferred in a feed-forward manner from monocular RGB video? Recent concurrent works suggest that this direction is promising. ReconPhys [ 23 ] shows that learned predictors can transfer physical estimates across objects, while PhysSplat [ 33 ] shows that MLLMs provide useful visual priors about material and physical properties. However, neither work learns an explicit, transferable material representation for predicting simulator parameters: ReconPhys relies on global video conditioning in synthetic gravity-driven settings, and PhysSplat queries object-level material priors that collapse heterogeneous parts into a single material assumption. These limitations leave open two key challenges: identifying semantic parts that correspond to changes in different physical behavior, and maintaining consistent physical parameters for the same material across scenes and interactions. To address these two issues, we propose MatPhys, a material-aware feed-forward framework that predicts spring-mass simulation parameters from monocular RGB video. Our key idea is to use semantic parts as the unit for material reasoning: instead of assigning one material prior to the whole object, each part receives its own material evidence and can induce different local spring behavior, which relaxes the homogeneous material assumption and provides a structured interface between visual appearance and physical prediction. Concretely, given an input video, we first reconstruct a 3D Gaussian object representation from a keyframe using a feed-forward 3D reconstruction model [ 25 ] , and use the Gaussians as the geometry representation for simulation. We then lift DINO features [ 14 ] onto the object and cluster them to obtain semantically meaningful parts. For each part, we query an MLLM to obtain a material class prior and a distribution of its corresponding physics parameters. We further estimate a part-aware topology and abstract the Gaussians as mass points, yielding a spring-mass system. On top of this representation, we train a feed-forward physics predictor to estimate the spring-mass parameters. We argue that the prediction should be conditioned on three complementary signals: material features, which capture transferable physical tendencies; motion features, which reveal how the object responds under interaction; and local geometric features, which encode the edge scale, orientation, and neighborhood structure that modulate local spring behavior. To obtain transferable material representations, we introduce a learnable material codebook shared by all objects and scenes. Each part-level material class prior is mapped through this codebook into a material embedding, which serves as the material representation. In parallel, a pretrained video encoder [ 21 ] along with a small projector extracts motion features from the input video, and a geometry encoder computes local edge features from the mass-point coordinates and spring connectivity. A physics decoder fuses these features to predict per-edge stiffness, controller-object stiffness, and global damping/contact parameters. During training, we minimize the discrepancy between simulated results and observations while using the part-level physical distribution as a reference for constraining the physical consistency. Together, these designs enable MatPhysto learn transferable material-to-physics relations and predict simulation-ready parameters in a single forward pass. Experiments show that our method matches per-scene optimization baselines in reconstruction and future prediction, while achieving stronger generalization to unseen objects. Our contributions are threefold: • We propose MatPhys, a material-aware feed-forward framework for predicting simulation-ready digital twins from monocular RGB video. • We introduce a learnable material codebook, enabling transferable material-to-physics prediction and more consistent physical parameters across scenes and interactions. • We use semantic part decomposition and part-level MLLM material queries to build part-aware spring-mass models, relaxing the homogeneous material assumption for heterogeneous deformable objects.