Gaussian-Augmented Physics Simulation and System Identification with Complex Colliders

Federico Vasile, Ri-Zhao Qiu, Lorenzo Natale, Xiaolong Wang · 2025

Read Gaussian-Augmented Physics Simulation and System Identification with Complex Colliders by Federico Vasile, Ri-Zhao Qiu, Lorenzo Natale, Xiaolong Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06846.txt.

Introduction

Federico Vasile, Ri-Zhao Qiu, Lorenzo Natale, and Xiaolong Wang estimate the physical features of objects from video, a task called system identification. Recent methods optimize geometry, appearance, and physics together using differentiable simulation, but they assume flat surfaces for collisions. Real objects collide with curved, complex shapes, where those methods fail.

Their system, AS-DiffMPM, extends the differentiable Material Point Method with a collision mechanism that handles arbitrarily shaped rigid bodies while keeping the whole pipeline differentiable. A falling or rolling object can thus be fitted against the true geometry of what it hits. The system plugs into various novel view synthesis methods, widening the range of scenes where substance features, shape, and appearance can be recovered from plain video observations.

Catalog record

CREATORFederico Vasile, Ri-Zhao Qiu, Lorenzo Natale, Xiaolong Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06846

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gaussian-Augmented Physics Simulation and System Identification with Complex Colliders Federico Vasile, Ri-Zhao Qiu, Lorenzo Natale, Xiaolong Wang Abstract System identification involving the geometry, appearance, and physical properties from video observations is a challenging task with applications in robotics and graphics. Recent approaches have relied on fully differentiable Material Point Method (MPM) and rendering for simultaneous optimization of these properties. However, they are limited to simplified object-environment interactions with planar colliders and fail in more challenging scenarios where objects collide with non-planar surfaces. We propose AS-DiffMPM, a differentiable MPM framework that enables physical property estimation with arbitrarily shaped colliders. Our approach extends existing methods by incorporating a differentiable collision handling mechanism, allowing the target object to interact with complex rigid bodies while maintaining end-to-end optimization. We show AS-DiffMPM can be easily interfaced with various novel view synthesis methods as a framework for system identification from visual observations. Figure 1: We present the Any-Shape Differentiable Material Point Method ( AS-DiffMPM ), a particle-based framework for simulating collisions with arbitrarily shaped rigid bodies. When integrated with Gaussian-based rendering, it enables physically plausible animation and system identification (estimation of object physical parameters from visual observations). 1 Introduction Bending but not breaking , a branch dances with the wind—yielding to its force yet never surrendering. From the gentle drift of a balloon to the violent shatter of glass, the world reveals its hidden truths through motion. We, too, learn not only by seeing but by understanding —tracing back from movement to the forces that shaped it. In this work, we give machines this same power: to see, reason, and uncover the physical essence of objects through their dance with the world, recovering their hidden properties from mere pixels. Estimating the physical parameters of objects from visual observations is a challenging task in computer vision, with applications in robotics, virtual reality, and graphics. While existing methods such as PhysGaussian 35 can animate static scenes by simulating plausible dynamics, they do not support the identification of object properties from dynamic scenes. Humans naturally infer an object’s geometry and dynamics through a two-step process. First, they recognize and understand its shape (e.g., a ball). Then, by observing its interactions with the environment (e.g., let the ball fall on the ground), they deduce its physical properties. Inspired by this intuition, modern system identification methods follow a similar paradigm, combining two components: a rendering model to reconstruct geometry from images and a physics engine to simulate motion, enabling the estimation of physical properties from video sequences. Recent advances in scene representations — particularly point-based models 17 —and particle-based physics simulation 33 , have facilitated the animation of static geometry with physics-grounded motion. This process involves establishing correspondences between the points used for rendering and the particles in simulation, allowing the radiance field to be animated by applying the motion derived from simulated particle trajectories. As a result, for physical properties estimation, this approach can be applied by reconstructing the object’s geometry, initializing an estimate of its physical parameters, and refining them through gradient-based optimization, comparing simulated motion against real-world observations. PAC-NeRF 19 is a pioneering model for system identification from visual observations; combining voxel-based radiance fields 34 with a Differentiable Material Point Method (DiffMPM) 9 to enable gradient-based optimization of physical parameters from images. However, this approach, along with other recent works 16 ; 3 ; 38 , share a significant limitation: they perform system identification only when the object interacts with simple boundaries such as the ground. While effective, this setup fails to capture more complex scenarios, where objects interact with arbitrarily shaped rigid bodies. In this work, we extend system identification beyond simple boundary interactions, enabling estimation of physical properties in more complex settings. To the best of our knowledge, this problem has not been previously addressed, primarily due to two key limitations: (i) existing system identification methods are restricted to interactions with a planar boundary 19 ; 16 ; 3 ; 38 , and (ii) available MPM-based simulators that accurately handle arbitrarily shaped colliders are not differentiable 10 , thus not suitable for physical parameters optimization. Our contribution is threefold: 1. Any-Shape Differentiable MPM. We propose AS-DiffMPM, a Differentiable MPM capable of handling collisions between the target object and arbitrarily shaped colliders. Quantitative experiments on Newtonian , non-Newtonian and granular materials demonstrate the necessity of an ad hoc solution for rigid body interactions, which previous system identification methods have overlooked. 2. Versatile Collision Handling Framework. We design a general interface to integrate colliders of various formats (i.e., mesh and 2D Gaussians 11 ) into AS-DiffMPM. 3. Rendering-Physics Integration for System Identification. We combine our AS-DiffMPM with multiple rendering models and establish a benchmark for system identification from visual observations when the object collides with arbitrarily shaped rigid bodies. 2 Related Work Figure 2: Overview. Given multi-view input images, we separately reconstruct the continuum object using a rendering model (e.g., 34 ; 11 ) and the rigid body collider with 2DGS 11 . Particle trajectories are subsequently advected using AS-DiffMPM. Finally, the updated particle positions are mapped back to rendering primitives for photorealistic and physically plausible animation. Scene Representations for Photorealistic Rendering. The topic of scene reconstruction and synthesizing photorealistic novel views has been studied in the computer vision community for decades 1 ; 8 ; 30 . Recently, there has been a growing interest in using learning-based differentiable rendering or rasterization for such a purpose 17 ; 25 ; 27 ; 7 ; 37 . These methods are often categorized into implicit rendering methods based on ray-marching queries 25 ; 27 ; 7 ; 37 or explicit rasterization methods 17 ; 11 . To manipulate and animate these static scene representations, researchers have shown that it is possible to edit both implicit methods (often via remapping voxel-based representations 19 ; 13 ) and explicit methods (via direct manipulation of explicit units 3 ; 35 ; 28 ). Our method is mostly orthogonal to the development in scene representations. The AS-DiffMPM is a plug-and-play module to both NeRF- and 3DGS-based methods, as quantitatively shown in the experiment. Physics-integrated Scene Synthesis. While high-quality reconstruction methods offer a pathway to synthesize static scenes from a set of images in novel viewpoints, they are limited to understanding static appearances. Humans, on the other hand, have the ability to picture physically plausible interactions with the scenes. To equip machines with such an intelligence, researchers have sought to animate static scenes with physics simulators to generate animated scenes that are both visually and physically appealing 35 ; 5 ; 28 ; 15 ; 19 ; 6 ; 26 . Due to the indirection caused by implicitness, few methods animate implicit NeRFs 6 . Many researchers have combined Material Point Method 33 with Gaussian Splatting 17 due to their naturally aligned point-based representations. These methods have shown to provide photorealistic animations 35 ; 2 with extensions to VR applications 15 , language-driven animations 28 , and complex fluid dynamics 5 . However, these methods provide physically plausible animations of static scenes rather than estimating physical parameters of objects. Another line of work leverages object dynamics priors from video generative models to endow static 3D objects with interactive behaviors 36 ; 21 ; 12 ;

Read the complete text