Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning
Amine Elhafsi, Daniel Morton, Marco Pavone · 2025
Read Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning by Amine Elhafsi, Daniel Morton, Marco Pavone (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14938.txt.
Introduction
Amine Elhafsi, Daniel Morton, and Marco Pavone published a paper in 2025 on giving robots a way to predict what their actions will physically do. Robots in messy real settings fail when they cannot foresee a push, a slip, or a bad landing. Their framework, Scan Materialize Simulate, chains four modern tools into one planning loop.
First, a rendering method rebuilds the scene in three dimensions from images. Vision models then label the objects, a language-vision model guesses what each object is made of, and a physics simulator plays candidate actions forward before the robot commits. Tests covered a billiards-style manipulation task and a drone landing in a tricky spot. Performance held up across simulated shifts and in real-world trials, suggesting the loop generalizes.
Catalog record
| CREATOR | Amine Elhafsi, Daniel Morton, Marco Pavone |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14938
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning Amine Elhafsi, Daniel Morton, Marco Pavone Abstract Autonomous robots must reason about the physical consequences of their actions to operate effectively in unstructured, real-world environments. We present Scan, Materialize, Simulate (SMS), a unified framework that combines 3D Gaussian Splatting for accurate scene reconstruction, visual foundation models for semantic segmentation, vision-language models for material property inference, and physics simulation for reliable prediction of action outcomes. By integrating these components, SMS enables generalizable physical reasoning and object-centric planning without the need to re-learn foundational physical dynamics. We empirically validate SMS in a billiards-inspired manipulation task and a challenging quadrotor landing scenario, demonstrating robust performance on both simulated domain transfer and real-world experiments. Our results highlight the potential of bridging differentiable rendering for scene reconstruction, foundation models for semantic understanding, and physics-based simulation to achieve physically grounded robot planning across diverse settings. Keywords: 3D Gaussian Splatting, Scene Segmentation, Physics Simulation, Model-Based Planning, Sim-to-Real Figure 1: Overview of the Scan, Materialize, Simulate (SMS) framework for physics-informed robot action planning. SMS consists of three steps: scanning to build a geometric environment model, materializing to convert this reconstruction into a simulation-ready representation, and simulating to optimize actions in a virtual environment prior to target-environment execution. 1 Introduction In recent years, autonomous robotic systems have demonstrated remarkable capabilities, paving the way for deployment across a wide range of real-world applications. However, tasks that require a robot to reason about the physical consequences of its actions remain particularly challenging. Many existing systems tackle these challenges only in highly structured environments, under rigid assumptions, or with highly specialized algorithms; for example, custom solutions have been developed to enable robots to play physically challenging games such as Jenga [ 1 ] , billiards [ 2 , 3 ] and table tennis [ 4 ] . Beyond such recreational settings, diverse domains motivate the need for robust reasoning about physical interactions in unstructured environments, whether a robot is negotiating unstable rubble in disaster response, stacking irregular loads in warehouses, or handling equipment in microgravity space environments. In each of these scenarios, the robot’s success hinges not just on accurate control, but on anticipating the dynamics of the interactions between itself and the world. Developing a methodology for physics-informed planning requires the careful integration of several capabilities. The robot must perceive and reconstruct its environment with sufficient geometric detail to support nuanced interactions. Yet, a bulk geometric reconstruction alone is not enough; an understanding of the scene’s semantics is also required to segment individual entities and assign material characteristics (e.g., density, friction), which directly affect how each object reacts to contact and/or manipulation. Moreover, the system should incorporate a model to accurately predict the outcomes of both robot-object and object-object interactions, enabling the robot to plan actions that reliably anticipate the environment’s response. Motivated by these challenges and building upon recent breakthroughs in differentiable rendering, foundation models, and physics simulation, we propose Scan, Materialize, Simulate (SMS), a framework for physics-informed robot action planning, comprised of the three eponymous stages: Scan: The robot observes the scene from multiple viewpoints to acquire a set of color-and-depth (RGBD) images used to construct a detailed geometric representation using 3D Gaussian Splatting (3DGS) [ 5 ] . Visual foundation models (VFMs) perform object detection [ 6 ] and segmentation [ 7 ] on the RGBD images, allowing volumetric primitives to be associated with their corresponding objects. Materialize: These volumetric primitives are converted into physics-ready meshes, with estimated material attributes such (e.g., density, friction) inferred by a visual language model (VLM). Simulate: The reconstructed meshes are instantiated in a physics simulation wherein candidate robot actions are optimized before being enacted in the target environment. Collectively, SMS blends the generalist, semantic understanding provided by learning-based foundation models with the principled consistency of physical simulation, enabling generalizable, semantically-aware physics-informed robot planning. We illustrate this framework in Figure 1 . 2 Related Work Learning-Based Robot Policies: Reinforcement learning (RL) [ 8 , 9 , 10 ] and imitation learning (IL) [ 11 , 12 , 13 , 14 ] have reigned as the dominant paradigms for developing robot policies in recent years. Although physical priors are generally not explicitly enforced, these policies must implicitly encode knowledge of the physical phenomena pertaining to the task at hand (e.g., gravity, gripper friction). This knowledge is provided implicitly via the training environments [ 15 , 16 ] in the case of RL, or the training demonstrations for IL. However, these learned policies are well known to suffer significant performance degradation when confronted with out-of-distribution (OOD) data [ 17 , 18 ] , and we cannot expect their implicit physical knowledge to generalize to the infinitude of possible world configurations a robot may encounter. High-capacity vision-language-action (VLA) models [ 19 , 20 , 21 , 22 , 23 , 24 ] for robotics have also begun to emerge alongside the broader wave of foundation models [ 25 ] . Trained on massive, internet-scale, datasets, these models demonstrate strong generalization, semantic reasoning, and zero-shot performance on a variety of manipulation and navigation tasks. However, their primary strength lies in their semantic and contextual understanding of diverse tasks and scenes, while their demonstrated applications rarely extend to the physically dynamic tasks we consider in this work. Differentiable Rendering in Robotics: The advent of modern differentiable rendering methods [ 26 , 27 , 28 , 5 ] has provided powerful new tools for scene reconstruction in robotics. Methods built around these representations have quickly made their way into a range of robotics applications, serving as highly detailed obstacle maps for motion planning [ 29 , 30 , 31 ] , providing new means for object tracking [ 32 , 33 ] , performing system identification and visuomotor control [ 34 , 35 ] , and underpinning visually accurate virtual environments for policy training in simulation [ 36 , 37 , 38 , 39 , 40 , 41 ] and synthetic data generation [ 42 ] . Semantic features have also been integrated into these reconstructions [ 43 , 44 , 45 ] , enabling robots to interpret local scene semantics and manipulate objects according to natural language task specifications [ 46 , 47 ] . Of particular relevance to this work are the methods which integrate physics into the reconstructed environment geometry, such as [ 48 , 49 , 50 ] , which deform the scene geometry according to the material point method (MPM) [ 51 ] . However, these methods are primarily oriented towards graphics applications, with the main objective being to produce plausible and visually appealing animated scenery. Some of the aforementioned works such as [ 32 , 40 , 33 ] have explored physically simulating the motion of reconstructed scene components under predicted forces or interactions, but their focus has largely been on manipulation scenarios, often targeting specific object interactions. In contrast, our work aims to enable robots to plan and reason about physically grounded interactions, bridging the gap between physical scene reconstruction and robotic action planning. Statement of Contributions: In this work, we present SMS, a novel framework for physics-informed robot action planning. SMS integrates 3D Gaussian Splatting (3DGS) for accurate reconstruction of