Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions

Kaifeng Zhang, Shuo Sha, Hanxiao Jiang, Matthew Loper, Hyunjong Song, Guangyan Cai, Zhuo Xu, Xiaochen Hu, Changxi Zheng, Yunzhu Li · 2025

Read Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions by Kaifeng Zhang, Shuo Sha, Hanxiao Jiang, Matthew Loper, Hyunjong Song, Guangyan Cai, Zhuo Xu, Xiaochen Hu, Changxi Zheng, Yunzhu Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04665.txt.

Introduction

Kaifeng Zhang leads ten authors evaluating robot policies in simulations built from real videos. Testing manipulation policies in the real world is slow, costly, and hard to repeat. Soft objects make it worse, since their look and physics couple tightly.

Their real-to-sim framework builds digital twins of deformable objects from recorded footage. Three-dimensional Gaussian Splatting renders robots, objects, and scenes with photo-like fidelity. Test tasks include packing a plush toy, routing a rope, and pushing a T-block. Simulated rollouts correlate strongly with real outcomes, suggesting cheap, repeatable screening of policies before hardware trials.

Catalog record

CREATORKaifeng Zhang, Shuo Sha, Hanxiao Jiang, Matthew Loper, Hyunjong Song, Guangyan Cai, Zhuo Xu, Xiaochen Hu, Changxi Zheng, Yunzhu Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04665

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions Kaifeng Zhang, Shuo Sha, Hanxiao Jiang, Matthew Loper, Hyunjong Song, Guangyan Cai, Zhuo Xu, Xiaochen Hu, Changxi Zheng, Yunzhu Li Abstract Robotic manipulation policies are advancing rapidly, but their direct evaluation in the real world remains costly, time-consuming, and difficult to reproduce, particularly for tasks involving deformable objects. Simulation provides a scalable and systematic alternative, yet existing simulators often fail to capture the coupled visual and physical complexity of soft-body interactions. We present a real-to-sim policy evaluation framework that constructs soft-body digital twins from real-world videos and renders robots, objects, and environments with photorealistic fidelity using 3D Gaussian Splatting. We validate our approach on representative deformable manipulation tasks, including plush toy packing, rope routing, and T-block pushing, demonstrating that simulated rollouts correlate strongly with real-world execution performance and reveal key behavioral patterns of learned policies. Our results suggest that combining physics-informed reconstruction with high-quality rendering enables reproducible, scalable, and accurate evaluation of robotic manipulation policies. Website: https://real2sim-eval.github.io/ I Introduction Robotic manipulation policies have advanced rapidly across a wide range of tasks [ 17 , 82 , 18 , 41 , 61 ] . However, their evaluation still relies heavily on real-world trials, which are slow, expensive, and difficult to reproduce. As the community shifts toward training foundation models for robotics [ 11 , 35 , 9 , 27 , 47 , 64 , 63 ] , whose development depends on rapid iteration and large-scale benchmarking, this reliance has become a significant bottleneck. Simulation offers a scalable and systematic alternative and is widely used for data generation and training [ 48 , 67 , 72 , 37 , 5 , 65 ] . Yet it is far less common as a tool for policy evaluation, primarily due to poor sim-to-real correlation: a policy that performs well in simulation often fails to translate to similar real-world success. Narrowing this gap would allow simulation to serve as a trustworthy proxy for real-world testing, greatly accelerating development cycles. This raises the central question: how can we design simulators that are sufficiently realistic to evaluate robot policies with confidence? To answer this question, we propose a framework for building high-fidelity simulators and investigate whether they can predict real-world policy performance reliably. We identify two key factors for aligning simulation with reality: appearance and dynamics . On the appearance side, rendered scenes must closely match real-world observations. This is particularly challenging for policies that rely on wrist-mounted cameras, where simple green-screen compositing [ 40 ] is insufficient. We address this by leveraging 3D Gaussian Splatting (3DGS) [ 33 ] , which reconstructs photorealistic scenes from a single scan and supports rendering from arbitrary viewpoints. Beyond prior uses of 3DGS for simulation [ 2 , 57 , 38 , 7 ] , we enhance it with automatic position and color alignment and object deformation handling, which are essential for closing the appearance gap. Dynamics present another major source of sim-to-real discrepancy. Traditional simulators rely on low-dimensional parameter tuning, which is insufficient for deformable objects with many degrees of freedom. To address this challenge, we adopt PhysTwin [ 31 ] , a framework that reconstructs deformable objects as dense spring-mass systems optimized directly from object interaction videos. This approach yields efficient system identification while closely matching real-world dynamics. We integrate these appearance and dynamics components into a unified simulator and expose it through a Gym-style interface [ 10 ] . We evaluate this framework on representative rigid- and soft-body manipulation tasks, including plush toy packing, rope routing, and T-block pushing, using widely adopted imitation learning algorithms: ACT [ 82 ] , Diffusion Policy (DP) [ 17 ] , SmolVLA [ 58 ] , and Pi-0 [ 9 ] . By comparing simulated and real-world success rates and performing ablation studies, we observe a strong correlation and confirm that rendering and dynamics fidelity are both crucial to the trustworthiness of simulation-based evaluation. In summary, our main contributions are: (1) A complete framework for evaluating robot policies in a Gaussian Splatting-based simulator using soft-body digital twins. (2) Empirical evidence that simulated rollouts strongly correlate with real-world success rates across representative tasks, using policies trained exclusively on real-world data (no co-training). (3) A detailed analysis of design choices that improve the reliability of simulation as a predictor of real-world performance, offering guidance for future simulation-based evaluation pipelines. II Related Works II-A Robot Policy Evaluation Evaluating robot policies is essential for understanding and comparing policy behaviors. Most systems are still evaluated directly in the real world [ 64 , 49 , 69 , 51 , 34 , 85 , 4 ] , but such evaluations are costly, time-consuming, and usually tailored to specific tasks, embodiments, and sensor setups. To enable more systematic study, prior works have introduced benchmarks, either in the real world through standardized hardware setups [ 13 , 68 , 19 , 84 , 20 ] or in simulation through curated assets and task suites [ 62 , 28 , 37 , 59 , 55 , 45 , 86 , 44 , 76 , 70 , 6 ] . Real-world benchmarks offer high fidelity but lack flexibility and scalability, while simulators often suffer from unrealistic dynamics and rendering, which limits their reliability as proxies for physical experiments. This is widely referred to as the “sim-to-real gap” [ 52 , 14 , 50 , 26 ] . We aim to narrow this gap by building a realistic simulator that combines high-quality rendering with faithful soft-body dynamics. Compared to SIMPLER [ 40 ] and RobotArena \infty [ 29 ] which relies on green-screen compositing, Ctrl-World [ 24 ] which relies on 2D video world models, and Real-is-sim [ 2 ] which focuses on rigid-body simulation, our method integrates Gaussian Splatting-based rendering with soft-body digital twins derived from interaction videos, eliminating the dependence on static cameras and providing more realistic appearance and dynamics. II-B Physical Digital Twins Digital twins seek to realistically reconstruct and simulate real-world objects. Many existing frameworks rely on pre-specified physical parameters [ 42 , 15 , 32 , 73 , 56 ] , which limits their ability to capture complex real-world dynamics or leverage data from human interaction. While rigid-body twins are well studied [ 8 , 75 , 1 , 78 ] , full-order parameter identification for deformable objects remains challenging. Learning-based approaches have been proposed to capture such dynamics [ 53 , 79 , 80 , 66 ] , but they often sacrifice physical consistency, which is critical for evaluating manipulation policies in contact-rich settings. Physics-based methods that optimize physical parameters from video observations [ 39 , 81 , 83 , 16 ] offer a more promising path. Among them, PhysTwin [ 31 ] reconstructs deformable objects as dense spring-mass systems directly from human-object interaction videos, achieving state-of-the-art realism and efficiency. Our work builds on PhysTwin and integrates its reconstructions with a Gaussian Splatting simulator to bridge the dynamics gap in policy evaluation. II-C Gaussian Splatting Simulators Building simulators that closely match the real world requires high-quality rendering and accurate physics. Gaussian Splatting (3DGS) [ 33 ] has recently emerged as a powerful approach for scene reconstruction, enabling photorealistic, real-time rendering from arbitrary viewpoints [ 32

Read the complete text