HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human–Scene Interactions
Yukang Cao 1, Haozhe Xie 1, Fangzhou Hong 1, Long Zhuo 3, Zhaoxi Chen 1, Liang Pan 2, Ziwei Liu 1, ✉ · 2026
Read HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human–Scene Interactions by Yukang Cao 1, Haozhe Xie 1, Fangzhou Hong 1, Long Zhuo 3, Zhaoxi Chen 1, Liang Pan 2, Ziwei Liu 1, ✉ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15612.txt.
Introduction
Cao and co-authors present HSImul3R, a method that rebuilds human-scene contact in 3D for physics simulators. Pretty reconstructions often break physical rules and fail inside physics engines. Their pipeline lets the simulator guide fixes in both directions. Reinforcement learning tunes human motion for fidelity and contact stability. Simulation rewards then refine scene geometry for gravity and task success. The team also built HSIBench, a test set with varied objects and scenes. Their reconstructions run on real humanoid robots.
Catalog record
| CREATOR | Yukang Cao 1, Haozhe Xie 1, Fangzhou Hong 1, Long Zhuo 3, Zhaoxi Chen 1, Liang Pan 2, Ziwei Liu 1, ✉ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35069 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15612
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human–Scene Interactions Yukang Cao 1, Haozhe Xie 1, Fangzhou Hong 1, Long Zhuo 3, Zhaoxi Chen 1, Liang Pan 2, Ziwei Liu 1, ✉ Abstract We present HSImul3R 1 1 1 Pronunciation: / ”sImjul@(r) / , a unified framework for simulation-ready 3D reconstruction of human-scene interactions (HSI) from casual captures, including sparse-view images and monocular videos. Existing methods suffer from a perception-simulation gap : visually plausible reconstructions often violate physical constraints, leading to instability in physics engines and failure in embodied AI applications. To bridge this gap, we introduce a physically-grounded bi-directional optimization pipeline that treats the physics simulator as an active supervisor to jointly refine human dynamics and scene geometry. In the forward direction, we employ Scene-targeted Reinforcement Learning to optimize human motion under dual supervision of motion fidelity and contact stability. In the reverse direction, we propose Direct Simulation Reward Optimization , which leverages simulation feedback on gravitational stability and interaction success to refine scene geometry. We further present HSIBench , a new benchmark with diverse objects and interaction scenarios. Extensive experiments demonstrate that HSImul3R produces the first stable, simulation-ready HSI reconstructions and can be directly deployed to real-world humanoid robots. Figure 1 : Examples of real-world humanoid robotic deployment. Given casual captures, our approach achieves simulation-ready 3D reconstruction of human–scene interactions by refining the human motions and scene geometry via a physically-grounded bi-directional optimization pipeline. Our optimized human motions can be seamlessly transferred and deployed in humanoid robotics. 1 Introduction Embodied artificial intelligence aims to integrate intelligent agents into daily life through physically grounded systems. Unlike disembodied models [ 83 , 10 , 11 , 8 ] limited to virtual domains, embodied AI [ 94 , 79 , 92 , 91 ] learns transferable motions that enable perception, reasoning, and action in real-world environments. A key challenge is modeling humanoid–scene interactions, requiring understanding of human motion, spatial layouts, and interaction stability. Reconstructing human–scene interactions (HSI) [ 4 , 82 , 41 ] from images or videos provides high-fidelity supervision and enables scalable, simulation-ready datasets, helping bridge passive observation and active robotic deployment. Current methods suffer from a perception–simulation gap , where visually plausible reconstructions violate physical constraints and fail in embodied AI applications. This gap largely stems from the fragmented modeling of humans and environments, as existing approaches rarely capture their explicit physical coupling and instead fall into three separate directions: 1) 3D scene reconstruction ( e.g , NeRF [ 51 ] , Gaussian Splatting [ 28 ] , DUSt3R [ 72 ] ), which prioritizes environment geometry while largely ignoring human dynamics. 2) Human motion estimation [ 9 , 70 , 57 , 30 ] , which achieves robustness under occlusion but reconstructs motion in isolation, without modeling physical contact or environmental constraints. 3) Interaction modeling [ 89 , 25 , 13 , 15 , 55 ] , typically based on SMPL-driven HSI datasets [ 5 , 24 , 40 ] that remain limited in scale, diversity, and physical validation. Recent unified frameworks ( e.g , HOSNeRF [ 36 ] , HSfM [ 52 ] ) attempt joint modeling but optimize mainly in the 2D image space, prioritizing visual alignment over geometric and physical validity. Consequently, the resulting reconstructions lack metric and contact fidelity, making them unsuitable for simulation and preserving the gap between visual realism and embodied deployment. To close this gap, we introduce HSImul3R, a simulation-ready Human–Scene Interaction 3D reconstruction framework that formulates reconstruction as a bi-directional physics-aware optimization problem. A physics simulator acts as an active supervisor, enabling closed-loop refinement between human motion and scene geometry. HSImul3R operates along two complementary directions. Forward optimization refines human motion under fixed scene geometry. After establishing metric-consistent human–scene alignment with structural priors from image-to-3D generative models, we integrate the reconstruction into the simulator and perform scene-targeted reinforcement learning. Motion is optimized using physically grounded signals, including keypoint tracking consistency and geometric contact constraints, improving interaction stability. Reverse optimization refines scene geometry under physically validated motion. To address instability caused by structurally deficient geometry, we introduce Direct Simulation Reward Optimization (DSRO), which leverages simulator-derived rewards to enhance gravitational stability and interaction feasibility. Figure 2 : Examples of HSIBench and corresponding simulation results by HSImul3R. Our approach enables simulation-ready 3D reconstruction of human–scene interactions from casual captures. In addition, we collect HSIBench, a dataset comprising 16-view synchronized captures of diverse human–scene interactions, covering a wide range of scene objects, human subjects, and motions. To support the training and benchmarking of this framework, we collect HSIBench , a new dataset comprising 19 objects and over 50 motion sequences recorded by two male and one female participants, totaling 300 unique interaction instances. An overview of HSIBench and simulation results of our method is provided in Fig. 2 . We conduct extensive experiments to evaluate HSImul3R against state-of-the-art baselines in terms of simulation stability, post-simulation human motion quality, and improvements in image-to-3D generation through DSRO fine-tuning. Experimental results demonstrate that HSImul3R is the first approach to achieve stable, simulation-ready reconstructions of human–scene interactions, offering robust performance across diverse scenarios and significantly outperforming existing techniques. Finally, we demonstrate the real-world utility of our framework by (1) retargeting the refined motions to a Unitree humanoid robot, and (2) training a whole-body motion tracking policy for physical deployment. Examples of real-world deployment are presented in Fig. 1 . 2 Related Works 3D Scene Reconstruction Early approaches are dominated by geometry-based methods, such as structure-from-motion [ 65 ] and multi-view stereo [ 66 ] , which estimate camera poses and dense geometry from multiple views. With the rise of deep learning, data-driven approaches emerge, including monocular depth prediction [ 87 , 88 ] and learning-based multi-view stereo [ 22 ] , enabling reconstruction from sparse or unstructured imagery. Other works adopt explicit 3D representations such as voxels [ 68 , 37 ] , point clouds [ 14 , 80 ] , and meshes [ 54 ] , often optimized through differentiable rendering. More recently, implicit neural representations, such as signed distance functions [ 56 ] , occupancy fields [ 6 ] , neural radiance fields [ 33 , 76 ] , and explicit but differentiable formulations like 3D Gaussian Splatting [ 28 , 77 ] , become central to high-quality scene modeling. Beyond static reconstruction, dynamic scene modeling [ 86 , 78 ] expands these methods to time-varying environments. In parallel, recent works such as Dust3R [ 72 ] and VGGT [ 71 ] introduce pre-trained transformers that enable end-to-end 3D reconstruction directly from uncalibrated and unlocalized images, eliminating the need for expensive post-optimization. Physically-sounded Modeling Recent works have sought to embed physical soundness into modeling, which can be broadly categorized into three paradigms. Physics-constrained and physics-integrated generation methods unify simulation and content creation by leveraging simulation-derived losses or physical priors. For example, PhyRecon [ 53 ] ensures stable scene reconstruction, Atlas3D [ 12 ] and BrickGPT [ 61 ] produce self-supporting structures, and DSO [ 31 ] or PhysDeepSDF [ 50 ] align generators with simulation feedback. PhysGaussian [ 81 ] evolves Gaussian splats via continuum mechanics, while PhyCAGE [ 85 ] , VR-GS [ 26 ] , and GASP [ 7 ] optimize assets through MPM; PAC/iPAC-NeRF [