EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting
Changjing Liu, Yiming Huang, Long Bai, Beilei Cui, Hongliang Ren · 2026
Read EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting by Changjing Liu, Yiming Huang, Long Bai, Beilei Cui, Hongliang Ren (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16022.txt.
Introduction
Changjing Liu and four colleagues build EndoGSim, a framework for physics-aware models of moving endoscopic scenes. Such scenes matter for robot-aided keyhole surgery. 4D Gaussian splatting shows soft tissue and tools. A property field, first set by a language model and then refined by a differentiable material point method, sets how tissues behave. Tested on open-source and in-house data sets, it beats state-of-the-art methods on how true to life and true in physics its scenes are.
Catalog record
| CREATOR | Changjing Liu, Yiming Huang, Long Bai, Beilei Cui, Hongliang Ren |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19767 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16022
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting Changjing Liu, Yiming Huang, Long Bai, Beilei Cui, Hongliang Ren Abstract In robot-assisted minimally invasive surgery, high-fidelity dynamic endoscopic scene reconstruction and simulation are crucial to enhancing downstream tasks and advancing surgical outcomes. However, existing methods primarily focus on visual reconstruction, lacking physics-based descriptions of the scene required for realistic simulation. We propose a unified framework that achieves physics-aware reconstruction and physical simulation of endoscopic scenes through Multi-modal Large Language Models (MLLMs)-guided Gaussian Splatting. Our approach utilizes 4D Gaussian Splatting (4DGS) integrated with pre-trained segmentation and depth estimation to represent deformable tissues and tools. To achieve automatic inference of physical properties, we introduce an object-wise material field that initializes material parameters via MLLM and refines them through a differentiable Material Point Method (MPM) under joint supervision from rendered images and optical flow. Validated on both open-source and in-house datasets, our framework achieves superior simulation fidelity and physical accuracy compared to state-of-the-art methods, underscoring its potential to advance robot-assisted surgical applications. 1 Introduction Endoscopic procedures are central to minimally invasive surgery with reduced trauma and faster recovery, motivating the integration of robot-assisted systems to achieve superior precision, consistency, and efficiency [ 24 , 8 , 4 ] . In this context, accurate dynamic reconstruction and simulation of the endoscopic scene are essential to enable reliable spatial intelligence, support pre-operative planning, and ensure robust autonomous control [ 16 ] . Recent endoscopic scene reconstruction benefits from Neural Radiance Fields (NeRFs) [ 23 , 28 , 32 ] and 3D Gaussian Splatting (3DGS) [ 17 , 31 , 5 ] , achieving superior reconstruction quality along with real-time rendering performance. Building upon this foundation, 4D Gaussian Splatting (4DGS) [ 29 , 16 , 10 ] extends static representations to the spatiotemporal domain, making it suitable for deformable surgical scenes reconstruction. Table 1: Comparison of methods by Video input, Auto Initialization, Object-wise material field, and Physical simulation. Method PhysGen Physics3D PhysGaussian xx GIC PhysFlow Ours Video Input ✗ ✗ ✗ ✓ ✓ ✓ Auto Initialization ✓ ✗ ✗ ✗ ✓ ✓ Material Field ✗ ✗ ✗ ✗ ✗ ✓ Physical Simulation ✗ ✓ ✓ ✓ ✓ ✓ Despite remarkable advances in scene reconstruction, existing methods [ 17 , 29 ] primarily focus on appearance reconstruction and seldom capture the material properties of objects. This limits physically accurate spatial perception, high-fidelity dynamic simulation, and the generation of realistic data essential for surgical automation. To address this challenge, several studies integrate material property estimation and perform simulations on reconstructed deformable scenes [ 30 , 3 , 18 , 21 ] . PhysGaussian [ 30 ] combines 3DGS with a differentiable Material Point Method (MPM) for physics-based simulation, with material parameters specified through manual initialization. To enable automatic identification of material properties, GIC [ 3 ] captures explicit shapes using 3D Gaussian Splatting and infers physical properties through continuum mechanics for simulation. Subsequent works [ 20 , 21 ] further integrate multi-modal large language models (MLLMs) to enhance physics-aware reconstruction and dynamic simulation. PhysGen [ 20 ] introduces physics reasoning using large pre-trained visual foundation models, eliminating the need for manual parameter initialization. Physflow [ 21 ] leverages multi-modal foundation models and video diffusion to achieve enhanced 4D dynamic scene simulation. However, there remains a notable gap in adapting these techniques for physics-aware reconstruction and simulation of surgical scenes. In particular, 3DGS-based physical simulation methods [ 30 , 18 ] struggle with dynamic video input, and existing material estimation strategies [ 30 , 3 , 18 ] often lack informed initialization, potentially leading to physically inconsistent simulations. Furthermore, existing approaches mainly focus on general scenes and rarely consider multi-object material estimation, which is critical in surgical environments involving both instruments and deformable tissues. To overcome these challenges, we propose a unified framework that reconstructs deformable endoscopic scenes from video and simultaneously estimates material parameters to enable realistic physical simulation, as shown in Fig. 1 . Specifically, we employ 4DGS [ 16 ] with pre-trained depth and segmentation models to construct a Gaussian splat representation of the surgical scene. Then, we propose an object-wise material field to estimate the physical properties of the tissues and tools. Material parameters are automatically initialized using large pre-trained multi-modal large language models, and then refined via gradient-based optimization within a differentiable MPM, guided by rendering and optical flow, to ensure both visual realism and physical fidelity. Table 1 summarizes the comparison of our approach with existing methods. Code and datasets will be released upon publication. The primary contributions of this work are summarized as follows: • We introduce a unified framework for the automatic, physics-aware reconstruction and simulation of surgical scenes from endoscopic videos. • The material field is proposed for material estimation, coarsely initialized via MLLM estimation and then jointly refined with render and optical flow loss in a differentiable MPM, achieving physically plausible simulation. • Experiments on open-source and in-house datasets demonstrate that our method achieves accurate physics and realistic simulations, offering substantial potential to advance robotic-assisted minimally invasive surgery. Figure 1: Overview of our physics-aware framework for surgical scene reconstruction and 4D dynamic simulation with automatic estimation of physical parameters. 2 Methods 2.1 Preliminaries Inspired by [ 16 ] , we reconstruct the surgical scene via 4D Gaussian \mathcal{G}^{\prime}=\mathcal{G}+\Delta\mathcal{G} , including a static 3D Gaussian \mathcal{G} and its deformation \Delta\mathcal{G}=\mathcal{F}(\mathcal{G},t) , where \mathcal{F} denotes the deformation network and t is the time. The spatial-temporal encoder \mathcal{H} is defined to parameterize the 4D trajectory of each Gaussian point. A multi-head Gaussian deformation decoder \mathcal{D} is designed for decoding the deformation of position \mu , rotation R , scaling S , opacity o and spherical harmonics SH using five efficient and lightweight tiny MLPs. The final 4D Gaussian representation can be formulated as follows: \mathcal{G}^{\prime}=\{\mu+\Delta{\mu},\textbf{R}+\Delta{\textbf{R}},\textbf{S}+\Delta{\textbf{S}},o+\Delta{o},\textbf{SH}+\Delta{\textbf{SH}}\}. (1) For dynamic scene video V=\{I_{0},...I_{t}\} , we adopt 4D Gaussian Splatting to construct a time-dependent 3D scene representation. Specifically, the \pi^{3} model [ 27 ] is integrated to provide depth estimation, and Surgical-SAM-2 [ 19 ] is utilized for efficient segmentation of soft tissues and surgical instruments. 2.2 Continuum Mechanics Continuum mechanics provides the foundational description of the motion of soft tissues during surgical procedures. Specifically, the deformation of a continuum is described via the time-dependent continuous map \mathbf{x}={\phi}(\mathbf{X},t) , where \mathbf{X} represents the undeformed material space and \mathbf{x} denotes the deformed space at time t . The deformation gradient \mathbf{F}(\mathbf{X},t)=\nabla_{\mathbf{X}}\phi(\mathbf{X},t) captures the local deformation of the material, encoding its rotation, stretch, and shear. Our work employs a hyperelastic model parameterized by a vector \theta_{p}=\{E,\nu\} , where E is Young’s modulus and \nu is Poisson’s ratio. To accurately represent these properties, the evolution of \phi must satisfy mass and momentum conservation: mass conservation ensures uniform density during deformation, while momentum conservation is formulated as follows: \rho(\mathbf{x},t)\dot{\mathbf{v}}(\mathbf{x},t)=\nabla\cdot\mathbf{\sigma}(\mathbf{x},t)+\mathbf{f}^{ext}, (2) where \rho is the mass density, \sigma is the Cauchy stress tensor, and \mathbf{f}^{ext} denotes external forces (tool-tissue interactions). The deformation gradient \mathbf{F}=\mathbf{F}^{E}\mathbf{F}^{P} decomposes into elastic ( \mathbf{F}^{E} ) and plastic ( \mathbf{F}^{P} ) contributions, allowing realistic modeling of tissue elastic recoil and permanent tearing under surgical forces. 2.3 Material Point Method (MPM) The Material Point Method (MPM) [ 25 , 7 ] provides an efficient solver