Feature-EndoGaussian: Feature Distilled Gaussian Splatting in Surgical Deformable Scene Reconstruction

Kai Li 1 Junhao Wang 1 William Han 2 Ding Zhao 2 1 University of Toronto 2 Carnegie Mellon University {kaii.li, hry.wang}@mail.utoronto.ca · 2025

Read Feature-EndoGaussian: Feature Distilled Gaussian Splatting in Surgical Deformable Scene Reconstruction by Kai Li 1 Junhao Wang 1 William Han 2 Ding Zhao 2 1 University of Toronto 2 Carnegie Mellon University {kaii.li, hry.wang}@mail.utoronto.ca (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06161.txt.

Introduction

Li, Wang, Han, and Zhao build in twenty twenty-five a real-time system that rebuilds moving surgical scenes in three dimensions and labels their parts at once. Keyhole surgery needs crisp live views of soft, low-texture tissue that shifts as tools work. Their method grows a cloud of soft blobs whose shapes track the tissue, enriched with meaning features distilled from a pretrained image model, so labels deform along with the tissue.

One rendering pass yields both the colour view and the label view. It runs at real-time frame rates with a compact footprint. On standard surgery datasets it set top marks for image fidelity and matched strong two-dimensional tools on labelling. The authors pitch it for safer surgical guidance.

Catalog record

CREATORKai Li 1 Junhao Wang 1 William Han 2 Ding Zhao 2 1 University of Toronto 2 Carnegie Mellon University {kaii.li, hry.wang}@mail.utoronto.ca
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06161

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Feature-EndoGaussian: Feature Distilled Gaussian Splatting in Surgical Deformable Scene Reconstruction Kai Li 1 Junhao Wang 1 William Han 2 Ding Zhao 2 1 University of Toronto 2 Carnegie Mellon University {kaii.li, hry.wang}@mail.utoronto.ca Abstract Minimally invasive surgery (MIS) has transformed clinical practice by reducing recovery times, minimizing complications, and enhancing precision. Nonetheless, MIS inherently relies on indirect visualization and precise instrument control, posing unique challenges. Recent advances in artificial intelligence have enabled real-time surgical scene understanding through techniques such as image classification, object detection, and segmentation, with scene reconstruction emerging as a key element for enhanced intraoperative guidance. Although neural radiance fields (NeRFs) have been explored for this purpose, their substantial data requirements and slow rendering inhibit real-time performance. In contrast, 3D Gaussian Splatting (3DGS) offers a more efficient alternative, achieving state-of-the-art performance in dynamic surgical scene reconstruction. In this work, we introduce Feature-EndoGaussian (FEG), an extension of 3DGS that integrates 2D segmentation cues into 3D rendering to enable real-time semantic and scene reconstruction. By leveraging pretrained segmentation foundation models, FEG incorporates semantic feature distillation within the Gaussian deformation framework, thereby enhancing both reconstruction fidelity and segmentation accuracy. On the EndoNeRF dataset, FEG achieves superior performance (SSIM of 0.97, PSNR of 39.08, and LPIPS of 0.03) compared to leading methods. Additionally, on the EndoVis18 dataset, FEG demonstrates competitive class-wise segmentation metrics while balancing model size and real-time performance. 1 Introduction Surgical techniques have continuously evolved by addressing shortcomings in previous methods and observing their impact on patient recovery ( Mithany et al. 2023 ) . Traditionally, open surgery has been associated with extended healing periods, increased risk of infection, and permanent scarring. In contrast, minimally invasive surgery (MIS), which gained popularity in the 1990s, offers reduced tissue damage, fewer post-operative complications, quicker recovery times, and shorter hospital stays ( Tsui et al. 2013 ) . MIS employs smaller incisions and utilizes an endoscope with an integrated camera to visualize internal organs, thereby generating surgical images and video data that can be further modeled and analyzed ( Ali et al. 2023 ) . Although more than 80% of surgical procedures have transitioned to MIS ( Tsui et al. 2013 ) , its success depends on high levels of technical skill and precision. Surgeons must operate while viewing an external monitor displaying the endoscopic feed, demanding exceptional hand-eye coordination and precise instrument control to avoid damage to adjacent tissues. This reliance on surgeon expertise underscores the need for assistive technologies that enhance patient safety and surgical performance. Recent advancements in artificial intelligence (AI) and the decreasing cost of computational resources have enabled the development of automated methods to assist MIS in real time ( Ali et al. 2023 ) . Tasks such as image classification ( Subedi et al. 2024 ) , object detection ( Yu et al. 2022 ) , semantic segmentation ( Zhu et al. 2024 ) , tissue tracking ( Wang et al. 2024 ) , and surgical scene reconstruction ( Zha et al. 2023 ) have achieved state-of-the-art performance, significantly benefiting robotic-assisted MIS. Surgical scene reconstruction has recently attracted considerable attention ( Zha et al. 2023 ; Long et al. 2021 ; Liu et al. 2020 ; Liu et al. 2024b ) . By rendering the target area in real time, this technique provides surgeons with a comprehensive view that enhances navigation and instrument control, and it holds promise for enabling robotic surgery automation. Early methods were dominated by traditional techniques such as simultaneous localization and mapping (SLAM) ( Chen et al. 2018 ; Song et al. 2018 ; Zhou and Jayender 2021a ) , but the field has since transitioned to neural network–based approaches ( Long et al. 2021 ; Li et al. 2020 ) and, more recently, to neural radiance fields (NeRFs) ( Mildenhall et al. 2020 ; Zha et al. 2023 ) . However, NeRF-based methods require large volumes of data and suffer from slow rendering speeds, motivating the development of 3D Gaussian Splatting (3DGS) ( Kerbl et al. 2023 ) as a more efficient alternative. Landmark studies such as EndoGaussian ( Liu et al. 2024b ) and LGS ( Liu et al. 2024a ) have successfully applied 3DGS to surgical scene reconstruction. More recently, feature field distillation techniques—initially developed for the NeRF domain ( Zhi et al. 2021 ; Siddiqui et al. 2022 ; Kobayashi et al. 2022 ) —have been extended to 3DGS ( Zhou et al. 2025 ; Zhou et al. 2024 ) beyond their application in surgical scene reconstruction. Inspired by these advances, we extend the 3DGS framework to support real-time segmentation and rendering by leveraging 2D segmentation foundation models ( Ravi et al. 2024 ; Zhu et al. 2024 ) . Our approach augments the traditional Gaussian deformation decoder with a lightweight MLP to predict per-Gaussian semantic feature updates and employs a CNN-based semantic decoder to align the rendered semantic feature map with features extracted by 2D segmentation models. This dual-level integration enables real-time, accurate semantic segmentation alongside high-fidelity scene reconstruction, effectively capturing dynamic tissue deformations and occlusions in MIS while identifying critical intraoperative objects. In summary, our contributions are as follows: 1. We introduce Feature-EndoGaussian (FEG), a semantic extension of 3DGS for surgical scene reconstruction that integrates 2D segmentation cues directly into the 3D rendering process. 2. Despite the additional complexity introduced by semantic feature distillation, our approach enhances surgical scene understanding by producing high-quality segmentations comparable to standalone 2D models while simultaneously improving reconstruction fidelity and maintaining model size and FPS. 2 Related Work 2.1 Surgical Scene Reconstruction Reconstructing soft tissues in robotic surgery from endoscopic stereo videos has been extensively studied, given its critical role in intraoperative navigation and automation of robotic-assisted procedures ( Lu et al. 2021 ; Penza et al. 2017 ; Liu et al. 2020 ) . Traditional approaches predominantly rely on Simultaneous Localization and Mapping (SLAM) techniques ( Chen et al. 2018 ; Song et al. 2018 ; Zhou and Jayender 2021a ; Gao and Tedrake 2019 ) , though these methods often struggle in complex surgical environments due to dynamic, non-rigid tissue deformations, specular reflections, and occlusions introduced by surgical instruments. More recently, learning-based methods leveraging convolutional neural networks (CNNs) and vision transformers have improved stereo depth estimation and the modeling of non-rigid deformations ( Long et al. 2021 ; Li et al. 2020 ) . However, these approaches still face limitations when generalizing to highly dynamic and unstructured surgical scenes, particularly in conditions with poor depth cues and significant occlusions. Figure 1 : Overview of FEG. The process begins with holistic Gaussian initialization, where 3D Gaussians are re-projected from image pixels to form the initial scene representation. A 4D voxel encoder extracts latent features, which are then processed by a deformation decoder that predicts updates to the Gaussian parameters (position, rotation, scale, opacity) and incorporates semantic feature adjustments via a lightweight MLP. The updated Gaussians are rendered through a differentiable rasterizer to produce both radiance and semantic feature maps. Finally, a CNN-based decoder upsamples and aligns the rendered semantic features with those extracted from a 2D segmentation model (SAM), ensuring accurate semantic consistency across the scene. Neural Radiance Fields Neural radiance fields (NeRFs) ( Mildenhall et al. 2020 ) have emerged as

Read the complete text