EndoMUST: Monocular Depth Estimation for Robotic Endoscopy via End-to-end Multi-step Self-supervised Training

Liangjing Shao, Linxin Bai, Chenkang Du, Xinrong Chen · 2025

Read EndoMUST: Monocular Depth Estimation for Robotic Endoscopy via End-to-end Multi-step Self-supervised Training by Liangjing Shao, Linxin Bai, Chenkang Du, Xinrong Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16017.txt.

Introduction

Liangjing Shao and colleagues estimated depth inside the body from a single endoscope camera in a 2025 paper. Robotic endoscopy needs depth and camera motion to navigate safely, but the scene fights back: lighting swings, textures are sparse, and wet tissue reflects. Their EndoMUST framework trains self-supervised, with no depth labels, by splitting each training round into three focused steps so modules do not interfere with one another.

Optical flow registration, multi-scale image decomposition, and transformation alignment each train only their own networks in turn. Parameter-light fine-tuning adapts a foundation model underneath. The result set state-of-the-art depth scores on a standard endoscopy dataset and transferred zero-shot to another, promising safer surgical navigation.

Catalog record

CREATORLiangjing Shao, Linxin Bai, Chenkang Du, Xinrong Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-16017

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EndoMUST: Monocular Depth Estimation for Robotic Endoscopy via End-to-end Multi-step Self-supervised Training Liangjing Shao, Linxin Bai, Chenkang Du, Xinrong Chen Abstract Monocular depth estimation and ego-motion estimation are significant tasks for scene perception and navigation in stable, accurate and efficient robot-assisted endoscopy. To tackle lighting variations and sparse textures in endoscopic scenes, multiple techniques including optical flow, appearance flow and intrinsic image decomposition have been introduced into the existing methods. However, the effective training strategy for multiple modules are still critical to deal with both illumination issues and information interference for self-supervised depth estimation in endoscopy. Therefore, a novel framework with multistep efficient finetuning is proposed in this work. In each epoch of end-to-end training, the process is divided into three steps, including optical flow registration, multiscale image decomposition and multiple transformation alignments. At each step, only the related networks are trained without interference of irrelevant information. Based on parameter-efficient finetuning on the foundation model, the proposed method achieves state-of-the-art performance on self-supervised depth estimation on SCARED dataset and zero-shot depth estimation on Hamlyn dataset, with 4% ∼ \sim 10% lower error. The evaluation code of this work has been published on https://github.com/BaymaxShao/EndoMUST. I INTRODUCTION Endoscopy has been one of the most important minimally invasive surgical procedures. Robotic system could highly enhance the stability and accuracy of endoscopy. Due to the limited visual field and 2D vision of the endoscope, 3D scene perception is a critical task for more efficient, safe and accurate robot-assisted endoscopy. Depth estimation has been a common technique for 3D perception, which can also served as a basis of 3D reconstruction, registration and AR screening. [ 1 , 2 ] The traditional methods for depth estimation are mainly based on feature matching from multi-view images. [ 3 ] Due to sparse feature in endoscopic images, the performance of traditional stereo matching methods is severely limited in endoscopy. [ 4 ] With high performance of feature representation, stereo matching based on deep learning has been proposed for depth estimation in endoscopy. Moreover, the inference speed of deep learning-based stereo matching methods cannot achieve the real-time requirements. Based on the most recent works [ 5 ] , the highest inference speed of the deep learning-based stereo matching is only 11 frames per second [ 6 ] in endoscopic scenes. Thus, instead of traditional methods and binocular stereo matching, the deep learning-based monocular depth estimation methods with higher efficiency have gained common attention with the development of large-scale datasets. Fig. 1: Challenges for depth estimation in robotic endoscopy. Highlights of our method are marked by green boxes. Deep learning-based methods can be categorized into supervised depth estimation and self-supervised depth estimation. However, fully-supervised depth estimation methods, such as [ 7 ] and [ 8 ] , will face the high cost for obtaining the ground truth of depth maps, especially for the narrow and enclosed space in endoscopy. Therefore, Mahmood et al. [ 9 ] and Chen et al. [ 10 ] utilize synthetic data to overcome the unavailability of large-scale and accurate RGB-D dataset for endoscopy. Nonetheless, the large domain gap between synthetic data and real data will make such methods fall short of desirable performance in realistic endoscopic scenes. Although a large quantity of self-supervised monocular depth estimation methods [ 11 , 12 , 13 ] have been proposed for natural scene, the depth estimation task is still challenging because of the lighting inconsistency and non-Lambertian reflection on soft tissues. In recent years, lots of researches have been proposed to tackle such problems in endoscopic scenes. Shao et al. [ 14 ] deal with the lighting inconsistency by generalized dynamic image constraint based on the appearance flow, while Yang et al. [ 15 ] propose to filter the lighting reflectance by intrinsic image decomposition. However, the effective training strategy is still needed to efficiently and jointly tackle all illumination challenges in endoscopy. Meanwhile, for robotic endoscopy, the occlusion of the end effector is also a potential challenge for depth estimation. In this work, a self-supervised framework with a novel multi-step training strategy is proposed to jointly cope with the challenges in robotic endoscopy shown as Fig. 1 with low cost. Specifically, each end-to-end training epoch consists of three training steps for different networks to prevent from the misleading of the irrelevant constraints. The contribution of this work can be summarized as the following: 1. A novel three-step self-supervised training strategy is proposed to isolate different aspects of constraints and functions, with well-designed loss functions as well. 2. In the second step, a novel multiscale self-supervised training method is proposed to capture more detail representations for optimization of intrinsic image decoposition. 3. In the third step, a full-module parameter-efficient finetuning based on DV-LoRA [ 16 ] is applied into the training of depth map generation network, to control training cost and increase the adaptation. 4. The proposed method achieves the state-of-the-art performance in both self-supervised depth estimation on SCARED dataset and zero-shot depth estimation on Hamlyn dataset, compared with existing methods. II RELATED WORK Self-supervised depth estimation has been developed for a long period. Turan et al. [ 17 ] proposed the first method to estimate camera pose and depth information from image sequences using self-supervised learning for endoscopic capsule robots. Liu et al. [ 18 ] introduced the use of Structure from Motion (SfM) to extract supervision signals from monocular endoscopic videos. Li et al. [ 19 ] proposed to use peak signal-to-noise ratio to evaluate the similarity between synthesized and target images. In recent years, some works enhanced the network structure by utilizing attention mechanisms. Endo-SfMLearner [ 20 ] combined a spatial attention mechanism to highlight textured regions. Based on this, Liu et al. [ 21 ] added position and channel attention modules to the network to capture multi-scale context, improving depth estimation accuracy. Furthermore, Yang et al. [ 2 ] combined CNNs and Transformers in the depth estimation part to extract both local texture and global contour features. However, the issues of illumination and sparse texture are not considered in the above works. In recent years, based on depth estimation for natural scenes, several works have focused on addressing illumination issues in endoscopic scenes. Shao et al. [ 14 ] introduced appearance flow, which accounts for changes in brightness patterns and incorporates the generalized dynamic image constraint. Differently, Li et al. [ 15 ] proposed the Intrinsic Image Decomposition (IID) method, which decomposes an image into two fundamental components, albedo and shading for self-supervised learning. Based on the self-supervised training of [ 14 ] , some works begin to focus on the training strategy. With the development of foundation model for depth estimation [ 22 , 23 ] , EndoDAC [ 16 ] proposed Dynamic Vector-Based Low-Rank Adaptation and convolutional neck blocks to efficiently finetune Depth Anything model [ 22 ] for endoscopic depth estimation. Recently, DVSMono [ 24 ] proposed a new dynamic view selection mechanism to address the pose alignment error from minimal camera motion. In this work, different from above works, a well-designed three-step training framework is proposed to jointly deal with feature gap, illumination inconsistency and reflection interference based on full-module parameter-efficient training. III PROPOSED METHOD III-A Overview Fig. 2: The

Read the complete text