Boosting Generalizable Depth Estimation in Endoscopy by Mixture of Lightweight Experts and Intrinsic Image Alignment

Liangjing Shao, Beilei Cui, Yiming Huang, Changjing Liu, Hongliang Ren · 2026

Read Boosting Generalizable Depth Estimation in Endoscopy by Mixture of Lightweight Experts and Intrinsic Image Alignment by Liangjing Shao, Beilei Cui, Yiming Huang, Changjing Liu, Hongliang Ren (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00415.txt.

Introduction

Shao and co-authors boost generalizable depth estimation in endoscopy with a self-supervised framework. Depth reading is key for 3D perception in endoscopic surgery. Illumination interference and scene diversity still block general depth models. Their framework, EndoMINI, attacks both issues. A mixture of low-rank experts fine-tunes the model with few parameters.

Those experts also lift adaptation to scenes with distinct traits. An intrinsic image alignment term joins the training loss. It eases the influence of light reflectance through a decomposition network. Tests cover supervised depth on SCARED data. Zero-shot tests run on Hamlyn and SERV-CT sets. The model shows outstanding performance against state-of-the-art works, they report. Ablations credit both main contributions.

Catalog record

CREATORLiangjing Shao, Beilei Cui, Yiming Huang, Changjing Liu, Hongliang Ren
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2608.00415

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Boosting Generalizable Depth Estimation in Endoscopy by Mixture of Lightweight Experts and Intrinsic Image Alignment Liangjing Shao, Beilei Cui, Yiming Huang, Changjing Liu, Hongliang Ren Abstract Depth estimation is a significant task for 3D perception in endoscopic surgeries. However, illumination interference and feature diversity in various endoscopic scenes are still challenges for generalizable depth estimation and ego-motion estimation. Based on this, a novel self-supervised framework, EndoMINI, is proposed for depth estimation in endoscopic scenes. Specifically, mixture of low-rank experts (MiLoRE) is proposed to perform parameter-efficient fine-tuning, which can also boost the model adaptation to scenes with different characteristics. Meanwhile, an intrinsic image alignment (IIA) is introduced into the training loss to alleviate the influence of light reflectance in endoscopy with a novel intrinsic image decomposition network. The proposed method is evaluated on SCARED datasets for supervised depth estimation, and two endoscopic datasets, Hamlyn and SERV-CT, for zero-shot depth estimation, compared with state-of-the-art works as well. The experimental results demonstrate outstanding performance of the proposed model and the effects of the main contributions. 1 1 footnotetext: Corresponding Author 1 Introduction Depth estimation has been a vital task for minimally invasive endoscopic surgery, especially for robotic surgery, which plays an important role in navigation and 3D perception. [ 3 ] A series of self-supervised learning frameworks have become the mainstream methods for depth estimation in natural scenes [ 7 , 19 , 18 ] . However, features of endoscopic scenes are quite different from natural scenes. Meanwhile, illumination inconsistency is also another critical problem in endoscopic scenes. Currently, a large amount of novel frameworks are proposed for depth estimation in endoscopic scenes, which tackle these problems with various alignment losses [ 13 , 9 , 20 ] , parameter-efficient fine-tuning(PEFT) [ 4 ] , and efficient feature encoders [ 17 , 11 ] . However, there are still two challenges for existing methods as Fig. 1 shows. Firstly, for endoscopic data, even from the same dataset, the features of endoscopic scenes are quite different due to various characteristics of tissues. Therefore, feature encoders usually suffer from corresponding representation extraction for various scenes and objects. Furthermore, although the existing method [ 13 ] alleviates the illumination inconsistency with appearance flow and optical flow, the reflectance on the surface of tissues in different illumination conditions still affects the accuracy of depth estimation. Figure 1: The diversity in endoscopic scenes. Recently, some remarkable works are proposed to deal with these two challenges. In EndoDAC [ 4 ] , a novel LoRA module is proposed to perform PEFT on Depth Anything [ 15 ] , a general depth estimation model. However, single LoRA layer still faces the challenge of feature adaptation for various scenes. Moreover, a recent work, IID-SfMLearner [ 9 ] , proposed to utilize intrinsic image decomposition to deal with the reflectance problem in endoscopy. However, the style gap between decomposed images and illumination inconsistency are ignored in their proposed framework. Currently, mixture of experts (MoE) [ 2 ] has been an excellent parameter-efficient fine-tuning method, which can adjust the activation of different experts according to the features from various data. Therefore, to tackle the problem from various endoscopic characteristics, a novel PEFT module is proposed, in which LoRA layers are used as experts in a MoE framework. Furthermore, in this work, an intrinsic image alignment based on a novel self-supervised style-controlled intrinsic image decomposition is proposed and integrated with reconstruction alignments based on optical flow and appearance flow [ 13 ] . To this end, our main contributions are summarized as the following: 1. A novel parameter-efficient fine-tuning framework with integration of MoE and LoRA (MiLoRE) is proposed to boost the generalization of the model for endoscopic scenes with various characteristics. 2. With a novel self-supervised intrinsic image decomposition network, intrinsic image alignment including an illumination-free alignment and an intrinsic image reconstruction alignment is introduced into the training stage to alleviate the influence of various light conditions. 3. The depth estimation network based on self-supervised learning with the proposed framework outperforms state-of-the-art methods on SCARED dataset for supervised depth estimation, Hamlyn and SERV-CT dataset for zero-shot depth estimation. Besides, the proposed framework provides the most accurate ego-motion estimation and camera intrinsics prediction. 2 Proposed Method 2.1 Self-supervised Pipeline Figure 2: Self-supervised pipeline for endoscopy depth estimation. The proposed pipeline in this work is shown in Fig. 2 . The depth map generation network in the proposed framework mainly consists of a pretrained ViT encoder with convolution blocks inserted and a DPT-like decoder from EndoDAC [ 4 ] and Depth Anything [ 15 ] . Specifically, a novel mixture of low-rank experts is proposed to perform parameter-efficient fine-tuning on each Transformer block in the pretrained ViT encoder. Apart from existing reconstruction alignments and smoothness optimization [ 13 ] , an illumination-free alignment and an intrinsic image reconstruction alignment based on intrinsic image decomposition are introduced to optimize the depth estimation and ego-motion estimation. Besides, a novel self-supervised intrinsic image decomposition network is proposed to obtain accurate reflectance images and shading images for the illumination-free alignment and the intrinsic image reconstruction alignment. 2.2 Mixture of Low-Rank Experts Figure 3: Parameter-efficient finetuning on each Transformer block in ViT-B encoder based on mixture of low-rank experts. In the proposed mixture of low-rank experts, the low-rank adaptation modules can adaptively adjust the output of the finetuned module based on the input data. To achieve this, the probability distribution of experts is generated by a router network \mathcal{R} based on the input feature at first. Based on this, k experts with highest probability are selected to perform weighted Low-Rank adaptation (LoRA [ 8 ] ). For the finetuned module with the frozen weight W_{0}\in\mathbf{R}^{d\times k} , given the input x , the output y is adjusted by selected k low-rank experts as Eq. 1 shows. In Eq. 1 , B_{i}\in\mathbb{R}^{d\times r} and A_{i}\in\mathbb{R}^{r\times k} are trainable weights, r\ll min(d,k) . To increase the robustness of the router, a noise vector \epsilon(x) is predicted based on the input and integrated with a random noise vector \delta . y=Wx=W_{0}x+\sum_{i=1}^{k}(\mathcal{R}(x)+\epsilon(x)\times\delta)_{i}B_{i}A_{i}x (1) To finetune the pretrained ViT encoder for adaptation to the endoscopic scenes, three modules based on MiLoRE are implemented in each Transformer block. As Fig. 3 shows, one MiLoRE module is utilized to finetune ’QKV Linear’ (QKV) which generates query vector, key vector and value vector for self-attention operation. Moreover, another two MiLoRE modules are set to respectively finetune two MLPs in the feed forward network (FFN) of the block. 2.3 Intrinsic Image Decomposition Figure 4: The framework of the proposed intrinsic image decomposition network. The pipeline of the proposed intrinsic image decomposition network is shown in Fig. 4 . Due to the style gap between the reflectance image and the shading image, two MLPs are introduced into the proposed intrinsic image decomposition network to generate the style vector z^{R} and z^{S} , respectively. To control the style of the generated image, referring to [ 10 ] , Adaptive Instance Normalization (AdaIN) is utilized instead of original normalization layer in the decomposition decoder. The parameters of AdaIN are from the corresponding style vector dynamically generated by the specific MLP. The process of AdaIN is shown as Eq. 2 , in which

Read the complete text