CoGE: Sim-to-Real Online Geometric Estimation for Monocular Colonoscopy

Liangjing Shao, Beilei Cui, Hongliang Ren · 2026

Read CoGE: Sim-to-Real Online Geometric Estimation for Monocular Colonoscopy by Liangjing Shao, Beilei Cui, Hongliang Ren (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13038.txt.

Introduction

Shao, Cui and Ren present CoGE, a framework for online geometric estimation during monocular colonoscopy, giving surgeons 3D spatial perception and navigation. True labels are hard to obtain inside the narrow, closed colon, and simulated data differs sharply from real footage in artifacts and lighting. CoGE counters with illumination-aware supervision from Retinex theory and structure-aware perception via wavelet decomposition. Trained only on simulated data, it still leads on both simulated and real scenes.

Catalog record

CREATORLiangjing Shao, Beilei Cui, Hongliang Ren
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15339
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13038

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoGE: Sim-to-Real Online Geometric Estimation for Monocular Colonoscopy Liangjing Shao, Beilei Cui, Hongliang Ren Abstract Geometric estimation including depth estimation and scene reconstruction is a crucial technique for colonoscopy which can provide surgeons with 3D spatial perception and navigation. However, geometric ground truth in colonoscopy is difficult to obtain due to narrow and enclosed space of the colon, while there is a large feature gap between simulated data and realistic data caused by artifacts and illumination. In this paper, we present CoGE, a novel framework for online monocular geometric estimation during colonoscopy. Firstly, we propose an illumination-aware supervision module based on the Retinex theory to address illumination diversity in different colonoscopy scenes. Moreover, a structure-aware perception module is proposed based on wavelet decomposition to extract common structural and local features of the colon. Both quantitative and qualitative results demonstrate that the proposed model solely trained on simulated data achieves state-of-the-art performance in geometric estimation for both simulated and realistic scenes. 1 1 footnotetext: Corresponding Author 1 Introduction Geometric estimation from monocular endoscopic videos is crucial for 3D perception, spatial navigation and path planning in colonoscopy, due to 2D vision of endoscope and long narrow closed pathway of colon. Recently, numerous 3D reconstruction foundation models [ 13 , 8 , 18 , 14 , 4 ] have been proposed for cross-scene geometric estimation. However, existing methods usually need a large amount of realistic data with geometric ground truth including depth map, point clouds and camera parameters. Such datasets with geometric ground truth are difficult and costly to obtain in realistic colonoscopy due to the narrow and closed space of colon. Thus, several self-supervised learning-based methods [ 6 , 16 ] have been developed for colonoscopy geometric estimation. However, the performance of self-supervised models is limited compared with supervised models. Nevertheless, geometric ground truth can be obtained by simulation software such as VR-Caps [ 5 , 11 ] . Therefore, sim-to-real geometric estimation has become a potential approach for 3D perception in endoscopic scenes. Currently, some existing works [ 9 , 10 ] have proposed sim-to-real methods for endoscopy geometric estimation. However, the proposed pipeline is evaluated after transforming the realistic data into the simulation-like data by a style translation network. Hence, the inference efficiency will be decreased due to data preprocessing, while inherent gaps between diverse realistic data and simulated data may degrade the cross-scene performance of appearance transformation models. Figure 1: Given observations from realistic and unseen scenes, our model only trained on simulated data outputs depth maps and 3D point clouds. In this paper, we propose a foundation model-based framework, CoGE , for sim-to-real g eometric e stimation in co lonoscopy. Firstly, to extract common structure features of colon, a structure-aware perception module based on wavelet transformation is proposed to extract local and edge features in colonoscopy images. Moreover, features of colon surface are easily interfered by illumination, so a illumination-aware supervision module is proposed to control the confidence of the estimated results based on light distribution. Besides, considering long distance of colonoscopy sequence, we propose a memory forget mechanism based on attention map between feature of current frame and memory cache to pretend interference of redundant memory. Our main contribution can be summarized as the following: 1) A structure-aware perception block is embedded based on wavelet transformation to extract common structural features in both simulated and realistic scenes. 2) An illumination-aware supervision module is proposed to control the confidence of the estimated results to pretend the influence of diverse illumination on colon surface. 3) A memory forget mechanism is proposed to filter redundant irrelevant memory for current observation against interference. 4) The proposed model only trained on simulated data achieves state-of-the-art performance for geometric estimation on both unseen simulated scenes and realistic scenes. 2 Method Figure 2: The pipeline of the proposed framework. 2.1 Overview Given a pair of observations including cached previous observation I_{t-1} and current observation I_{t} , the proposed model will output the corresponding point map X_{t} , camera pose P_{t} , illumination map L_{t} and confidence map C_{t} . Meanwhile, inspired by Spann3R [ 13 ] , two spatial memory caches M_{k},M_{v} are set and they will be updated to \hat{M_{k}},\hat{M_{v}} after memory reading, memory forgetting and memory updating based on the observations. The model architecture is demonstrated in Fig. 2 . 2.2 Structure-aware Encoding Firstly, we utilize six ViT-based modules to extract scene features from the observations f_{t-1}=\mathcal{F}_{enc}(I_{t-1}),f_{t}=\mathcal{F}_{enc}(I_{t}) . Each module consists of four ViT-L blocks followed by a structure-aware perception (SAP) block, which could extract common structural features from colonoscopy images. In SAP block, we firstly utilize discrete wavelet transformation to decompose the input graph into four subgraphs including low-low I_{ll} with low-frequency main approximation of the input, low-high I_{lh} with horizontal high-frequency details, high-low I_{hl} with vertical high-frequency details and high-high I_{hh} with diagonal high-frequency details. Convolution can extract high-frequency local details while attention is better for low-frequency information extraction. Thus, we utilize window-based multi-head attention (W-MHSA) to extract features f_{ll} from I_{ll} , while vertical-band convolution with 7\times 1 kernel, horizontal-band convolution with 1\times 7 kernel and diagonal convolution with 3\times 3 kernel are respectively utilized to extract features f_{hl},f_{lh},f_{hh} from I_{hl} , I_{lh} and I_{hh} . Then, features from different wavelet compositions are fused based on Eq. 1 - 3 . f_{h}=\mathcal{F}^{1\times 1}_{conv}(\mathcal{F}_{GELU}(\mathcal{F}^{1\times 1}_{conv}([f_{hl},f_{lh},f_{hh}]))) (1) f_{l}=\mathcal{F}_{lin}(\mathcal{F}_{GELU}(\mathcal{F}_{lin}(f_{hh}))) (2) f_{SAP}=\mathcal{F}_{idwt}(\mathcal{F}^{1\times 1}_{conv}([f_{h},f_{l}])) (3) where \mathcal{F}_{lin} denotes a linear layer, \mathcal{F}_{idwt} denotes inverse discrete wavelet transformation. 2.3 Memory-related Decoding Memory Reading. The feature from previous observation f_{t-1} is utilized to retrieve the related memory information from caches by Eq. 5 . W_{mem}=softmax(\frac{f_{t-1}M^{T}_{k}}{\sqrt{d}}) (4) f^{mem}_{t-1}=W_{mem}M_{v}+f_{t-1} (5) Figure 3: Attention weights distribution between feature tokens and memory caches. Memory Forget. For attention maps W_{mem}\in\mathbb{R}^{d\times s} between feature f_{t-1}\in\mathbb{R}^{d\times 1} and memory cache M_{k},M_{v}\in\mathbb{R}^{s\times 1} , if there is at least 5% feature tokens f_{t-1}(j) with an attention weight W_{mem}(j,i) larger than 5e-4 for each memory token M_{k}(i),M_{v}(i) , the memory token is seemed as the relevant tokens to be retained, demonstrated as Fig. 3 . M_{k}(i),M_{v}(i)\rightarrow\begin{cases}retain&|\{W_{mem}(j,i)\geq\text{5e-4}\}|\geq 5\%|\{W_{mem}(\cdot,i)\}|\\ delete&\text{otherwise}\end{cases} (6) Decoding and Memory Updating. Given memory-related feature f^{mem}_{t-1} and current feature f_{t} , the output feature of current frame and previous frame can be respectively obtained by six decoding blocks, each of which consists of self-attention and cross-attention \hat{f}_{t}=\mathcal{F}_{dec}(f_{t},f^{mem}_{t-1}),\hat{f}_{t-1}=\mathcal{F}_{dec}(f^{mem}_{t-1},f_{t}) . The new memory of key and value are encoded from memory-related feature \Delta M_{k}=\mathcal{F}^{key}_{enc}(f^{mem}_{t-1}) and output feature of previous frame \Delta M_{v}=\mathcal{F}^{val}_{enc}(\hat{f}_{t-1}) . The memory cache is updated by concatenating the new memory and filtered cached memory, \hat{M_{k}},\hat{M_{v}}=[M_{k},\Delta M_{k}],[M_{v},\Delta M_{v}] . Downstream Heads. Given the output feature of the current observation \hat{f}_{t} , DPT heads \mathcal{H}_{dpt} are utilized to output the corresponding point map X_{t} , camera pose P_{t} and confidence map C_{t} , following CUT3R [ 14 ] . Furthermore, the corresponding depth map D_{t} can be calculated based on the point map X_{t} , camera parameters P_{t} and K : X_{t},P_{t},C_{t}=\mathcal{H}_{dpt}(\hat{f}_{t}),D_{t}=\mathcal{Z}(X_{t},P_{t},K) (7) 2.4 Illumination-aware Supervision Figure 4: Input images and corresponding light influence maps. Considering diverse and unstable illumination in colonoscopy, we propose a self-supervised illumination-aware (IAS) module to adjust confidence map for supervision. Based on retinex theory [ 7 ] , the invariance of intrinsic component A_{t} of image I_{t} to illumination condition

Read the complete text