Unifying Scale-Aware Depth Prediction and Perceptual Priors for Monocular Endoscope Pose Estimation and Tissue Reconstruction
Muzammil Khan Enzo Kerkhof Matteo Fusaglia Koert Kuhlmann Theo Ruers Françoise J. Siepel, [0.5em] The Department of Surgical Oncology, The Netherlands Cancer Institute-Antoni van LeeuwenhoekPlesmanlaan 121, 1066 CX, Amsterdam, The Netherlands[0.5em] [email protected] , [email protected] , m.fusaglia@ · 2025
Read Unifying Scale-Aware Depth Prediction and Perceptual Priors for Monocular Endoscope Pose Estimation and Tissue Reconstruction by Muzammil Khan Enzo Kerkhof Matteo Fusaglia Koert Kuhlmann Theo Ruers Françoise J. Siepel, [0.5em] The Department of Surgical Oncology, The Netherlands Cancer Institute-Antoni van LeeuwenhoekPlesmanlaan 121, 1066 CX, Amsterdam, The Netherlands[0.5em] [email protected] , [email protected] , m.fusaglia@ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11282.txt.
Introduction
Muzammil Khan and co-authors present a 2025 system that rebuilds tissue surfaces from a single endoscope camera during surgery. Knowing where the scope is and what the tissue looks like in three dimensions helps surgeons steer and stay aware of space.
The task is hard because depth is unclear from one view, tissue shifts and bends, and the view is narrow. Their system first guesses depth with a sense of scale, then refines it over time by matching pixels across frames and blending them by how alike they look. A second part aligns each frame in space. At the end, the frames are fused into a full three dimensional mesh.
Catalog record
| CREATOR | Muzammil Khan Enzo Kerkhof Matteo Fusaglia Koert Kuhlmann Theo Ruers Françoise J. Siepel, [0.5em] The Department of Surgical Oncology, The Netherlands Cancer Institute-Antoni van LeeuwenhoekPlesmanlaan 121, 1066 CX, Amsterdam, The Netherlands[0.5em] [email protected] , [email protected] , m.fusaglia@ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-11282
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unifying Scale-Aware Depth Prediction and Perceptual Priors for Monocular Endoscope Pose Estimation and Tissue Reconstruction Muzammil Khan Enzo Kerkhof Matteo Fusaglia Koert Kuhlmann Theo Ruers Françoise J. Siepel, [0.5em] The Department of Surgical Oncology, The Netherlands Cancer Institute-Antoni van LeeuwenhoekPlesmanlaan 121, 1066 CX, Amsterdam, The Netherlands[0.5em] [email protected] , [email protected] , [email protected] , [email protected] , [email protected] , [email protected] Abstract Accurate endoscope pose estimation and 3D tissue surface reconstruction significantly enhances monocular minimally invasive surgical procedures by enabling accurate navigation and improved spatial awareness. However, monocular endoscope pose estimation and tissue reconstruction face persistent challenges, including depth ambiguity, physiological tissue deformation, inconsistent endoscope motion, limited texture fidelity, and a restricted field of view. To overcome these limitations, a unified framework for monocular endoscopic tissue reconstruction that integrates scale-aware depth prediction with temporally-constrained perceptual refinement is presented. This framework incorporates a novel MAPIS-Depth module, which leverages Depth Pro for robust initialisation and Depth Anything for efficient per-frame depth prediction, in conjunction with L-BFGS-B optimisation, to generate pseudo-metric depth estimates. These estimates are temporally refined by computing pixel correspondences using RAFT and adaptively blending flow-warped frames based on LPIPS perceptual similarity, thereby reducing artefacts arising from physiological tissue deformation and motion. To ensure accurate registration of the synthesised pseudo-RGBD frames from MAPIS-Depth, a novel WEMA-RTDL module is integrated, optimising both rotation and translation. Finally, truncated signed distance function-based volumetric fusion and marching cubes are applied to extract a comprehensive 3D surface mesh. Evaluations on HEVD and SCARED, with ablation and comparative analyses, demonstrate the framework’s robustness and superiority over state-of-the-art methods. K eywords Monocular reconstruction \cdot Depth estimation \cdot Endoscope pose estimation \cdot Foundation model \cdot Monocular endoscopy. 1 Introduction Minimally invasive surgery (MIS), including monocular and stereoscopic endoscopy as well as robotic systems, has transformed modern surgical practice by reducing incision size, limiting tissue trauma and lowering postoperative morbidity and recovery times [ 1 , 2 ] . Among these approaches, monocular minimally invasive surgery (MMIS) remains the most widely adopted technique due to its lower cost, simpler setup, and broad accessibility across diverse clinical settings [ 3 , 4 ] . However, MMIS is inherently constrained by a limited field of view and the absence of direct depth cues, forcing surgeons to rely on indirect visual indicators such as shading, motion, or anatomical landmarks to interpret spatial relationships. These perceptual limitations are further exacerbated in anatomically complex regions such as those involving hidden tumours, overlapping tissues, or proximity to critical vessels, posing challenges to accurate navigation and intraoperative decision-making. Whilst stereoscopic and robotic systems partially alleviate these issues through binocular disparity or enhanced visual interfaces, their adoption remains limited due to higher costs and integration complexities. In this context, recent advances in computer vision and deep learning have enabled computational techniques that can infer depth and spatial structure directly from monocular endoscopic images [ 5 , 6 , 7 , 8 ] . These approaches offer a hardware-agnostic solution to enhance spatial awareness and intraoperative guidance within existing MMIS workflows, representing a significant step towards safer, more precise, and context-aware MIS procedures. Despite these advancements, achieving accurate 3D surface reconstruction in MMIS remains technically challenging, primarily due to the lack of metric depth cues, which are essential for precise spatial measurements and accuracy in navigation [ 9 , 10 ] . Monocular systems lack stereoscopic disparity, relying instead on secondary visual cues such as motion parallax, shading gradients, and occlusion boundaries [ 11 , 12 , 13 , 14 ] . However, these inferred depth cues often result in non-metric or relative depth estimates, which are prone to noise and heavily dependent on the structural properties of the scene. Furthermore, the MMIS environment introduces dynamic, non-rigid deformations due to physiological processes such as respiration, complicating robust depth estimation and temporal alignment. These challenges significantly hinder the ability to achieve consistent metric depth accuracy [ 5 , 6 , 7 ] . Another major challenge in 3D reconstruction is the reliable registration of RGB-D frames, particularly in low-texture, homogeneous tissue regions commonly encountered in MMIS. The scarcity of distinctive features in these regions impairs the identification of stable spatial reference points, which are critical for effective feature matching and alignment [ 8 , 15 ] . This issue is further exacerbated in complex anatomical environments, where the endoscope must navigate through narrow, curved, and multi-layered pathways. These constraints necessitate high-precision depth and endoscope pose estimation strategies to ensure accurate and stable reconstructions [ 16 , 17 ] . 1.1 Contribution The primary contributions of this research are encapsulated in the design and development of a unified framework for monocular endoscope pose estimation and tissue reconstruction, which integrates scale-aware depth prediction with perceptually guided refinement and leverages both data-driven and optimisation-based methodologies. It comprises the following three core modules: • Metric-Aware Perceptual Image Similarity Depth (MAPIS-Depth) module: A novel depth estimation module that combines the strengths of the foundation models Depth Pro [ 18 ] and Depth Anything [ 19 ] , the former used for robust initialisation and the latter for efficient per-frame inference. The resulting depth maps are further refined using L-BFGS-B optimisation. To enhance temporal consistency and suppress artefacts from tissue deformation and motion, RAFT-based optical flow [ 20 ] is used to establish inter-frame correspondences, and LPIPS (Learned Perceptual Image Patch Similarity) [ 21 ] is employed to guide adaptive perceptual blending. • Weighted Exponential Moving Average-based Rotation-Translation Dog-Leg (WEMA-RTDL) module: A robust, multiscale \mathcal{SO}(3) – \mathcal{SE}(3) decoupled dog-leg optimisation strategy [ 22 ] , developed to improve frame-to-frame endoscope pose estimation in low-texture, homogeneous regions. It incorporates a WEMA refinement mechanism [ 22 , 23 ] to ensure stable and accurate transformation estimates. • Volumetric fusion module: A truncated signed distance function (TSDF)-based [ 24 ] approach for smooth monocular tissue surface reconstruction, with Marching Cubes [ 25 ] used to extract a triangulated mesh [ 26 ] . Furthermore, the proposed framework is validated across multiple benchmark datasets, including Hamlyn Endoscopic Video Dataset (HEVD) [ 26 ] and Stereo Correspondence and Reconstruction of Endoscopic Data (SCARED) [ 16 ] , with comprehensive performance assessments conducted through ablation and comparative studies. 2 Related Works Recent years have witnessed significant progress in monocular surface reconstruction, driven by advances in both learning-based and optimisation-based methodologies [ 27 , 28 , 29 , 30 , 31 ] . These approaches can be broadly categorised into offline and online methods, with online reconstruction being essential for real-time applications such as MMIS procedures. Online techniques typically involve real-time depth estimation, spatial correspondence modelling, and surface integration to produce temporally consistent monocular reconstructions [ 32 , 33 ] . These techniques are further categorised into geometry-driven, end-to-end deep learning-based, and hybrid approaches, each offering distinct advantages in balancing computational efficiency, accuracy, and adaptability to complex environments [ 34 , 35 , 36 , 37 , 38 ] . 2.1 Geometry-driven techniques Classical methods for monocular reconstruction commonly rely on geometric techniques involving feature detection, matching, and camera motion estimation [ 39 , 40 ] . Algorithms such as ORB-SLAM [ 41 ] , ORB-SLAM3 [ 42 ] , and PTAM [ 43 ] employ local descriptors including SIFT, SURF, and ORB [ 44 , 45