PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation

Xinqi Xiong, Andrea Dunn Beltran, Jun Myeong Choi, Marc Niethammer, Roni Sengupta · 2025

Read PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation by Xinqi Xiong, Andrea Dunn Beltran, Jun Myeong Choi, Marc Niethammer, Roni Sengupta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17067.txt.

Introduction

Xinqi Xiong and co-authors translate synthetic colonoscopy images toward realism while keeping their depth truth, in a 2025 paper. Judging depth inside the colon would aid navigation and diagnosis, but real procedures yield no ground-truth depth to train on. Synthetic scenes supply depth, yet models trained on them stumble on real tissue because the look differs.

Their framework, PPS-Ctrl, steers a diffusion model with maps of per-pixel shading, which capture how light plays across the moist surface. Shading pins the scene's structure more firmly than a depth map alone, so textures can turn realistic without warping anatomy. The translated images improved depth estimation over a GAN-based rival, and the code is public.

Catalog record

CREATORXinqi Xiong, Andrea Dunn Beltran, Jun Myeong Choi, Marc Niethammer, Roni Sengupta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-17067

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation Xinqi Xiong, Andrea Dunn Beltran, Jun Myeong Choi, Marc Niethammer, Roni Sengupta Abstract Accurate depth estimation enhances endoscopy navigation and diagnostics, but obtaining ground-truth depth in clinical settings is challenging. Synthetic datasets are often used for training, yet the domain gap limits generalization to real data. We propose a novel image-to-image translation framework that preserves structure while generating realistic textures from clinical data. Our key innovation integrates Stable Diffusion with ControlNet, conditioned on a latent representation extracted from a Per-Pixel Shading (PPS) map. PPS captures surface lighting effects, providing a stronger structural constraint than depth maps. Experiments show our approach produces more realistic translations and improves depth estimation over GAN-based MI-CycleGAN. Our code is publicly accessible at https://github.com/anaxqx/PPS-Ctrl . 1 Introduction Colorectal cancer is among the most lethal malignancies globally [ 1 ] , and colono-scopy remains the gold standard to detect intestinal lesions, such as polyps and colon cancer [ 17 ] . Accurate depth estimation in colonoscopy enhances spatial understanding and procedural guidance, playing a crucial role in developing assistive navigation and diagnostic tools. Improving estimation accuracy aids in guiding exploration to unsurveyed regions [ 23 ] , detecting blind spots [ 12 ] and polyps [ 21 ] , and robotic navigation systems [ 23 , 3 ] . As most endoscopes do not support high-quality 3D sensing, clinical colono-scopy datasets often lack ground-truth depth annotations. Consequently, most deep learning-based depth estimation models rely on synthetic datasets where depth annotations can be easily generated [ 5 , 10 , 15 ] . However, synthetic datasets often lack fine-grained textures, realistic lighting conditions, and complex anatomical structures, such as mucus-covered surfaces and polyp variations, present in real clinical data, making it difficult for models trained on synthetic data to generalize to real clinical data. Sim-to-real image translation has been explored to mitigate this gap, primarily through GAN-based methods [ 22 , 10 ] . However, GANs struggle with mode collapse, structural inconsistencies, and unstable training. Diffusion models offer a more stable alternative [ 19 ] , but they require stronger constraints to preserve structure in complex environments like endoscopy. We propose a novel sim-to-real image translation framework that leverages a diffusion model instead of GAN and introduces the Per-Pixel Shading (PPS) map [ 15 ] as a structural constraint, instead of a depth map, for guiding the Stable Diffusion model with a ControlNet. We further introduce an encoder-decoder architecture that extracts more meaningful structural information from PPS map as input to the ControlNet. Per-Pixel Shading map calculates the effect of lighting at each point on the surface defined by the depth map and accounts for the fact that points closer to the endoscope and facing the endoscope should receive a larger intensity of light compared to points further away. Explicit conditioning with the PPS map instead of depth leads to better alignment between the translated image and the structure of the source image while improving realism. To summarize, the contributions of our work are as follows: • We show that Per-Pixel Shading (PPS) map is a better structural constraint for sim-to-real translation than depth map. • We propose a novel structure-preserving sim-to-real image translation framework using Stable Diffusion and ControlNet, incorporating an encoder-decoder architecture trained to extract meaningful structural information from the PPS map to guide the ControlNet. • We demonstrate improved realism and depth preservation compared to state-of-the-art GAN-based method MI-CycleGAN [ 22 ] , as evidenced by 40% improvement in FID metric for C3VD (phantom) \rightarrow Colon10K (clinical) translation and 20% improvement in downstream depth prediction accuracy on the C3VD dataset for SimCol3D (simulation) \rightarrow C3VD (phantom) translation. 2 Related Work Colonoscopy datasets are derived from clinical procedures [ 2 , 14 ] , phantom-based data [ 4 ] , and fully synthetic sources [ 4 , 28 ] . While synthetic and phantom datasets provide depth and pose annotations, they lack real-world complexities such as dynamic lighting, specular reflections, and tissue deformations. Unpaired image translation using GAN-based approaches [ 9 ] with depth map [ 10 , 29 ] or mutual information [ 22 ] has been explored to preserve structure integrity. However, GANs can be unstable, prone to model collapse, and may exhibit limited diversity. Recently, due to its improved mode coverage and stability [ 19 ] , Stable Diffusion has been applied for clinical colonoscopy synthesis [ 8 , 24 ] and sim-to-real translation in surgical data [ 11 , 20 ] , but preserving structural integrity remains a challenge [ 6 , 26 , 7 ] , which we address in this paper by designing a novel PPS-Controlled Diffusion model. 3 Methods Figure 1: Our method takes a depth map from any synthetic colon datasets as input and generates textures similar to real endoscopy videos while preserving the depth in the generated image. Our proposed pipeline consists of a Stable Diffusion model that can capture image statistics of both real and synthetic domains using text prompts (trained in Stage 1), and a ControlNet that guides the diffusion model to perform depth-preserving image generation through a latent encoding of a Per-Pixel Shading (PPS) map (trained in Stage 2). Our goal is to perform synthetic-to-real image translation for endoscopy images while preserving the depth of the original synthetic image using a novel adaptation of Stable Diffusion [ 19 ] and ControlNet [ 27 ] , as shown in Fig. 1 . We first finetune Stable Diffusion on both the synthetic source and real target domains using text conditions to differentiate them, described in Sec. 3.1 . In Sec. 3.2 , we outline our novel adaptation of ControlNet [ 27 ] conditioned on latent Per-Pixel Shading (PPS) features obtained using an auto-encoder that significantly improves depth-preserving image translation quality over existing techniques. 3.1 Domain Separation using Text in Stable Diffusion When Stable Diffusion (SD) is naively finetuned on images across multiple domains, we observe that it ends up blending features between two different domains, reducing the effectiveness of adaptation. Therefore, we refine SD on both source and target domains with text-based domain separation during training and inference. To achieve this, domain separation is controlled via text conditioning: p_{\text{source}} = "A photo of synthetic colonoscopy" and p_{\text{target}} = "A photo of real colonoscopy", where p_{source} and p_{target} are text prompts corresponding to synthetic and real colonoscopy images respectively. By enforcing domain separation using textual prompts, we ensure that the SD generative model maintains distinct image characteristics for each domain. Finally, we train the SD model with parameters \theta^{S} with text prompt p and image latent code z_{t} at time-step t using the standard loss function for denoising diffusion model: \mathcal{L}=\mathbb{E}_{z_{t},\epsilon\sim\mathcal{N}(0,1),t,p}\left[\left\|\epsilon-\epsilon_{\theta^{S}}(z_{t},t,p)\right\|^{2}_{2}\right]\,. (1) 3.2 PPS-Ctrl for Depth Preservation Figure 2: We calculate error between ground-truth per-pixel shading map and intensity of the original image (Col 4), image generated by depth conditioning (Col 6) and PPS condition (Col 8) to show that PPS conditioning significantly decreases structural inconsistency and closely matches that of the original image. Existing techniques for content preserving image generation [ 6 , 26 ] , often use ControlNet [ 27 ] conditioned on depth maps to guide the Stable Diffusion to generate depth-preserving high-quality images. However, in

Read the complete text