Jasmine: Harnessing Diffusion Prior for Self-Supervised Depth Estimation
Jiyuan Wang, Chunyu Lin, Cheng Guan, Lang Nie, Jing He, Haodong Li, Kang Liao, Yao Zhao · 2025
Read Jasmine: Harnessing Diffusion Prior for Self-Supervised Depth Estimation by Jiyuan Wang, Chunyu Lin, Cheng Guan, Lang Nie, Jing He, Haodong Li, Kang Liao, Yao Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15905.txt.
Introduction
Jiyuan Wang and co-authors present a depth estimation system called Jasmine in a paper from two thousand twenty-five. It guesses the depth of a scene from a single camera image, learning without labeled depth data. The system borrows visual knowledge from a large image generation model.
Earlier uses of such models needed exact supervision, while self-taught methods suffer from blocked views and flat regions that blur results. Jasmine avoids this with a hybrid task that rebuilds images themselves, keeping fine detail. A new recurrent module aligns the scale of the borrowed knowledge with the self-taught depth. Tests on a standard driving benchmark reached top scores, and the method carried well to new data sets.
Catalog record
| CREATOR | Jiyuan Wang, Chunyu Lin, Cheng Guan, Lang Nie, Jing He, Haodong Li, Kang Liao, Yao Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Jasmine: Harnessing Diffusion Prior for Self-Supervised Depth Estimation Jiyuan Wang, Chunyu Lin, Cheng Guan, Lang Nie, Jing He, Haodong Li, Kang Liao, Yao Zhao Abstract In this paper, we propose Jasmine , the first Stable Diffusion (SD)-based self-supervised framework for monocular depth estimation, which effectively harnesses SD’s visual priors to enhance the sharpness and generalization of unsupervised prediction. Previous SD-based methods are all supervised since adapting diffusion models for dense prediction requires high-precision supervision. In contrast, self-supervised reprojection suffers from inherent challenges ( e.g. , occlusions, texture-less regions, illumination variance), and the predictions exhibit blurs and artifacts that severely compromise SD’s latent priors. To resolve this, we construct a novel surrogate task of mix-batch image reconstruction. Without any additional supervision, it preserves the detail priors of SD models by reconstructing the images themselves while preventing depth estimation from degradation. Furthermore, to address the inherent misalignment between SD’s scale and shift invariant estimation and self-supervised scale-invariant depth estimation, we build the Scale-Shift GRU. It not only bridges this distribution gap but also isolates the fine-grained texture of SD output against the interference of reprojection loss. Extensive experiments demonstrate that Jasmine achieves SoTA performance on the KITTI benchmark and exhibits superior zero-shot generalization across multiple datasets. Project page and code are available at here . Figure 1: Without any high-precision depth supervision , Jasmine achieves remarkably detailed and accurate depth estimation results through zero-shot generalization across diverse scenarios. 1 Introduction Estimating depth from monocular images is a fundamental problem in computer vision, which plays an essential role in various downstream applications such as 3D/4D reconstruction [ 62 , 61 ] , autonomous driving [ 10 ] , etc. Compared with supervised methods [ 25 , 33 , 21 ] , self-supervised monocular depth estimation (SSMDE) mines 3D information solely from video sequences, significantly reducing reliance on expensive ground-truth depth annotations. These methods derive supervision from geometric constraints ( e.g. , scene depth consistency) through cross-frame reprojection loss, and the ubiquitous video data further suggests an unlimited working potential. However, view reconstruction-based losses suffer from occlusions, texture-less regions, and illumination changes [ 83 ] , which severely restrict the model’s capacity to recover fine-grained details and may cause pathological overfitting to specific datasets. Recent studies [ 25 ] demonstrated that SD possesses powerful visual priors to elevate depth prediction sharpness and generalization, which offers promising potential to address the above limitations. In addition, E2E FT [ 33 ] and Lotus [ 21 ] further reveal that single-step denoising can achieve better accuracy, which is particularly critical for self-supervised paradigms. It not only accelerates the inference denoising process but also significantly reduces the training costs in self-reprojection supervision, thereby creating opportunities to integrate SD into the SSMDE framework. However, fine-tuning diffusion models for dense prediction requires high-precision supervision to preserve their inherent priors [ 25 ] . Supervised methods typically employ synthetic RGB-D datasets, where clean depth annotations align with the high-quality SD’s training data, thereby keeping its latent space intact. In contrast, directly applying self-supervision introduces a critical challenge: reprojection losses or pre-trained depth pseudo-labels propagate perturbed gradients caused by artifacts and blurs into SD’s latent space, rapidly corrupting its priors during early training stages. Namely, high-precision “supervision" must exist at the beginning to protect SD’s latent space. Such supervision seems impossible in self-supervised learning, but we find a handy and valuable alternative: the RGB image. In fact, the image inherently contains complete visual details, avoids external depth dependencies in self-supervision, and aligns perfectly with SD’s original objective of image generation. Therefore, we construct a surrogate task of mix-batch image reconstruction (MIR) through a task switcher, where the same SD model alternately reconstructs synthesized/real images and predicts depth maps within each training batch. This strategy repurposes self-supervised reprojection loss to tolerate color variations while maintaining structural consistency [ 14 ] , intentionally decoupling color fidelity from depth accuracy, and finally preserving SD priors successfully. Another challenge is the output range of SD’s VAE [ 27 ] that is inherently bounded within a fixed range, i.e. , [-1, 1]. Existing methods typically normalize GT depth maps to this range in the training procedure. During inference, these supervised approaches perform least-squares alignment to recover absolute scale and shift, yielding scale- and shift-invariant (SSI) depth predictions. However, self-supervised frameworks rely on coupled depth-pose optimization, which theoretically requires shift invariance to be strictly zero for stable convergence, ultimately producing scale-invariant (SI) depth predictions. To bridge this inherent distribution gap, we propose a gated recurrent unit (GRU)-based novel transform module termed Scale-Shift GRU (SSG) . It not only iteratively aligns SSI depth to SI depth by refining scale-shift parameters but also acts as a gradient filter, which suppresses anomalous gradients caused by artifact-contaminated in self-supervised training, thereby preserving the fine-grained texture details of SD’s output while enforcing geometric consistency. Extensive experiments show that our proposed method, Jasmine, ❶ achieves the SoTA performance among all SSMDEs on the competitive KITTI dataset, ❷ shows remarkable zero-shot generalization across multiple datasets (even surpassing models trained with augmented data), ❸ demonstrates unprecedented detail preservation . As the first work to bridge self-supervised and zero-shot depth estimation paradigms, we also provide an in-depth analysis of the effects of different depth de-normalization strategies employed in their respective domains. To sum up, the main contributions are summarized as follows: • We first introduce SD into a self-supervised depth estimation framework. Our methods eliminate the dependency on high-precision depth supervision while retaining SD’s inherent advantages in detail sharpness and cross-domain generalization. • We proposed a surrogate task of MIR that anchors SD’s priors via self-supervised gradient sharing to avoid SD’s latent-space corruption caused by reprojection artifacts. • We proposed Scale-Shift GRU (SSG) to dynamically align depth scales while filtering noisy gradients to solve SSI versus SI distribution mismatch problems in self-supervised depth estimation. 2 Relative Work Self-supervised Depth Estimation Due to the high costs of GT-depth collection from LiDARs and other sensors, self-supervised depth estimation (SSDE) has gained significant research attention. SSDE can be broadly categorized into stereo-based(learning depth from synchronized image pairs) [ 52 , 14 , 12 , 3 , 2 ] and monocular-based methods (using sequential video frames) [ 31 , 87 , 85 , 32 , 37 , 83 , 20 , 34 , 19 , 29 ] . Additionally, when focusing on inference capability, existing methods can further diverge into single-frame and multi-frame approaches [ 55 , 17 , 9 , 1 ] . Comparisons between these paradigms are detailed in Sec. E.2 . Recently, DepthAnything v1/v2 [ 68 , 69 ] has revealed that we can obtain an accurate single image depth prediction model with strong generalization by training on large-scale image depth pairs. However, we argue that such datasets still remain a small fraction of the ubiquitous video data available. This observation motivates our exploration of the most challenging configuration: training exclusively on video sequences while maintaining single-frame inference capability, thereby laying the groundwork for developing genuinely versatile “DepthAnything” models. Diffusion for Depth Perception As the diffusion paradigm showcases its talents in generative tasks [ 22 , 46 , 28 , 76 , 51 , 59 ] , DDP [ 24 ] first reformulates depth perception as a depth