Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues
Chinmay Talegaonkar, Nikhil Gandudi Suresh, Zachary Novack, Yash Belhe · 2025
Read Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues by Chinmay Talegaonkar, Nikhil Gandudi Suresh, Zachary Novack, Yash Belhe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17358.txt.
Introduction
Chinmay Talegaonkar and co-authors turn a relative depth model into a metric one using blur, in a 2025 paper. Marigold, a diffusion model, judges depth order well but not real distances, and it fades on unfamiliar scenes. The trick needs no retraining: shoot the same view twice, once with a small and once with a wide aperture, and let defocus blur betray the true scale.
At run time the method tunes the model's scale settings and noise seeds against a blur formation equation. On a self gathered real dataset it beat standing zero shot metric methods in both numbers and judged quality.
Catalog record
| CREATOR | Chinmay Talegaonkar, Nikhil Gandudi Suresh, Zachary Novack, Yash Belhe |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues Chinmay Talegaonkar, Nikhil Gandudi Suresh, Zachary Novack, Yash Belhe Abstract Recent monocular metric depth estimation (MMDE) methods have made notable progress towards zero-shot generalization. However, they still exhibit a significant performance drop on out-of-distribution datasets. We address this limitation by injecting defocus blur cues at inference time into Marigold, a pre-trained diffusion model for zero-shot, scale-invariant monocular depth estimation (MDE). Our method effectively turns Marigold into a metric depth predictor in a training-free manner. To incorporate defocus cues, we capture two images with a small and a large aperture from the same viewpoint. To recover metric depth, we then optimize the metric depth scaling parameters and the noise latents of Marigold at inference time using gradients from a loss function based on the defocus-blur image formation model. We compare our method against existing state-of-the-art zero-shot MMDE methods on a self-collected real dataset, showing quantitative and qualitative improvements. 1 Introduction Estimating metric depth from a single camera viewpoint is a central problem in computer vision with numerous downstream applications, including 3D reconstruction [ 24 ] , autonomous driving [ 49 ] , and endoscopy [ 31 ] . This task, known as monocular metric depth estimation (MMDE), is fundamentally ill-posed due to inherent depth-scale ambiguity [ 45 ] . Multi-view methods [ 63 ] avoid this ambiguity but are often expensive and impractical in settings like endoscopy or microscopy. Training data-driven MMDE methods is challenging, as it requires accounting for a diverse set of camera parameters and metric depth scales. As a result, existing MMDE models struggle in zero-shot settings, i.e., they generalize poorly to unseen datasets. Recent advances in zero-shot (MMDE) [ 73 ] have demonstrated improved generalization, but there is still a considerable performance drop on unseen datasets. In contrast to MMDE, monocular relative depth estimation (MDE) methods recover a relative depth map, factoring out the physical depth scale. This enables using large-scale datasets with diverse depth ranges [ 41 ] for training data-driven MDE methods. As a result, MDE methods achieve better zero-shot generalization at significantly lower training cost than MMDE methods, as shown by recent results [ 68 , 69 , 26 ] . However, despite favorable performance on benchmarks, the absence of metric scale in MDE outputs precludes their applicability in downstream tasks requiring absolute depth. Existing data-driven MMDE methods commonly suffer from two failure modes: undesirable coupling between image texture and depth predictions ( fig. 3 ), and inaccurate estimation of the scene’s physical scale (see fig. 4 ). The first issue of texture coupling also affects MDE methods (see fig. 3 ) unless explicitly mitigated through complex training procedures that account for texture variation [ 22 , 53 ] . We demonstrate that incorporating camera physics, particularly defocus blur, with a data-driven MDE model at inference time can effectively address both failure modes without any re-training . We use Marigold [ 26 ] , a diffusion-based MDE model, as it enables backpropagation-based inference time optimization using defocus cues, while staying on its learned manifold of plausible depth maps. To provide these defocus blur cues, we capture two images from a single viewpoint: a small-aperture all-in-focus (AIF) image, which is given as input to Marigold, and a large-aperture image that provides physical cues for metric depth. Our approach requires only a variable aperture camera, such as a DSLR, and knowledge of the lens focal length, focus distance, and F-stop, which are readily available from image metadata; this avoids the need for extrinsic calibration required by multiview setups. Figure 1: Method overview. We capture two images (same viewpoint) from a camera with focal length f and focused at a distance F : an all-in-focus (AIF) image \mathbf{x} (F-stop: N=22 ) and a blurred image \mathbf{x}_{\text{b}} (F-stop: N<22 ). Using the AIF \mathbf{x} and an initial learnable noise vector \mathbf{z}_{T}^{(\mathbf{d})} , Marigold predicts the relative depth \mathbf{d} . We then affine transform \mathbf{d} with learnable parameters ( \alpha,\beta ), obtaining the metric depth \mathbf{d}^{\text{m}} . Given the AIF \mathbf{x} , depth \mathbf{d}^{\text{m}} , and camera parameters ( f,F,N ), we synthesize a blurred image \hat{\mathbf{x}}_{\text{b}} using the defocus blur forward model. To update the learnable parameters, we compute their gradients w.r.t the L2 loss between \hat{\mathbf{x}}_{\text{b}} and \mathbf{x}_{\text{b}} . Defocus blur is known to provide coarse metric depth cues in monocular settings [ 33 , 57 , 67 ] , but has rarely been explored alongside recent data-driven advances in MMDE. We show in our work that revisiting this classical cue and integrating it with modern relative depth methods like Marigold improves metric depth estimation performance, without any retraining. While most depth from defocus methods use multiple measurements [ 21 , 30 , 61 ] , we require only two. Prior work has also explored other hardware novelties to encode depth cues, such as dual pixel cameras [ 19 , 66 ] and coded apertures [ 28 ] . However, these depth cues are coarse, and training MMDE models for these systems is limited by the lack of datasets captured with a sufficient camera parameter diversity. This motivates the need for an approach that combines strong data-driven pre-trained MDE/MMDE models trained on existing RGB datasets with physics-based cues from non-pinhole cameras. To this end, our contributions are as follows: We formulate MMDE as an inverse problem under the defocus blur image formation model. Using inference-time optimization, we recover the metric depth scale and correct for texture-depth coupling – without retraining Marigold. We devise a hardware setup consisting of a rigidly coupled DSLR (RGB camera) and an Intel RealSense depth camera to capture RGB images and the ground truth metric depth. We collect a dataset of 7 diverse real-world indoor scenes, captured at different defocus blur levels. We compare our method on this dataset with current MMDE methods and demonstrate quantitative improvements on real data. 2 Related Work Zero-shot depth models Recent approaches for Monocular (relative) Depth Estimation (MDE) and Monocular Metric Depth Estimation (MMDE) can be classified as discriminative or generative. Discriminative approaches largely rely on vision transformer-based architectures [ 7 , 5 , 6 , 72 , 39 ] trained on large-scale datasets. Transformer-based methods have been more successful for MDE [ 68 , 69 ] compared to MMDE accuracy-wise, as MMDE is a more ill-posed task than MDE. These approaches have very low inference times, but for MMDE, the performance degrades on out-of-distribution test scenes 1 1 1 Refer to [ 73 ] for a detailed survey on MMDE methods. . Most of these methods are purely data-driven and ignore depth-based visual effects in images such as defocus cues. It is also difficult to incorporate physics-based depth refinement in these methods at test time without any re-training [ 71 ] . Since we use a generative MDE, our approach can incorporate physical cues at test time in a training-free manner. Current state-of-the-art generative MDE/MMDE methods [ 46 , 13 ] are predominantly diffusion-based. Several previous methods [ 13 , 46 , 32 ] incorporate diffusion-based depth denoising in their pipelines, achieving highly detailed depth maps – but are not zero-shot. [ 47 ] achieves zero-shot MMDE with a diffusion-based approach by incorporating diverse field