MultiDepth: Multi-Sample Priors for Refining Monocular Metric Depth Estimations in Indoor Scenes
Sanghyun Byun, Graduate Student Member, IEEE, Jacob Song, Woo Seong Chung · 2024
Read MultiDepth: Multi-Sample Priors for Refining Monocular Metric Depth Estimations in Indoor Scenes by Sanghyun Byun, Graduate Student Member, IEEE, Jacob Song, Woo Seong Chung (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01048.txt.
Description
Abstract Monocular metric depth estimation (MMDE) is a crucial task to solve for indoor scene reconstruction on edge devices. Despite this importance, existing models are sensitive to factors such as boundary frequency of objects in the scene and scene complexity, failing to fully capture many indoor scenes.
Catalog record
| CREATOR | Sanghyun Byun, Graduate Student Member, IEEE, Jacob Song, Woo Seong Chung |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01048
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MultiDepth: Multi-Sample Priors for Refining Monocular Metric Depth Estimations in Indoor Scenes Sanghyun Byun, Graduate Student Member, IEEE, Jacob Song, Woo Seong Chung Abstract Monocular metric depth estimation (MMDE) is a crucial task to solve for indoor scene reconstruction on edge devices. Despite this importance, existing models are sensitive to factors such as boundary frequency of objects in the scene and scene complexity, failing to fully capture many indoor scenes. In this work, we propose to close this gap through the task of monocular metric depth refinement (MMDR) by leveraging state-of-the-art MMDE models. MultiDepth proposes a solution by taking samples of the image along with the initial depth map prediction made by a pre-trained MMDE model. Compared to existing iterative depth refinement techniques, MultiDepth does not employ normal map prediction as part of its architecture, effectively lowering the model size and computation overhead while outputting impactful changes from refining iterations. MultiDepth implements a lightweight encoder-decoder architecture for the refinement network, processing multiple samples from the given image, including segmentation masking. We evaluate MultiDepth on four datasets and compare them to state-of-the-art methods to demonstrate its effective refinement with minimal overhead, displaying accuracy improvement upward of 45%. (a) Original Image (b) Ground-Truth Depth (c) UniDepth (d) GT PUD Error (e) UniDepth PUD Error Fig. 1 : UniDepth [ 1 ] prediction inconsistency between pixel-unshuffled images. (d) shows L1 loss between ground-truth and Gaussian-blurred image for forgiveness, and (e) shows L1 loss between ground-truth and pixel-shuffled image of prediction on the pixel-unshuffled original image. PUD stands for pixel-unshuffle downscaling. I Introduction Accurate pixel-wise depth estimation is critical to various applications such as indoor 3D scene reconstruction, autonomous driving, and robotics. For indoor 3D scene modeling in real-world use cases, attaining posed image sequences is uncommon, especially on edge devices. Thus, monocular metric depth estimation becomes an important step in surface reconstruction for this purpose. However, variables such as hardware limitations, scene complexity, and view-dependent effects (VDE) present a unique challenge, which we propose to solve by using the task of Monocular Metric Depth Refinement (MMDR). While advancements in monocular metric depth refinement (MMDE) methods [ 1 , 2 ] with the introduction of ViT [ 3 , 4 ] demonstrate impressive metric accuracy, detail, and camera invariance, the change in image resolution, scene complexity, and boundary frequency impact their capabilities significantly. Such inconsistencies across image capture factors can be seen when we apply sampling algorithms such as pixel-unshuffle [ 5 ] and random subsampling. Fig. 1 demonstrates the inconsistency in UniDepth [ 1 ] between image pairs that have been pixel-unshuffled. For forgiveness around extremely high-frequency surface areas, we give less weight to the boundary with canny edge detection. As shown, it struggles to predict the depth when a lower-resolution image is fed into the network. Similarly, Fig 2 shows inconsistency between subsampled and original image predictions. Although some surfaces are accurately predicted, they completely miss certain separable areas, displaying the model’s dependency on object relationships derived from its self-attention layer. As such discrepancies display an area of improvement, we directly address these issues in our proposed method. (a) GT sample (b) UniDepth sample (c) Sample Error Map Fig. 2 : UniDepth [ 1 ] prediction inconsistency between sub-sampled images. (a) shows ground-truth sub-sample. (b) shows UniDepth prediction on a sub-sampled image. (c) shows L1 loss between ground truth and depth prediction, where green indicates greater loss. To improve upon some of the discrepancies in the predicted depth maps, iterative depth refinement methods [ 6 , 7 , 8 ] have been proposed. However, current methods largely rely on segmented object normals as priors. Such methods produce impressive results if accurate normal estimation is made. However, the presence of a large prediction error causes such an error to propagate to the depth refinement. In addition to this limitation, it raises model size and complexity, which is often a limiting factor for devices on Edge. In this work, we attempt to address the task of iteratively refining the predicted depth map without significantly increasing the model size or complexity. Our proposed method, MultiDepth, is the first to delve into scene complexity to improve model accuracy and detail. Unlike existing refinement pipelines, MultiDepth does not predict normal maps of objects, which creates unnecessary overhead. Instead, we employ various sampling strategies for quick iterations, which improve Edge compatibility. Due to its lightweight nature, it can be quickly utilized for mesh initialization for surface reconstruction methods. To solve the task of MMDR by refining depth estimations generated by state-of-the-art MMDE models, we propose an iterative mechanism for leveraging various image sampling strategies to obtain view-independent depth maps that are less sensitive to boundary frequency and image scale while being aware of model size to allow easier integration on edge devices. The proposed architecture can be finetuned on any depth estimation backbone, improving its flexibility for future research in depth estimation. Our major contributions are listed as follows: • We employ multiple sampling strategies such as segmentation, random subsampling, and pixel-unshuffling to input into a depth refinement network for better environment-independent depth-map prediction of the given scene from a single image. • We propose a novel depth refinement method to iteratively improve the monocular metric depth estimation output result through a lightweight encoder-decoder network that can be run on Edge. • We design an aggregation method that considers multi-sample depth maps and concatenates them into the most probable depth map estimation. II Related Works Existing methods for monocular depth estimation can be divided into three focus areas: scale-agnostic depth estimation, monocular metric depth estimation, and segmented depth estimation. II-A Monocular Depth Estimation Scale-agnostic depth estimation models [ 9 , 10 , 11 , 12 ] tend to focus more on the detail of the depth map, concentrating on the accurate representation of high-frequency and semi-transparent objects. Some of the distinct features of such methods include a diverse range of dataset usage, allowing various models to become more universal through exposure. For example, MiDas [ 10 ] can be trained on various models, although its DPT-based version is the most accurate. Monocular Metric Depth Estimation (MMDE) is a problem initially addressed by [ 13 ] . Recent developments in advanced architectures [ 1 , 2 , 14 , 15 , 16 , 17 ] utilize transformers and diffusion models to achieve more universal MMDE. Especially with the introduction of ViT [ 3 ] and xFormers [ 3 ] for acceleration, MMDE has evolved significantly as pixel-to-pixel relationships can be accessed more efficiently without losing context. Despite such improvements, universal MMDE models struggle in high-frequency and multi-zoom scenarios, displaying the need for robust consistency optimization against various sampling methods such as zoom and masking. Segmented Depth Estimation models [ 18 , 19 ] breaks down the depth estimation into multiple pipelines by handling each segmented component separately. In SuperPrimitive [ 18 ] , the input image is further segmented into unlabeled masked images, and normal maps are predicted for each object. Such methods allow smoother object depth consistency as each prediction is bound to the corresponding normal. However, these methods often come short in scenarios of large normal and depth prediction disparity, as they may have successfully generated each