Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks

Alessio Quercia, Erenus Yildiz, Zhuo Cao, Kai Krajsek, Abigail Morrison, Ira Assent, Hanno Scharr · 2025

Read Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks by Alessio Quercia, Erenus Yildiz, Zhuo Cao, Kai Krajsek, Abigail Morrison, Ira Assent, Hanno Scharr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12824.txt.

Introduction

Alessio Quercia and co-authors present a 2025 paper on judging depth from a single photo. Depth maps guide robots and cars, but labeled depth data are costly and scarce. The team borrows data from sister tasks, such as labeling image regions. A shared decoder on a pre-trained vision model trains in turns on depth and helper tasks, with depth weighted most.

Across wide tests, helper data lifted depth quality by about 11 percent on average. Not all helper tasks helped alike, so choice matters more than volume. Region labeling data, recast as a dense multi-label task, often added extra gains. The method also kept quality while cutting depth data needs by at least 80 percent. Code is public, the team notes.

Catalog record

CREATORAlessio Quercia, Erenus Yildiz, Zhuo Cao, Kai Krajsek, Abigail Morrison, Ira Assent, Hanno Scharr
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12824

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks Alessio Quercia, Erenus Yildiz, Zhuo Cao, Kai Krajsek, Abigail Morrison, Ira Assent, Hanno Scharr Abstract Monocular depth estimation (MDE) is a challenging task in computer vision, often hindered by the cost and scarcity of high-quality labeled datasets. We tackle this challenge using auxiliary datasets from related vision tasks for an alternating training scheme with a shared decoder built on top of a pre-trained vision foundation model, while giving a higher weight to MDE. Through extensive experiments we demonstrate the benefits of incorporating various in-domain auxiliary datasets and tasks to improve MDE quality on average by ∼ 11 % \sim 11\% . Our experimental analysis shows that auxiliary tasks have different impacts, confirming the importance of task selection, highlighting that quality gains are not achieved by merely adding data. Remarkably, our study reveals that using semantic segmentation datasets as Multi-Label Dense Classification (MLDC) often results in additional quality gains. Lastly, our method significantly improves the data efficiency for the considered MDE datasets, enhancing their quality while reducing their size by at least 80%. This paves the way for using auxiliary data from related tasks to improve MDE quality despite limited availability of high-quality labeled data. Code is available at https://jugit.fz-juelich.de/ias-8/mdeaux . Figure 1 : (Top) NYUv2 results with MIX6 auxiliary MLDC. From left to right: input, ground truth, error difference (w.r.t. ground truth) between the DINOv2 baseline and ours. Green indicates ours is better, while red vice versa. (Bottom) AbsRel ( \downarrow ) for varying values of the task focusing parameter \alpha with multiple tasks (markers). The solid and dashed lines represent the mean and standard error of DINOv2, respectively. 1 Introduction Monocular depth estimation (MDE) is a well-established task in computer vision, with possible applications ranging from autonomous vehicles to augmented reality. It is inherently data-hungry, necessitating extensive, high-quality labeled datasets for effective training. Yet, the procurement of such datasets poses a significant challenge, often being a costly and time-consuming endeavor. Recent advances in monocular depth estimation have been driven by suitable adoption of Vision Transformers (ViT) [ 63 , 18 ] . In particular DPT [ 51 ] is the first attempt to relative MDE using ViT. Recent architectures, like MiDaS 3.1 [ 6 ] and ZoeDepth [ 4 ] extend DPT to multiple ViT backbones and metric monocular depth prediction, respectively. Furthermore, DINOv2 [ 46 ] is a vision foundation model with excellent zero-shot MDE capabilities when combined with a DPT decoder, and Depth Anything [ 69 ] fine-tunes this into an MDE foundation model using a large composition of labeled MDE and unlabeled vision data. Vision transformers have been primarily employed for individual tasks or in self-supervised learning paradigms. Recently, their application in Multi-Task Learning (MTL) is a growing area of research. In particular, there is an increasing interest in using a single model to predict multiple dense tasks, often employing MTL approaches that propose either a unified architecture [ 27 , 70 , 45 , 5 , 24 , 48 ] or methods to balance the loss functions during the training [ 67 , 56 , 25 , 12 , 23 , 39 , 3 ] . While MTL has been applied to MDE in prior studies [ 60 , 32 , 15 , 29 ] , we propose Multi-Label Depth Classification (MLDC) as an auxiliary task to improve MDE. Motivated by the limited availability of high-quality labeled MDE data, we propose a data-efficient and resource-considerate approach that leverages related vision datasets without requiring extensive fine-tuning of a pre-trained foundation model. This strategy improves MDE performance while reducing the reliance on MDE labels. Our approach leverages a frozen DINOv2 ViT Giant model [ 46 ] as a feature extractor, bypassing the need for fine-tuning. We jointly train a shared DPT decoder [ 51 ] with auxiliary datasets from related tasks to improve MDE. We illustrate the qualitative and quantitative improvements of our method over the DINOv2 baseline in Figure 1 . Our key contributions are summarized as follows: • We propose an alternating training scheme leveraging auxiliary non-MDE datasets from related vision tasks to boost the MDE downstream task. This improves the MDE performance by weighting MDE steps more than auxiliary ones through our task-focusing parameter \alpha ; • To the best of our knowledge, we are the first to use Multi-Label Dense Classification (MLDC) as an auxiliary task for MDE. • We find that MLDC frequently outperforms semantic segmentation as auxiliary task, suggesting that classification aspects may be more beneficial than spatial details, especially with low-quality segmentation labels. • We thoroughly test our method across various MDE datasets, using multiple auxiliary datasets and tasks. Our results show an average quality gain of 11% compared to the DINOv2 baseline on in-domain datasets, confirming the robustness of our method; • We show that our method enhances the data efficiency of DINOv2, allowing for a reduction in MDE training data of 80-99%, while still improving over DINOv2; These contributions represent a novel advancement in the field, both algorithmically and scientifically. Notably, compared to the recent Depth Anything [ 69 ] , which reports no improvements when jointly training their model (using DINOv2 ViT-L encoder) together with semantic segmentation data using task-specific DPT decoders, our method successfully leverages auxiliary tasks to enhance MDE. In particular, our approach keeps the larger DINOv2 ViT-G model frozen and jointly trains a single shared DPT decoder with an auxiliary task to improve MDE. In addition, our method has a reduced training cost by design as it does not fine-tune the backbone, while being bounded by its quality. In this sense, we see Depth Anything and methods that fine-tune the backbone as orthogonal to our work. 2 Related Work Monocular Depth Estimation. Recent works on Monocular Depth Estimation [ 52 , 4 , 30 , 31 , 36 , 35 , 37 , 43 , 46 , 50 , 57 , 68 , 65 ] primarily use Vision Transformers [ 63 , 18 ] . In particular DPT [ 51 ] , also known as MiDaS 3.0, is the first attempt to relative monocular depth estimation using Vision Transformers, adapting the original MiDaS CNN-architecture [ 52 ] . DPT has been used as a baseline for recent MDE SOTA architectures, for instance, MiDaS 3.1 [ 6 ] which shows performance of multiple ViT backbones in the MiDaS model, and ZoeDepth [ 4 ] , which combines DPT for relative depth estimation with a new module for metric depth prediction. Furthermore, DINOv2 [ 46 ] proposes a general-purpose vision foundation model that can be used together with DPT decoder to build a powerful zero-shot MDE predictor. Lastly, the recent Depth Anything [ 69 ] builds on top of DINOv2 and trains using a large combination of labeled MDE and unlabeled vision datasets. These works are task-specific and therefore miss the potential synergies between different vision tasks. In this paper, we show that the representations extracted by DINOv2 can be leveraged as a strong foundation for multi-task dense prediction, without further fine-tuning DINOv2. Multi-Task Dense Prediction. Multi-Task Learning (MTL) [ 8 , 61 , 72 , 54 ] leverages the idea that related tasks can provide complementary insights

Read the complete text