ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimation

Yan Weilong, Zhang Xin, Robby T. Tan · 2025

Read ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimation by Yan Weilong, Zhang Xin, Robby T. Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00665.txt.

Introduction

Weilong Yan, with two co-authors, Xin Zhang and Robby T Tan, tackles monocular depth estimation, judging distance from a single image, in bad weather such as rain, fog, snow, and night. Ground truth is scarce in such scenes, synthetic data brings domain gaps, and self-supervised methods break when light patterns change.

Yan and his co-authors fine-tune vision foundation models efficiently using only a small amount of clear-weather data. Their Selecting-Tuning-Maintaining strategy splits pre-trained weights by effective rank measures, chooses ranks and directions to tune, then regularizes key directions to preserve prior knowledge. Across four real benchmarks, the method beats other efficient tuning methods, full fine-tuning, and even models trained on synthetic bad-weather data.

Catalog record

CREATORYan Weilong, Zhang Xin, Robby T. Tan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-00665

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimation Yan Weilong, Zhang Xin, Robby T. Tan Abstract Monocular depth estimation under adverse weather conditions (e.g., rain, fog, snow, and nighttime) remains highly challenging due to the lack of reliable ground truth and the difficulty of learning from unlabeled real-world data. Existing methods often rely on synthetic adverse data with pseudo-labels, which suffer from domain gaps, or employ self-supervised learning, which violates photometric assumptions in adverse scenarios. In this work, we propose to achieve weather-generalized depth estimation by Parameter-Efficient Fine-Tuning (PEFT) of Vision Foundation Models (VFMs), using only a small amount of high-visibility (normal) data. While PEFT has shown strong performance in semantic tasks such as segmentation, it remains underexplored for geometry-centric tasks like depth estimation—especially in terms of balancing effective adaptation with the preservation of pretrained knowledge. To this end, we introduce the Selecting-Tuning-Maintaining (STM) strategy, which structurally decomposes the pretrained weights of VFMs based on two kinds of effective ranks (entropy-rank and stable-rank). In the tuning phase, we adaptively select the proper rank number as well as the task-aware singular directions for initialization, based on the entropy-rank and full-tuned weight; while in the maintaining stage, we enforce a principal direction regularization based on the stable-rank. This design guarantees flexible task adaptation while preserving the strong generalization capability of the pretrained VFM. Extensive experiments on four real-world benchmarks across diverse weather conditions demonstrate that STM not only outperforms existing PEFT methods and full fine-tuning but also surpasses methods trained with adverse synthetic data, and even the depth foundation model, under both supervised and self-supervised settings. 1 Introduction Monocular Depth Estimation (MDE) is a core task in computer vision, supporting a wide range of applications including robotics yu2023udepth , autonomous driving zheng2024physical ; deep-homo , medical tasks wang2025monopcc ; pvchat , reconstruction ssnerf , and scene understanding schon2021mgnet ; 3dswapping . Recent advances have largely centered on self-supervised MDE 2017left-right-consistency ; godard2019digging ; litemono ; zhao2022monovit ; bian2021ijcv_scdepth ; 2023planedepth ; 2021epcdepth ; manydepth , which avoid the need for ground-truth depth by enforcing photometric consistency across image pairs. While effective in favorable environments, such methods often fail under adverse conditions (e.g., nighttime, fog, or rain), where the photometric assumption breaks down. Another line of work focuses on building depth foundation models depth_anything_v2 ; depthanything ; marigold ; gui2024depthfm ; zoedepth ; birkl2023midas ; shao2023IEBins via supervised learning on large-scale datasets, aiming to enhance cross-domain generalization. However, their performance is limited by sparse and noisy annotations, which are common in outdoor adverse settings—where ground-truth depth (e.g., LiDAR) often covers only a fraction of the scene and suffers from artifacts like rain-induced reflections gasperini_morbitzer2023md4all ; nuscenes2019 . Consequently, the inability to extract reliable supervision from unlabeled adverse data, combined with the inefficiency of relying on imperfect annotations, renders robust monocular depth estimation (RMDE) under diverse real-world conditions an unsolved and practically important problem. To improve robustness under adverse conditions, several RMDE methods wang2021RNW ; diffusionadverse ; gasperini_morbitzer2023md4all ; Saunders_2023_ICCV ; wang2023weatherdepth ; diffusion_contrast ; syn2real-depth adopt unsupervised domain adaptation (UDA) frameworks to avoid reliance on real-world depth labels. Early efforts vankadari2020ADFA ; wang2021RNW ; spencer2020defeat ; 23ICRA_Steps target a single adverse scenario, such as nighttime, but often fail to generalize beyond their specific conditions. Subsequent works liu2021ADIDS ; vankadari2023sun attempt to bridge daytime and nighttime domains, yet struggle to handle the broader diversity of real-world environments. More recent approaches gasperini_morbitzer2023md4all ; Saunders_2023_ICCV ; diffusionadverse ; wang2023weatherdepth ; syn2real-depth aim to improve generalization across diverse weather conditions by leveraging generative augmentation CycleGAN2017 ; zheng_2020_forkgan ; LDM ; controlnet or synthetic-to-real domain adaptation. However, their reliance on synthetic data and limited exposure to diverse target domains hinders generalization to unseen real-world conditions. These trends raise a critical question: Is synthetic data truly necessary to achieve robust depth estimation under adverse conditions? Currently, Vision Foundation Models (VFMs) MAE ; clip ; EVA ; oquab2023dinov2 have emerged as a powerful paradigm, demonstrating strong generalization across tasks and domains through large-scale self-supervised pretraining on natural images. Recent works have dug into parameter-efficient fine-tuning (PEFT) of VFMs for tasks under challenging scenarios such as domain-generalized semantic segmentation Rein ; yun2024soma ; bi2024fada ; mfuser and domain-generalized object detection yun2024soma ; chen2022adaptformer , achieving promising results. Motivated by their robustness and transferability, we naturally explore whether VFMs can be efficiently adapted to RMDE, particularly under unseen adverse conditions. However, existing PEFT methods hu2022lora ; yun2024soma ; miLoRA ; task_specific_direction ; Rein ; chen2022adaptformer are mainly designed for semantic tasks due to the semantic nature of VFM. What’s more, they overlook the full singular spectrum, have not considered critical direction protection, and underutilize the rich directional information in the frozen weight matrices. On the contrary, RMDE relies on the low-level features instead of semantic information, which leads to sub-optimal results by simply utilizing previous approaches. In this work, we present ER-LoRA, a novel PEFT pipeline for robust monocular depth estimation, based on the guidance of effective rank. We begin by analyzing the frozen weight matrices of VFMs oquab2023dinov2 via singular value decomposition (SVD), revealing the distribution of singular vectors across different singular values; also, we find how these weights have changed through full fine-tuning, with the structure similarity between the frozen weight and the residual weight. Based on these insights, we design a Selecting, Tuning, and Maintaining (STM) strategy, aiming to adapt the semantically biased representations of VFMs toward low-level geometric reasoning while retaining their inherent robustness and generalization. Guided by the definition of effective ranks (e-ranks) entropyrank ; stablerank3 ; stablerank1 ; stablerank2 , we adaptively select proper rank numbers for different weight matrices, as well as finding the most task-relevant directions. This helps with better initialization. In the tuning phase, we only focus on training the low-rank branch; in the maitaining phase, we apply a component regularization to minimally perturb critical directions. Compared to prior PEFT methods, our approach provides a more flexible task adaptation while preserving the strong generalization capability of the pretrained VFM, which is helpful for adapting strong VFMs to tasks that are semantics-agnostic. In our experiments, we conduct evaluations in both self-supervised and supervised settings for RMDE, through training only on daytime data, and evaluate through multiple real-world datasets nuscenes2019 ; robotcar ; drivingstereo ; cadc of diverse adverse and normal conditions (e.g., daytime, nighttime, rain, fog, snow) in a zero-shot manner. The experimental results of both settings show that our method surpasses previous synthetic-data-based methods, PEFT approaches, full fine-tuning (FFT), and the MDE foundation model. To conclude, our contributions can be listed as follows: • To our knowledge, we are the first to propose learning weather-generalized RMDE via Parameter-Efficient Fine-Tuning of VFMs. We design ER-LoRA, a novel PEFT pipeline for RMDE, which eliminates the need for synthetic adverse data used in prior methods. • We leverage the definition of effective ranks to propose a novel Selecting, Tuning, and Maintaining (STM) strategy for PEFT, which gives flexible tuning space while retaining the pre-trained generalization ability and robustness compared with existing methods. • Experimental results under both self-supervised and supervised settings across various real-world datasets of diverse conditions demonstrate our method outperforms existing synthetic-data-based methods, PEFT methods, full fine-tuning and MDE foundation models, with an average

Read the complete text