Learning A Zero-shot Occupancy Network from Vision Foundation Models via Self-supervised Adaptation
Sihao Lin, Daqi Liu, Ruochong Fu, Dongrui Liu, Andy Song, Hongwei Xie, Bing Wang, Xiaojun Chang · 2025
Read Learning A Zero-shot Occupancy Network from Vision Foundation Models via Self-supervised Adaptation by Sihao Lin, Daqi Liu, Ruochong Fu, Dongrui Liu, Andy Song, Hongwei Xie, Bing Wang, Xiaojun Chang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07125.txt.
Introduction
Sihao Lin and co-authors teach a 3D occupancy model without 3D labels in a 2025 paper. Labeling space in three dimensions by hand is slow and costly. Big image models already grasp scene meaning and rough depth. Their trick converts that rough depth into true scale using time.
Views of the same scene across moments pin down scale and shift. Meaning labels then lift into space along the corrected depth. The model learns which voxels are filled without any 3D ground truth. Road scene tests beat the prior best by a clear margin.
Catalog record
| CREATOR | Sihao Lin, Daqi Liu, Ruochong Fu, Dongrui Liu, Andy Song, Hongwei Xie, Bing Wang, Xiaojun Chang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07125
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning A Zero-shot Occupancy Network from Vision Foundation Models via Self-supervised Adaptation Sihao Lin, Daqi Liu, Ruochong Fu, Dongrui Liu, Andy Song, Hongwei Xie, Bing Wang, Xiaojun Chang Abstract Estimating the 3D world from 2D monocular images is a fundamental yet challenging task due to the labour-intensive nature of 3D annotations. To simplify label acquisition, this work proposes a novel approach that bridges 2D vision foundation models (VFMs) with 3D tasks by decoupling 3D supervision into an ensemble of image-level primitives, e.g., semantic and geometric components. As a key motivator, we leverage the zero-shot capabilities of vision-language models for image semantics. However, due to the notorious ill-posed problem - multiple distinct 3D scenes can produce identical 2D projections, directly inferring metric depth from a monocular image in a zero-shot manner is unsuitable. In contrast, 2D VFMs provide promising sources of relative depth, which theoretically aligns with metric depth when properly scaled and offset. Thus, we adapt the relative depth derived from VFMs into metric depth by optimising the scale and offset using temporal consistency, also known as novel view synthesis, without access to ground-truth metric depth. Consequently, we project the semantics into 3D space using the reconstructed metric depth, thereby providing 3D supervision. Extensive experiments on nuScenes and SemanticKITTI demonstrate the effectiveness of our framework. For instance, the proposed method surpasses the current state-of-the-art by 3.34% mIoU on nuScenes for voxel occupancy prediction. 1 Introduction The vision-centric paradigm Ma et al. 2024 ; Izadinia et al. 2017 ; Saxena et al. 2005 aims to understand the 3D environment using RGB images. It has witnessed great success in autonomous driving (AD) for versatile applications such as object detection Huang et al. 2021 ; Wang et al. 2021 and occupancy prediction Vobecky et al. 2024 ; Huang et al. 2024 . Nonetheless, the labour-intensive annotation process Behley et al. 2019 ; Caesar et al. 2020 ; Tian et al. 2024 has long been the Achilles’ heel in vision-centric AD systems. On one hand, we aim to train a 3D driving system from effortless 2D data. However, many camera sensor-based car models can only capture plaint images but not semantic and geometric priors, posing challenges to 3D annotations. To address this issue, this work strives to build a vision-centric 3D occupancy network using an image-only dataset. Figure 1: Adapting relative depth into metric depth . Existing VFM Yang et al. 2024a delivers promising relative depth while less capable of metric depth due to ill-posedness. Without access to ground truth, our method leverages the novel view synthesis ( \cref sec:scale) to calibrate the relative depth (zoom out if necessary) into metric one , which aligns well with ground truth depth . As a motivator, one may resort to 3D vision foundation models (VFMs) Zhou et al. 2023 ; Xu et al. 2023 ; Yang et al. 2023 ; Hong et al. 2023 ; Chen et al. 2023 to facilitate 3D annotations. Unfortunately, existing literature on 3D VFMs necessitates 3D input ( e.g . \xspace , point cloud) and deviates from the data sources of vision-centric systems. On the other hand, training a vision-centric variant of 3D VFMs from scratch leads to a chicken-and-egg problem. In parallel, the emergence of 2D VFMs, such as SAM Kirillov et al. 2023 and DINOv2 Oquab et al. 2023 , has empowered a series of 2D vision tasks Minderer et al. 2024 ; Li et al. 2022a ; Zhang et al. 2023a ; Peng et al. 2023 ; Kerr et al. 2023 , demonstrating a great ability to ease the acquisition of image-level primitives. Therefore, it is tempting to exploit the zero-shot image primitives derived from ready 2D VFMs for 3D tasks, which remains largely untapped. By definition, the spatial grids geometrically correspond to the image patches, which can be formulated by pixel-wise depth and camera pose. We therefore motivate connecting the 2D VFMs with 3D tasks by decoupling the 3D supervision signals into image-level semantics and geometry independently. Specifically, given an RGB image dataset without any semantic or geometric labels, we apply vision foundation models on the monocular image to obtain the pixel-wise semantic and depth primitives separately. Accordingly, we project the semantics into the 3D space using the depth information and camera parameters, yielding an alternative to 3D supervision (See Fig. 2 ). Despite decade-long research Saxena et al. 2005 ; Wei et al. 2023 ; Yang et al. 2024a , current depth estimation methods are less capable of metric depth in unseen domains, primarily due to the notorious ill-posedness. That is, different 3D scenes can lead to identical 2D projections. Thus, two images that appear identical may have different depth information, making it challenging to zero-shot metric depth. While some efforts Guizilini et al. 2023 ; Yin et al. 2023 ; Radford et al. 2018 have been devoted to zero-shot metric depth, we observe they are less capable of occupancy prediction. Existing VFMs Yang et al. 2024a ; Yang et al. 2024b offer appealing sources of relative depth in a zero-shot fashion, which can be calibrated to metric depth with appropriate scale and offset. Inspired by this, we propose to prompt the relative depth derived from VFMs into metric depth by optimising the scale and offset without access to ground-truth depth. Specifically, we leverage novel view synthesis (NVS) among video sequences, optimise the scale and offset to minimise the reconstruction error (See \cref fig:nvs). Zhang et al . \xspace Zhang et al. 2022 noted that recovering metric depth requires different scales and offsets for each image pixel, making optimisation exponentially difficult. Further, we found that naively applying gradient descent Kingma and Ba 2014 ; Loshchilov and Hutter 2019 leads to divergence due to sensitive initialisation. To this end, we develop a two-stage coarse-to-fine optimisation strategy. In the first stage, we traverse an array of candidate scales (single scalar) applied to the whole scene and select the one with minimum reconstruction error. In the next stage, we initialise pixel-wise scales with the holistic scene scale from the previous stage and optimise them, together with a scene offset, via gradient descent. We evaluate the proposed framework for 3D occupancy prediction on nuScenes Caesar et al. 2020 and SemanticKITTI Behley et al. 2019 , which requires classifying the voxel semantic of the 3D volume in the driving environment. Occupancy prediction is an important task for autonomous driving because it can serve as a proxy for scene understanding given the granularity. Our framework can surpass, sometimes strikingly, the existing methods Huang et al. 2024 ; Cao and de Charette 2023 with bird’s eye view (BEV) representation. For instance, our method outperforms SelfOcc Huang et al. 2024 by 3.34% mIoU in nuScenes. Our work unveils another direction for vision-centric 3D tasks by exploiting the 2D VFMs. We hope our findings can inspire other dimensions of 3D visions. To summarise, the contributions of this work are: • We present the first vision-centric framework to connect the image primitives derived from 2D VFMs with 3D tasks, enabling zero-shot learning for 3D occupancy prediction with effortless label-free images. • We propose a two-stage coarse-to-fine optimisation process to calibrate the high-fidelity relative depth of VFM into metric depth via temporary consistency where no ground truth