Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding

Jinlong Li Cristiano Saltori Fabio Poiesi Nicu Sebe · 2025

Read Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding by Jinlong Li Cristiano Saltori Fabio Poiesi Nicu Sebe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16707.txt.

Introduction

Jinlong Li, Cristiano Saltori, Fabio Poiesi, and Nicu Sebe present CUA-O3D, a model for open-vocabulary understanding of three-dimensional scenes, in a two thousand twenty-five computer vision paper. Open-vocabulary means the model can label objects described in free text, not just fixed categories.

Instead of relying on one vision-language model, CUA-O3D distills knowledge from CLIP, DINOv2, and Stable Diffusion together. A deterministic uncertainty estimate weighs each model's features, balancing semantic priors against geometric knowledge. Experiments on ScanNetV2 and Matterport3D show gains in open-vocabulary segmentation plus strong cross-domain alignment and spatial perception.

Catalog record

CREATORJinlong Li Cristiano Saltori Fabio Poiesi Nicu Sebe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16707

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding Jinlong Li Cristiano Saltori Fabio Poiesi Nicu Sebe Abstract The lack of a large-scale 3D-text corpus has led recent works to distill open-vocabulary knowledge from vision-language models (VLMs). However, these methods typically rely on a single VLM to align the feature spaces of 3D models within a common language space, which limits the potential of 3D models to leverage the diverse spatial and semantic capabilities encapsulated in various foundation models. In this paper, we propose Cross-modal and Uncertainty-aware Agglomeration for Open-vocabulary 3D Scene Understanding dubbed CUA-O3D , the first model to integrate multiple foundation models—such as CLIP, DINOv2, and Stable Diffusion—into 3D scene understanding. We further introduce a deterministic uncertainty estimation to adaptively distill and harmonize the heterogeneous 2D feature embeddings from these models. Our method addresses two key challenges: (1) incorporating semantic priors from VLMs alongside the geometric knowledge of spatially-aware vision foundation models, and (2) using a novel deterministic uncertainty estimation to capture model-specific uncertainties across diverse semantic and geometric sensitivities, helping to reconcile heterogeneous representations during training. Extensive experiments on ScanNetV2 and Matterport3D demonstrate that our method not only advances open-vocabulary segmentation but also achieves robust cross-domain alignment and competitive spatial perception capabilities. Project webpage: CUA-O3D . † † † Corresponding author: [email protected]. 1 Introduction 3D scene understanding serves as a crucial perception component for a wide array of real-world applications to help models better understand the physical world, including robot navigation, autonomous vehicles, and virtual reality Graham et al. 2018 ; Vu et al. 2022 ; Li and Ibanez-Guzman 2020 ; Behley et al. 2019 . Typical approaches necessitate a dataset of semantically annotated point clouds, which is both time-consuming ( e.g. , 22.3 minutes for annotating a single scene with 20 classes Dai et al. 2017 ) and unable to encompass all possible existing categories, thereby limiting their practical utility. To address this limitation, recent works have explored the open-vocabulary 3D scene understanding setting Jatavallabhula et al. 2023 ; Peng et al. 2023 ; Ding et al. 2023a ; Mei et al. 2024 ; Mei et al. 2025 , aiming to localize and recognize arbitrary object classes. This objective is achieved by leveraging Vision-Language Models (VLMs) Radford et al. 2021 ; Jia et al. 2021 that, having been pre-trained on billions of image-text pairs Schuhmann et al. 2022 , can gauge the alignment between textual and visual inputs, facilitating a range of 2D open-vocabulary tasks Gu et al. 2021 ; Zhong et al. 2022 ; Boyi et al. 2022 ; Ghiasi et al. 2022 ; Ko et al. 2023 . To adapt these models for 3D downstream tasks ( e.g., semantic segmentation), various strategies, predominantly based on distilling multi-view 2D visual features into a 3D-specific model Ha and Song 2022 ; Jatavallabhula et al. 2023 ; Peng et al. 2023 ; Ding et al. 2023a ; Xu et al. 2024 , have been employed. Figure 1 : Top is feature distribution analysis of different 2d projected feature embeddings from various foundation models (Lseg, DINOv2 and Stable Diffusion), enumerating on the overall ScanNetV2 train set and counting the frequency of all point features within each bin interval. Bottom is the sample utilizing K-Means to cluster projected 3D features into specified clusters to make segmentation comparisons. Different foundation models illustrate heterogeneous yet complementary results. However, these methods mainly focus on distilling knowledge from one single VLM which limits the potential of 3D models to leverage the diverse spatial and semantic capabilities that have been trained on large-scale images or image-text pairs corpus, as shown in Table 1 . Given the existence of various foundation models, such as CLIP Radford et al. 2021 , DINOv2 Oquab et al. 2023 and Stable Diffusion Rombach et al. 2022 , etc. , there has been under-explored on how to make better utility of these 2D foundation models to develop the 3D foundation model, wherein no large-scale 3D point-cloud or 3D-text pair corpus available. Very recently, some works El Banani et al. 2024 ; Man et al. 2024 have started probing the potential of these 2D foundation models on 3D tasks, since different VLMs or visual foundation models showcase unique characteristics. Probe3D El Banani et al. 2024 posits the 3D awareness of some visual foundation models, like DINOv2 present well for depth and surface normals. Lexicon3D Man et al. 2024 finds diffusion models benefit geometric tasks. Nonetheless, there is still a lack of studies on how to aggregate these heterogeneous foundation models into the 3D model which naturally excels at geometric knowledge extraction and locating 3D spatial objects. To explore various foundation model properties when fusing multi-view posed image features to 3D space and enabling model distillation, we first conduct a pilot study to analyze and compare the heterogeneous results shown in Fig. 1 . We can observe that the distribution in terms of the feature embeddings from each 2D foundation model mainly follows a gaussian-like feature distribution. At the same time, when clustering these features into specific clusters, the results illustrate heterogeneous and complementary effects across different foundation models. This is also due to the inconsistency across different posed images when the 2D model encounters complex image contexts. This motivates us to develop a new method to harmonize this heterogeneous knowledge into a 3D model and handle such noisy inconsistency from the fused 2D feature embeddings. Table 1 : Comparison of Vision Foundation Models. Although all utilize the same Vision Transformer (ViT) backbone, they greatly differ in their training paradigms, including data, image resolutions, and training objectives, which lead to diverse representation biases. Model Training Dataset Dataset Size Architecture Objective ViT ( Dosovitskiy 2020 ) ImageNet-1k/21k 1.2M/14.2M ViT-B/L/G Supervised classification DINOv2 ( Oquab et al. 2023 ) LVD-142M 142M ViT-L/14 Discriminative self-supervised learning CLIP ( Radford et al. 2021 ) WebImageText 400M ViT-L/14 Image-text contrastive learning Stable Diffusion ( Rombach et al. 2022 ) LAION 5B UNet Image-Text/Image Generation In this paper, we present C ross-modal and U ncertainty-aware A gglomeration for O pen-vocabulary 3D Scene Understanding, dubbed CUA-O3D , the first method to integrate multiple 2D foundation models into one 3D model for scene understanding. Deterministic uncertainty estimation is further introduced to adaptively distill and harmonize the heterogeneous 2D feature embeddings from these models. We show that there are potential inconsistencies yet complementarity when attempting to distill different foundation models. Based on our pilot study, we first propose to leverage distillation loss to supervise 3D model training given several available 2D feature embeddings. Our 3D model consists of independent projection layers to be mapped with one VLM or visual foundation model under feature supervision which helps with reconciling the entanglements from heterogeneous distributions. To resolve the potential noises from 2D models which mainly come from insufficient contexts, a novel deterministic uncertainty estimation is tailored to adaptively weight the knowledge distillation which can be modeled like a gaussian likelihood that follows the distributions as shown at the top of Fig. 1 . Specifically, regarding each projection layer to be mapped with the specific 2D model, we tailor an observation noise scalar prediction independently to capture how much the noise is contained in feature supervision during training, which is termed uncertainty-aware learning. Moving

Read the complete text