Exploring Modality Guidance to Enhance VFM-based Feature Fusion for UDA in 3D Semantic Segmentation

Johannes Spoecklberger 1 Wei Lin 2 Pedro Hermosilla 3 Sivan Doveh 4 Horst Possegger 1 M. Jehanzeb Mirza 5 1 Institute of Visual Computing, Graz University of Technology 2 JKU LINZ 3 TU Wien 4 IBM Research 5 MIT CSAIL [email protected] · 2025

Read Exploring Modality Guidance to Enhance VFM-based Feature Fusion for UDA in 3D Semantic Segmentation by Johannes Spoecklberger 1 Wei Lin 2 Pedro Hermosilla 3 Sivan Doveh 4 Horst Possegger 1 M. Jehanzeb Mirza 5 1 Institute of Visual Computing, Graz University of Technology 2 JKU LINZ 3 TU Wien 4 IBM Research 5 MIT CSAIL [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14231.txt.

Introduction

Johannes Spoecklberger and co-authors use image foundation models to improve 3D scene labeling in a 2025 paper. Their task is to label each point in a LiDAR scan, adapting from a labeled source set to an unlabeled target set. Paired images carry rich information that a vision foundation model can supply across domains.

Their method trains a 3D backbone on a mix of labeled source and unlabeled target data. A fusion network joins the image and point cloud streams, guided by both. The balance between the streams adjusts to the target domain. In several test settings, the method beat leading rivals by clear margins. One result showed an average gain of 6.5 points in the main overlap score across tasks.

Catalog record

CREATORJohannes Spoecklberger 1 Wei Lin 2 Pedro Hermosilla 3 Sivan Doveh 4 Horst Possegger 1 M. Jehanzeb Mirza 5 1 Institute of Visual Computing, Graz University of Technology 2 JKU LINZ 3 TU Wien 4 IBM Research 5 MIT CSAIL [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14231

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploring Modality Guidance to Enhance VFM-based Feature Fusion for UDA in 3D Semantic Segmentation Johannes Spoecklberger 1 Wei Lin 2 Pedro Hermosilla 3 Sivan Doveh 4 Horst Possegger 1 M. Jehanzeb Mirza 5 1 Institute of Visual Computing, Graz University of Technology 2 JKU LINZ 3 TU Wien 4 IBM Research 5 MIT CSAIL [email protected] Abstract Vision Foundation Models (VFMs) have become a de facto choice for many downstream vision tasks, like image classification, image segmentation, and object localization. However, they can also provide significant utility for downstream 3D tasks that can leverage the cross-modal information ( e.g., from paired image data). In our work, we further explore the utility of VFMs for adapting from a labeled source to unlabeled target data for the task of LiDAR-based 3D semantic segmentation. Our method consumes paired 2D-3D (image and point cloud) data and relies on the robust (cross-domain) features from a VFM to train a 3D backbone on a mix of labeled source and unlabeled target data. At the heart of our method lies a fusion network that is guided by both the image and point cloud streams, with their relative contributions adjusted based on the target domain. We extensively compare our proposed methodology with different state-of-the-art methods in several settings and achieve strong performance gains. For example, achieving an average improvement of 6.5 mIoU (over all tasks), when compared with the previous state-of-the-art. 1 Introduction In an ideal world with abundant resources, we could potentially manually label all the distributions present in our visual world and train neural networks to perform robustly across diverse environments. The success of supervised training is well-known and has been extensively studied in the literature [ 16 , 50 , 29 ] . However, in the real world, the concept of abundance is usually non-existent. Instead, we face several constraints – monetary, time, or human resources, among others – that demand we employ all the available resources optimally. Efficiency and cost reduction become even more critical in domains like autonomous driving, where vehicles ideally need to operate safely in a range of environments ( i.e., data distributions). Labeling data for each of these distributions, particularly for dense prediction tasks like 3D semantic segmentation, can quickly become prohibitively expensive due to the need for per-point or per-voxel annotations. Figure 1 : (a) Cross modal learning with frozen 2D (VFM) backbone features using a learned fusion representation. Fusion networks can lead to a suboptimal feature utilization and unwanted modality bias on the target domain. Therefore, we propose regularizing the fusion by the most effective modality in a certain environment ( e.g., based on lighting conditions). (b) mIoU Comparison of xMUDA with different fusion variants and Ours on NuScenes: USA \to Singapore. 3D Unsupervised Domain Adaptation (UDA) offers a practicable solution by focusing on adapting a neural network trained in a supervised manner on a labeled source domain to an unlabeled target domain. This approach leverages information from annotated 3D (source) data to address the variability across domains without requiring manually annotated labels from each new target domain. This paper builds on the principles of 3D domain adaptation, aiming to bridge the gap between labeled and unlabeled 3D environments, particularly addressing challenges inherent in dense 3D prediction tasks, related to fine-grained understanding of the 3D scene. Ideally, an autonomous vehicle can employ cues from both the 2D and 3D data. Although different modalities, they can offer complementary cues that can enhance the perception abilities of autonomous driving systems. Jaritz et al. [ 19 ] proposed a seminal work on using the complementary information from the two modalities for the task of 3D domain adaptation for semantic segmentation. This method inspired a set of subsequent works [ 7 , 43 , 45 ] , which combine data from different domains on a similar multi-modal design. However, these approaches usually rely on training a 2D feature extractor separately, thus, requiring disjoint training of 2D and 3D backbones, increasing the computation cost significantly as compared to uni-modal setups. In our work, motivated by the recent strong improvements offered by vision foundation models (VFM) on dense prediction tasks [ 34 , 39 ] , we tackle the multi-modal 3D-UDA task by employing these powerful models. However, leveraging these VFMs ( e.g., [ 39 ] ) effectively for cross-modal learning for 3D UDA comes with the problem of optimally fusing the information from both modalities. To address this, at the heart of our approach lies a fusion refinement network, which fuses the information obtained from 2D and 3D modalities into a combined representation. Although such fusion schemes are studied in [ 18 , 24 ] , our experiments show that training these fusion methods can lead to an over-reliance on features that perform well on the source domain but fail to adapt to the target domain. Moreover, despite the growing popularity of multi-modal feature fusion, few works have focused on designing fusion modules that specifically address modality-specific challenges that for example arise under degrading weather and low light conditions [ 4 ] . To address this issue, we propose to adaptively regularize the fusion network based on a simple intuition: in some environments, the imaging modality may be more robust while in others, reliance on the 3D modality may be more beneficial. To this end, we guide the fusion refinement network through adaptive predictive distillation from each modality, based on environmental factors (e.g., lighting conditions). This regularization effectively promotes learning to select the modality to rely on, thereby enabling more robust multi-modal domain adaptation. An overview of our approach is also laid out in Figure 1 . We evaluated our proposal on the four common DA tasks, showing improved performance over traditional (fusion) methods, achieving SOTA or comparable performance on all tasks without requiring the training of an additional 2D encoder. Figure 2 : Our architecture for the cross-modal learning consists of a Vision Foundation Model (VFM) as the 2D encoder and a 3D SparseConvNet as the 3D encoder. We use multiple main task heads of semantic segmentation and mimicry task heads for cross-modal alignment. Besides the 2D branch (green) and 3D branch (blue), we employ a fusion branch (purple) where we concatenate the 2D and 3D feature as the fusion feature. The network is trained with the supervised loss \mathcal{L}_{\text{seg}} , and the self-supervised cross-modal learning losses \mathcal{L}_{\text{align}} (the fusion branch guiding the 3D branch) and \mathcal{L}_{\text{guide}} (the fusion branch guided by the 2D or 3D branch). 2 Related Work Our method is related to works that study UDA for 3D semantic segmentation, VFMs, and approaches that propose different fusion schemes for cross-modal learning. UDA for 3D Semantic Segmentation. UDA for 3D semantic segmentation has been extensively studied in recent years. A major group of methods can be categorized as learning domain-invariant feature representations. Earlier works in 3D UDA focused on minimizing statistical feature discrepancies [ 42 , 33 ] . Next, adversarial training is another widely used approach to learn invariant representations [ 13 , 2 , 49 ] . Another line of research utilizes self-supervision to learn more domain invariant representations by constructing a label-free auxiliary optimization goal that is often modality specific [ 48 , 31 ] . Domain mapping

Read the complete text