GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

Xujing Tao Chuxin Wang Yubo Ai Zhixin Cheng Zhuoyuan Li, Liangsheng Liu Yujia Chen Xinjun Li Qiao Li Wenfei Yang Tianzhu Zhang · 2026

Read GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation by Xujing Tao Chuxin Wang Yubo Ai Zhixin Cheng Zhuoyuan Li, Liangsheng Liu Yujia Chen Xinjun Li Qiao Li Wenfei Yang Tianzhu Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26260.txt.

Introduction

Tao and co-authors present GeoGuide, a method for labelling points in 3D scenes with open-ended category names. Most methods lean on 2D models, which can pass on errors and limit true 3D learning. GeoGuide uses pretrained 3D models to keep geometry and meaning in step. One module weighs 2D features by doubt inside superpoints to cut noise. Other modules rebuild full instance masks and align links between objects. Tests on ScanNet v2, Matterport3D, and nuScenes show strong results.

Catalog record

CREATORXujing Tao Chuxin Wang Yubo Ai Zhixin Cheng Zhuoyuan Li, Liangsheng Liu Yujia Chen Xinjun Li Qiao Li Wenfei Yang Tianzhu Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39033
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation Xujing Tao Chuxin Wang Yubo Ai Zhixin Cheng Zhuoyuan Li, Liangsheng Liu Yujia Chen Xinjun Li Qiao Li Wenfei Yang Tianzhu Zhang Abstract Open-vocabulary 3D semantic segmentation aims to segment arbitrary categories beyond the training set. Existing methods predominantly rely on distilling knowledge from 2D open-vocabulary models. However, aligning 3D features to the 2D representation space restricts intrinsic 3D geometric learning and inherits errors from 2D predictions. To address these limitations, we propose GeoGuide , a novel framework that leverages pretrained 3D models to integrate hierarchical geometry-semantic consistency for open-vocabulary 3D segmentation. Specifically, we introduce an Uncertainty-based Superpoint Distillation module to fuse geometric and semantic features for estimating per-point uncertainty, adaptively weighting 2D features within superpoints to suppress noise while preserving discriminative information to enhance local semantic consistency. Furthermore, our Instance-level Mask Reconstruction module leverages geometric priors to enforce semantic consistency within instances by reconstructing complete instance masks. Additionally, our Inter-Instance Relation Consistency module aligns geometric and semantic similarity matrices to calibrate cross-instance consistency for same-category objects, mitigating viewpoint-induced semantic drift. Extensive experiments on ScanNet v2, Matterport3D, and nuScenes demonstrate the superior performance of GeoGuide. 1 Introduction 3D scene understanding is critical for applications such as autonomous driving 23 ; 2 and embodied intelligence 61 . As a core subtask, point cloud semantic segmentation aims to predict a category label for each point in 3D space. While fully-supervised methods have achieved great success, they rely on densely annotated 3D datasets and inherently fail to handle unseen categories in real-world scenarios. Figure 1 : (a) Occlusions and ambiguous boundaries in 2D predictions lead to inaccurate 3D segmentation, whereas our geometry-guided modeling effectively rectifies these errors. (b) The baseline suffers from the lack of geometric relation modeling, whereas our method exploits inter-instance geometric constraints to produce accurate predictions. To address this limitation, 3D open-vocabulary semantic segmentation has attracted significant attention, enabling models to segment 3D scenes based on arbitrary natural language descriptions. However, unlike 2D open-vocabulary tasks that benefit from abundant large-scale image-text pairs 60 ; 13 , 3D point-text paired data remain scarce due to prohibitive annotation costs 22 ; 35 ; 10 ; 11 ; 9 . Consequently, recent works tackle this challenge by transferring knowledge from pre-trained 2D open-vocabulary models. These methods generally follow two paradigms. (1) 2D-to-3D distillation 57 ; 42 ; 69 ; 38 ; 81 ; 12 ; 6 ; 8 ; 7 ; 14 , which projects pixel-level semantic features onto point clouds through geometric correspondences and aligns 3D representations with 2D features 36 ; 24 via knowledge distillation. (2) Point–text alignment 29 ; 22 ; 35 ; 77 ; 18 ; 44 ; 5 ; 39 ; 45 ; 40 , which generates textual descriptions from images, constructs point–text pairs using geometric correspondences, and aligns 3D features with textual embeddings through contrastive learning. Essentially, both paradigms train the 3D model to replicate the feature representations of 2D models, which restricts the learning of intrinsic 3D geometric structures. It is worth noting that 2D models are prone to producing erroneous object masks due to occlusions and viewpoint changes, causing the 3D model to inherit these errors and learn incorrect segmentation patterns, as shown in Figure 1 . Based on the above analysis, we argue that the core limitation of existing open-vocabulary 3D segmentation methods lies in how to effectively preserve intrinsic 3D geometric information during the 2D-to-3D knowledge distillation process. A natural idea is to leverage pretrained 3D models to correct the geometric biases from 2D representations. However, directly incorporating pretrained 3D features does not necessarily improve performance, as the heterogeneous supervision signals between different modalities introduce instability during training. To address this challenge, we identify three key issues that must be resolved to achieve stable and geometry-consistent open-vocabulary 3D segmentation. (1) Intra-Superpoint Consistency Modeling. Superpoints 56 , defined as geometrically homogeneous neighborhoods, should inherently share the same semantic label. However, when projecting 2D predictions onto 3D, semantics within a single superpoint often become inconsistent, particularly due to 2D-level occlusions or boundary ambiguities. Existing approaches typically enforce consistency through mean pooling, which suppresses discriminative features and is overly sensitive to prediction noise, thereby amplifying deviations from the correct semantic representation. Hence, maintaining reliable intra-superpoint semantic consistency remains a fundamental challenge. (2) Intra-Instance Consistency Modeling. Predictions from a single 2D view are inherently incomplete, capturing only partial masks that fail to represent the full 3D geometry of each instance. This incompleteness leads to fragmented instance-level semantics in 3D space. Thus, achieving consistent semantics across all parts of an instance and reconstructing its complete structure remains a key challenge in 2D-to-3D distillation. (3) Inter-Instance Relation Consistency Modeling. Aggregating 2D features from multiple views introduces feature distribution shifts among same-category instances due to viewpoint variations. As a result, instances of the same class may exhibit divergent feature representations, leading to semantic inconsistency across the scene. Therefore, modeling geometric relation constraints to ensure consistent semantics among same-category instances poses another open challenge. To tackle these challenges, we propose a hierarchical geometry-guided framework that explicitly integrates geometric priors at multiple levels to achieve fine-grained geometry–semantic alignment. The framework consists of three key modules: the Uncertainty-based Superpoint Distillation (USD) module, the Instance-level Mask Reconstruction (IMR) module, and the Inter-Instance Relation Consistency (IIRC) module. In the USD module , we fuse geometric features from a pretrained 3D backbone with 2D semantic features to estimate per-point feature uncertainty. This uncertainty guides weighted aggregation within each superpoint to highlight discriminative features while suppressing noise. Through superpoint-level 2D-to-3D feature distillation, this module enforces robust intra-superpoint semantic consistency. In the IMR module , we generate class-agnostic instance proposals following 79 and reconstruct complete instance masks to recover missing regions, achieving intra-instance consistency by unifying the semantics of all points within an instance. In the IIRC module , we exploit the observation that pretrained 3D models produce similar geometric representations for same-category objects. By aligning geometric and semantic similarity matrices across instances, we ensure inter-instance consistency, mitigating semantic drift caused by viewpoint variation. This relation consistency is jointly enforced at both the superpoint and instance levels, forming a local-to-global semantic alignment mechanism. Our main contributions are summarized as follows: (1) We identify the limitations of conventional 2D feature distillation in open-vocabulary 3D segmentation and present a novel framework that leverages pretrained 3D models to integrate hierarchical geometric priors for enhanced geometric awareness. (2) We design three complementary modules that enforce geometry–semantic consistency at the intra-superpoint, intra-instance, and inter-instance levels, achieving robust local-to-global semantic alignment. (3) Extensive experiments on multiple benchmarks demonstrate that our method achieves state-of-the-art performance and exhibits strong cross-domain generalization, while comprehensive ablation studies validate the effectiveness of each proposed component. 2 Related Work 2.1 Closed-Set 3D Scene Understanding 3D scene understanding underpins applications in autonomous driving, embodied intelligence, and robotic navigation. Significant progress has been made in various 3D vision tasks, including classification 4 ; 58 ; 41 ; 21 , semantic segmentation 59 ; 74 ; 20 ; 33 ; 83 , object detection 34 ; 54 ; 84 ; 19 ; 67 ; 66 ; 51 ; 68 ; 48 , and instance segmentation 30 ; 31 ; 65 ; 47 ; 50

Read the complete text