SAS: Segment Any 3D Scene with Integrated 2D Priors

Zhuoyuan Li, Jiahao Lu, Jiacheng Deng, Hanzhi Chang, Lifan Wu, Yanzhe Liang, Tianzhu Zhang · 2025

Read SAS: Segment Any 3D Scene with Integrated 2D Priors by Zhuoyuan Li, Jiahao Lu, Jiacheng Deng, Hanzhi Chang, Lifan Wu, Yanzhe Liang, Tianzhu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08512.txt.

Introduction

Zhuoyuan Li and co-authors pour the open-vocabulary skill of image models into three-dimensional scenes in a 2025 paper. A 3D segmenter trained on fixed categories goes blind before anything new. Flat image models, by contrast, can name almost anything from text. The trick is moving that fluency into point clouds without hand labels.

Their system first aligns several image models into one shared space, using text as the bridge. It then measures, with help from diffusion models, how well each model knows each category, no annotations needed. Point features from the models fuse under that capability map, and the combined skill distills into a 3D network. Across indoor and driving datasets the method beat predecessors by wide margins, and the transferred skill held up on finer tasks like instance segmentation.

Catalog record

CREATORZhuoyuan Li, Jiahao Lu, Jiacheng Deng, Hanzhi Chang, Lifan Wu, Yanzhe Liang, Tianzhu Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08512

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAS: Segment Any 3D Scene with Integrated 2D Priors Zhuoyuan Li, Jiahao Lu, Jiacheng Deng, Hanzhi Chang, Lifan Wu, Yanzhe Liang, Tianzhu Zhang Abstract The open vocabulary capability of 3D models is increasingly valued, as traditional methods with models trained with fixed categories fail to recognize unseen objects in complex dynamic 3D scenes. In this paper, we propose a simple yet effective approach, SAS , to integrate the open vocabulary capability of multiple 2D models and migrate it to 3D domain. Specifically, we first propose Model Alignment via Text to map different 2D models into the same embedding space using text as a bridge. Then, we propose Annotation-Free Model Capability Construction to explicitly quantify the 2D model’s capability of recognizing different categories using diffusion models. Following this, point cloud features from different 2D models are fused with the guide of constructed model capabilities. Finally, the integrated 2D open vocabulary capability is transferred to 3D domain through feature distillation. SAS outperforms previous methods by a large margin across multiple datasets, including ScanNet v2, Matterport3D, and nuScenes, while its generalizability is further validated on downstream tasks, e.g., gaussian segmentation and instance segmentation. Figure 1 : Left : The leading 2D open vocabulary models like LSeg [ 39 ] and SEEM [ 96 ] often misidentify objects, which makes the distilled 3D model perform the same misidentification. Middle : Our proposed SAS successfully correct the misidentified object. Right : SAS distills open vocabulary knowledge from multiple 2D models with novel designs, e.g., Annotation-free Model Capability Construction. 1 Introduction 3D scene understanding is a fundamental task that aims to predict the semantics for every 3D point. Numerous real-world applications, such as autonomous driving [ 1 , 19 , 17 ] , virtual reality [ 52 ] , and robot manipulation [ 65 ] , depend heavily on 3D scene understanding. Traditional methods in this area perform supervised training on labeled 3D datasets with a limited number of categories, which hinders the model’s ability to identify unseen objects. This prompts researchers to turn their attention to open vocabulary capabilities of 3D scene understanding models. In 2D open vocabulary understanding, a typical approach is to align image features to language features by contrast learning on a large number of image-text pairs, e.g., CLIP [ 60 ] . By analogy, contrast learning on a large number of point-text pairs is supposed to enable 3D open vocabulary understanding. However, point-text pairs are much harder to acquire than image-text pairs due to the high cost and time-consuming nature of point cloud annotation [ 15 , 53 ] . A compromise approach is to distill the open vocabulary capability of 2D open vocabulary models onto the 3D models [ 15 , 84 , 53 , 5 , 88 , 76 , 94 , 27 ] . For example, OpenScene [ 53 ] adopts 2D open vocabulary segmentaters to extract pixel features for point clouds to enable point-language alignment. These pixel features are distilled to transfer the 2D open vocabulary capability to 3D models. Although distillation-based methods have become dominant in 3D scene understanding, the distilled model inherited its 3D open vocabulary capability from the adopted 2D models, making the performance of the distilled model vary with the performance of the adopted 2D model. As shown in Fig. 1 , the 2D open vocabulary model may incorrectly recognize objects in the scene, which generates ambiguous supervision signals for the 3D model so that the 3D model will make the same mistake. For example, SEEM [ 96 ] misidentifies a picture as a wall (left), making the distilled model perform the same misidentification (left). Therefore, an intuitive idea is to integrate the capabilities of different 2D models to correct misidentified objects. However, there are two inherent difficulties in integrating the capability of different 2D models. First, different 2D models have unaligned image-text feature spaces. The unaligned features cannot be directly integrated. Second, testing the model on a batch of images reflects the capability of this model. However, since 3D open vocabulary tasks are inherently designed to recognize unseen categories in a zero-shot fashion, obtaining test images and their annotations is challenging. As a result, it is impractical to directly evaluate the model’s performance. To address the above problems, we propose SAS , to learn better 3D representations from the integration of multiple 2D models. Our key idea is to explicitly model 2D models’ capability of identifying different categories in the scene, and use this to guide the integration of different 2D models, which produces better 3D representations than a single 2D model can do. First, we propose Model Alignment via Text to align features of different 2D models using text as a bridge. Specifically, a 2D open vocabulary model, e.g., SEEM model [ 96 ] , predicts a label for each pixel (or mask), based on which we further generate a caption containing more complex information such as color, shape, etc., for each pixel (or mask), which enriches the semantic information and intra-class diversity. Now, different 2D open vocabulary models are aligned at the text-level. The caption is then inputted into a shared text encoder to get the aligned features. Second, we propose Annotation-free Model Capability Construction to quantitatively measure the capability of different 2D open vocabulary models. Specifically, it leverages the text-to-image diffusion model [ 62 ] to synthesize images of common types. We utilize the 2D open vocabulary model’s performance on these synthesized images to quantify the model’s capabilities, which later can guide the integration of different 2D models. Finally, SAS transfers the integrated open vocabulary capability of 2D models to 3D domain via feature distillation. Figure 2 : Overview of our proposed SAS. SAS first align features of different models in a unified embedding space (Sec. 3.1 ). Then SAS constructs models’ capability to recognize various objects (Sec. 3.2 ). With the constructed capability as guide, features from different 2D models are integrated (Sec. 3.3 ). Finally, a 3D network is distilled to enable 3D open vocabulary understanding (Sec. 3.4 ). We evaluate SAS on multiple datasets, including ScanNet v2 [ 9 ] , Matterport3D [ 3 ] and nuScenes [ 1 ] . Overall, we make the following contributions: • We propose SAS, the first approach to learn better 3D representations from multiple 2D models. • We propose Model Alignment via Text and Annotation-free Model-capability Construction to address the aforementioned difficulties in integrating different 2D models. The former aligns the features of models with different embedding spaces, while the latter explicitly models the model capability via synthetic images, which is then used to guide the integration of different model features. • We validate SAS on different 3D scene understanding tasks, including semantic segmentation, instance segmentation, and gaussian segmentation. The results show that SAS significantly outperforms previous methods while preserving strong generalizability. 2 Related Work Closed-set 3D Scene Understanding. Recent work on closed-set 3D scene understanding has seen significant advancement, spanning tasks such as 3D semantic segmentation [ 57 , 55 , 35 , 90 , 81 , 42 ] , object detection [ 93 , 36 , 51 , 66 , 11 , 33 ] , instance segmentation [ 25 , 73 , 68 , 46 , 34

Read the complete text