SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation
Chris Choy, Junha Lee, Chunghyun Park, Minsu Cho, Jan Kautz · 2026
Read SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation by Chris Choy, Junha Lee, Chunghyun Park, Minsu Cho, Jan Kautz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20395.txt.
Introduction
Choy and four colleagues present SpaCeFormer, a proposal-free transformer for open-vocabulary 3D instance segmentation that runs in 0.12 to 0.30 seconds per scene, far faster than multi-stage pipelines. They pair it with SpaCeFormer-3M, a dataset of three million multi-view-consistent captions over 604,000 instances from 7,400 scenes. The model pairs spatial window attention with Morton-curve ordering and predicts masks straight from learned queries. On ScanNet200 it reaches a zero-shot score of about eleven, nearly three times the prior best proposal-free result.
Catalog record
| CREATOR | Chris Choy, Junha Lee, Chunghyun Park, Minsu Cho, Jan Kautz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104295 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.20395
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation Chris Choy, Junha Lee, Chunghyun Park, Minsu Cho, Jan Kautz Abstract Open-vocabulary 3D instance segmentation is foundational for robotics and AR/VR, yet prior methods trade one bottleneck for another: multi-stage 2D+3D pipelines aggregate foundation-model outputs at hundreds of seconds per scene, while pseudo-labeled end-to-end approaches rely on fragmented masks and external region proposals. We present SpaCeFormer, a proposal-free space-curve transformer that runs at 0.14 seconds per scene—2–3 orders of magnitude faster than multi-stage 2D+3D pipelines—paired with SpaCeFormer-3M, the largest open-vocabulary 3D instance segmentation dataset (3.0M multi-view-consistent captions over 604K instances from 7.4K scenes) with 21 × \times higher mask recall than prior single-view pseudo-label pipelines (54.3% vs 2.5% at IoU > > 0.5), built via multi-view mask clustering and multi-view VLM captioning. SpaCeFormer combines spatial window attention with Morton-curve serialization for spatially coherent features, and uses a RoPE-enhanced decoder to predict instance masks directly from learned queries without external proposals. On ScanNet200, we achieve 11.1 zero-shot mAP, a 2.8 × \times improvement over the prior best proposal-free method; on ScanNet++ and Replica, we reach 22.9 and 24.1 mAP, surpassing all prior methods including those using multi-view 2D inputs. 1 Introduction Figure 1 : Accuracy vs. latency on Replica zero-shot open-vocabulary 3D instance segmentation. SpaCeFormer ( red star ) reaches 24.1 mAP at 0.14 s per scene —Pareto-optimal among methods that use no GT 3D supervision, and 2–3 orders of magnitude faster than multi-stage 2D+3D pipelines while using only 3D input (no 2D RGB-D streams, no external region proposals). SOLE is the only method above us on mAP (24.7), but requires ScanNet200 ground-truth mask supervision and is \sim 9 \times slower. SpaCeFormer is the only method that enters the “interactive” band ( < 1 s, shaded). Latency on a log scale; see Table 3 for full numbers. † SOLE’s latency is estimated from its Mask3D backbone. Understanding 3D scenes with open-vocabulary descriptions is fundamental for autonomous driving ( Zhang et al. 2024 ) , augmented reality ( Guo et al. 2024 ) , and embodied AI ( Rashid et al. 2023 ; Driess et al. 2023 ) . Unlike closed-set 3D segmentation ( Qi et al. 2017 ; Choy et al. 2019 ; Schult et al. 2023 ) that assumes a fixed label space, open-vocabulary 3D segmentation aims to identify and segment arbitrary objects described in natural language ( Peng et al. 2023 ) . This capability is crucial in real-world environments where the long-tail of object types far exceeds any predefined taxonomy ( Liu et al. 2019 ) . While end-to-end 3D approaches would be ideal, the lack of large-scale 3D-text paired data remains a fundamental barrier for training open-vocabulary 3D instance segmentation models ( Yang et al. 2024a ) . This challenge has driven research along two parallel directions: (1) multi-stage inference pipelines that aggregate 2D foundation model ( Kirillov et al. 2023 ; Radford et al. 2021 ) proposals into 3D ( Takmaz et al. 2023 ; Nguyen et al. 2024 ; Yin et al. 2024 ) , and (2) leveraging 2D models to generate pseudo-labeled 3D-text datasets for training end-to-end networks ( Luo et al. 2024 ; Luo et al. 2023 ; Jia et al. 2024 ; Yang et al. 2024b ; Lee et al. 2025a ) . Multi-stage inference pipelines suffer from compounding errors across stages, dataset-specific thresholds that limit generalization, and substantial inference latency from sequential proposal processing—often hundreds of seconds per scene (Fig. 1 ). Meanwhile, pseudo-labeling approaches offer end-to-end learning, yet existing methods rely on single-view captioning that produces inconsistent descriptions, and lifting 2D outputs to 3D yields fragmented instances due to occlusion. Additionally, open-vocabulary instance segmentation demands spatially coherent features to distinguish fine-grained object boundaries across arbitrary categories, yet existing 3D backbones prioritize efficiency over spatial coherence: architectures like Point Transformer ( Wu et al. 2024 ) and OctFormer ( Wang 2023 ) serialize point clouds via space-filling curves, scattering spatially adjacent points across attention windows, making it difficult to capture the sharp, instance-level boundaries necessary for generalizing to novel object types. To address these limitations in pseudo-labeled dataset construction and 3D backbone design, we introduce SpaCeFormer-3M, a large-scale open-vocabulary 3D instance segmentation dataset constructed via multi-view mask clustering and captioning, and SpaCeFormer ( Spa ce- C urv e Trans former ), an end-to-end network architecture tailored for open-vocabulary 3D instance segmentation. Our key insight is that multi-view consistency, both in mask aggregation and captioning, combined with spatially coherent 3D features enables effective proposal-free instance segmentation without multi-stage heuristics. On the data side, we address two key limitations of RegionPLC ( Yang et al. 2024b ) and Mosaic3D ( Lee et al. 2025a ) : fragmented masks and inconsistent captions caused by single-view processing. We aggregate partial 2D masks into complete, geometry-consistent 3D instances via multi-view mask clustering ( Yan et al. 2024 ) , and employ multi-view prompting to produce view-consistent descriptions. The resulting dataset, SpaCeFormer-3M (3.0M multi-view-consistent captions over 604K instance masks across 7.4K scenes), has significantly higher mask completeness and caption consistency than prior pseudo-label corpora—achieving 54.3% mask recall at IoU > 0.5 on ScanNet versus 2.5% for single-view pipelines like Mosaic3D, a 21 \times improvement (Sec. 3 ). Crucially, geometry-consistent training data is essential for our proposal-free architecture: the decoder must learn to predict complete instance masks from spatially coherent features, which requires training on complete, non-fragmented mask annotations. On the modeling side, SpaCeFormer builds on Point Transformer v3 ( Wu et al. 2024 ) with a key architectural innovation: space-curve attention that strategically combines spatial window attention with Morton curve serialization. While each component exists in prior work, their combination is novel and specifically motivated by our proposal-free setting—spatial windows preserve coherent local neighborhoods critical for instance boundary prediction (achieving 28.6% lower intra-window spatial distance than Morton-only attention), while Morton curves provide structured diversity at coarser scales. Together with 3D RoPE ( Su et al. 2024 ) in both backbone and decoder, these features support proposal-free mask prediction via learned queries ( Carion et al. 2020 ; Cheng et al. 2022 ) without multi-stage heuristics. Extensive experiments demonstrate that SpaCeFormer achieves state-of-the-art performance on open-vocabulary 3D instance segmentation benchmarks. On Replica, SpaCeFormer reaches 24.1 zero-shot mAP at 0.14 s per scene —the only open-vocabulary method that operates at interactive rates, and matched in accuracy only by SOLE ( Lee et al. 2025b ) (24.7 mAP), which requires ScanNet200 ground-truth mask supervision and is \sim 9 \times slower (Fig. 1 , Tab. 3 ). On ScanNet200 (Tab. 5 ), under the matched regime of proposal-free, 3D-only input, no GT 3D annotations , SpaCeFormer attains 11.1 mAP at 0.14 s per scene—2.8 \times over the next-best method in this regime (Mosaic3D+Decoder, 3.9 mAP). Methods that score higher on ScanNet200 rely on either GT-trained proposals ( e.g . Mask3D-based pipelines at up to 24.7 mAP, 16–554 s) or multi-view 2D streams with YOLO/SAM proposals (up to 26.0 mAP, \sim 356 s). On ScanNet++, we achieve 22.9 mAP, surpassing the best prior method OpenTrack3D (20.6 mAP) which relies on multi-view 2D inputs and requires \sim 320 seconds per scene (Tab. 4 ). Our contributions are four-fold: • SpaCeFormer-3M, the largest open-vocabulary 3D instance segmentation