Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds
Bin Yang, Mohamed Abdelsamad, Miao Zhang, Alexandru Paul Condurache · 2026
Read Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds by Bin Yang, Mohamed Abdelsamad, Miao Zhang, Alexandru Paul Condurache (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25165.txt.
Introduction
Yang and co-authors teach 3D scene models to notice single objects, not just broad scene sense. Self-taught point cloud models learn meaning well but place items poorly, and they often need full retraining later. PointINS adds a branch that predicts a small offset for each point. Two rules guide those offsets: one matches them to known shape habits, one keeps nearby points in step using rough item masks. Across five datasets, indoor item splits gain about 3.5 points of mAP. Outdoor panoptic scores gain about four points.
Catalog record
| CREATOR | Bin Yang, Mohamed Abdelsamad, Miao Zhang, Alexandru Paul Condurache |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53748 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25165
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds Bin Yang, Mohamed Abdelsamad, Miao Zhang, Alexandru Paul Condurache Abstract Recent advances in self-supervised learning (SSL) for point clouds have substantially improved 3D scene understanding without human annotations. Existing approaches emphasize semantic awareness by enforcing feature consistency across augmented views or by masked scene modeling. However, the resulting representations transfer poorly to instance localization, and often require full finetuning for strong performance. Instance awareness is a fundamental component of 3D perception, thus bridging this gap is crucial for progressing toward true 3D foundation models that support all downstream tasks on 3D data. In this work, we introduce PointINS , an instance-oriented self-supervised framework that enriches point cloud representations through geometry-aware learning. PointINS employs an orthogonal offset branch to jointly learn high-level semantic understanding and geometric reasoning, yielding instance awareness. We identify two consistent properties essential for robust instance localization and formulate them as complementary regularization strategies, Offset Distribution Regularization (ODR), which aligns predicted offsets with empirically observed geometric priors, and Spatial Clustering Regularization (SCR), which enforces local coherence by regularizing offsets with pseudo-instance masks. Through extensive experiments across five datasets, PointINS achieves on average +3.5% mAP improvement for indoor instance segmentation and +4.1% PQ gain for outdoor panoptic segmentation, paving the way for scalable 3D foundation models. 1 Introduction Figure 1 : We achieve superior results over state-of-the-art self-supervised approaches (DOS Abdelsamad et al. 2026 and Sonata Wu et al. 2025 ) on both indoor instance (left) and outdoor panoptic segmentation (right). Self-supervised learning (SSL) has achieved great success in 2D visual representation learning Caron et al. 2021 ; He et al. 2022b ; Xie et al. 2022 ; Assran et al. 2023 over the past decade. It enables models to extract powerful features from large-scale unlabeled data that generalize well across diverse downstream tasks. Extending this paradigm to 3D scene understanding holds significant potential, especially in domains such as autonomous driving and robotics, where large amounts of 3D data are available but manual annotation remains costly and labor-intensive Coors et al. 2019 ; Kong et al. 2023 ; Unal et al. 2022 ; Löwens et al. 2024 . Recent works have demonstrated promising progress in 3D SSL by applying contrastive Nunes et al. 2022 ; Nisar and Waslander 2025 ; Xie et al. 2020 and masked modeling Wu et al. 2025 ; Hermosilla et al. 2025 ; Abdelsamad et al. 2026 . These methods achieve impressive performance on semantic segmentation benchmarks. Their success largely stems from enforcing multi-view consistency, where models are trained to produce aligned embeddings across independently augmented views of the same point cloud Wu et al. 2025 ; Caron et al. 2021 ; Hermosilla et al. 2025 . This objective effectively captures global semantic structure (see Fig. 2 ). Nevertheless, such semantic-driven objectives inherently overlook instance awareness, an ability of a representation to preserve fine-grained geometric relationships. Consequently, these approaches underperform on instance-oriented tasks such as instance and panoptic segmentation. Closing this gap is essential for advancing unified 3D foundation models that can generalize across diverse tasks and domains. A key challenge lies in balancing semantic invariance with the geometric sensitivity needed for learning instance awareness. Prior works Wu et al. 2025 ; Hermosilla et al. 2025 have identified that point cloud SSL can collapse to trivial cues like normals or poses of points, thus, most methods enforce strong invariance to avoid such geometric shortcuts. While we acknowledge this concern, we argue that the geometric proximity required for instance-aware learning represents a high-level relational property rather than a low-level geometric cue. We refer to this capability as geometric reasoning . Beyond learning consistency for semantics, SSL features should encode where a point should direct, ideally toward the centroid of the instance it belongs to. Our hypothesis aligns with supervised instance and panoptic segmentation frameworks Jiang et al. 2020 ; Zhou et al. 2021 ; Li et al. 2023 ; Li et al. 2022 , where semantic categorization guides subsequent geometric grouping into class-agnostic instances. These architectures typically use a shared backbone with parallel semantic and offset branches: semantics restrict the candidate regions for instance boundaries, while offsets refine the spatial separation. By reinforcing each other, they jointly enhance the model’s overall understanding of 3D scenes. Figure 2 : Example of an indoor/outdoor scene and K-means clustering over their point features extracted from a self-supervised pre-trained model Wu et al. 2025 . Building upon this insight, we propose PointINS , the first self-supervised framework explicitly designed to learn both semantic consistency and geometric reasoning. Our framework augments the SSL framework with an additional instance-localization branch that predicts point-wise geometric offsets toward underlying instance centers. However, predicting the offsets without labels remains challenging. Without ground-truth guidance, the model struggles to infer meaningful geometric relationships, and may even collapse to trivial solutions Caron et al. 2021 ; Assran et al. 2023 . At the same time, we observe that offsets inherently exhibit several stable statistical and structural properties that can be exploited to guide and regularize the learning process. To this end, we introduce two complementary regularization strategies: Offset Distribution Regularization (ODR) provides a global constraint by aligning predicted offsets with empirical geometric priors observed in real scenes, while Spatial Clustering Regularization (SCR) introduces a local constraint by enforcing consistent centroid alignment among points through pseudo-instance masks derived from the model’s semantic understanding. Together, these components provide both global distributional alignment and local geometric coherence to enable robust instance-aware representation learning. We evaluate PointINS extensively on both indoor and outdoor scene datasets, covering semantic, instance, and panoptic segmentation under linear probing, decoder probing, and finetuning protocols. Our method achieves state-of-the-art performance among self-supervised approaches, yielding 2.5–4.6% mAP gains on indoor instance segmentation and 3.4–4.8% PQ gains on outdoor panoptic segmentation. To summarize, this work contains the following key contributions: • We introduce a novel self-supervised training framework for point clouds that jointly learns semantic consistency and geometric reasoning. • We identify two consistent properties essential for robust instance awareness and introduce complementary regularization strategies that prevent model collapse while enabling effective representation learning. • Our method surpasses the state-of-the-art methods by a clear margin on indoor instance and outdoor panoptic segmentation, representing a step toward unified 3D foundation models for holistic scene understanding. 2 Related Works 2.1 3D Instance & Panoptic Segmentation 3D instance segmentation methods typically rely on dense supervision to learn point-wise offsets or embeddings that facilitate object grouping. Early approaches Jiang et al. 2020 ; Chen et al. 2021 ; Vu et al. 2022 predict offset vectors pointing toward object centers and cluster points using mean-shift algorithms or connected components. Panoptic segmentation extends this paradigm by jointly predicting semantic labels and instance identities for every point in the scene Fong et al. 2022 ; Mei et al. 2022 . Many of these methods adopt a dual-branch design, where semantic and instance segmentation are learned simultaneously with a shared backbone. While highly effective, they depend on a large amount of instance and semantic annotations for fully supervised training. In this work, we seek to learn both semantic and instance-aware representations directly from unlabeled point clouds for enhancing performance on diverse segmentation tasks. 2.2 Point