GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding
Haoyi Jiang, Liu Liu, Tianheng Cheng, Xinjie Wang, Tianwei Lin, Zhizhong Su, Wenyu Liu, Xinggang Wang · 2024
Read GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding by Haoyi Jiang, Liu Liu, Tianheng Cheng, Xinjie Wang, Tianwei Lin, Zhizhong Su, Wenyu Liu, Xinggang Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13193.txt.
Introduction
Haoyi Jiang and colleagues teach a model to read 3D scenes as clouds of soft Gaussian blobs, aligned with big foundation models, so it needs no hand labels. Filling space with labelled voxels is slow and label-hungry. GaussTR predicts a sparse set of Gaussians in one pass, splats them into camera views, and matches the rendered features against foundation model features.
That self-supervision yields open-vocabulary occupancy prediction: the model can name regions it was never taught. On a standard driving dataset it set a zero-shot record score, while cutting training time by forty percent. The authors aim it at self-driving cars and embodied agents.
Catalog record
| CREATOR | Haoyi Jiang, Liu Liu, Tianheng Cheng, Xinjie Wang, Tianwei Lin, Zhizhong Su, Wenyu Liu, Xinggang Wang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13193
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding Haoyi Jiang, Liu Liu, Tianheng Cheng, Xinjie Wang, Tianwei Lin, Zhizhong Su, Wenyu Liu, Xinggang Wang Abstract 3D Semantic Occupancy Prediction is fundamental for spatial understanding, yet existing approaches face challenges in scalability and generalization due to their reliance on extensive labeled data and computationally intensive voxel-wise representations. In this paper, we introduce GaussTR , a novel Gauss ian-based TR ansformer framework that unifies sparse 3D modeling with foundation model alignment through Gaussian representations to advance 3D spatial understanding. GaussTR predicts sparse sets of Gaussians in a feed-forward manner to represent 3D scenes. By splatting the Gaussians into 2D views and aligning the rendered features with foundation models, GaussTR facilitates self-supervised 3D representation learning and enables open-vocabulary semantic occupancy prediction without requiring explicit annotations. Empirical experiments on the Occ3D-nuScenes dataset demonstrate GaussTR’s state-of-the-art zero-shot performance of 12.27 mIoU, along with a 40% reduction in training time. These results highlight the efficacy of GaussTR for scalable and holistic 3D spatial understanding, with promising implications in autonomous driving and embodied agents. The code is available at https://github.com/hustvl/GaussTR . 1 Introduction Figure 1 : Comparative performance of self-supervised 3D occupancy prediction methods. GaussTR achieves a 2.33 mIoU (23%) improvement over other counterparts while reducing training time by approximately 40%, using merely 3% of the scene representation parameters ( e.g . , voxels or Gaussians). Marker sizes are proportional to the logarithm of scene representation parameters. The pursuit of artificial general intelligence has underscored 3D spatial understanding as a cornerstone en route to higher levels of autonomy, empowering autonomous systems to perceive, reason about, and interact with complex 3D environments for applications in autonomous driving and embodied agents. While Vision Foundation Models (VFMs)—such as CLIP [ 38 ] , DINO [ 7 , 35 ] , and EVA [ 14 , 15 ] —have pushed the boundaries of 2D visual perception, their extension to self-supervised 3D spatial understanding remains relatively uncharted due to the inherent gap between 2D vision and 3D spatial intelligence. 3D Semantic Occupancy Prediction aims to generate comprehensive spatial cognition by predicting voxel-wise occupancy and semantics, forming a fundamental task for spatial understanding. While prevalent supervised methods [ 23 , 29 , 59 , 26 ] have attained superior performance, they heavily depend on extensive 3D annotations, which are costly and labor-intensive to acquire. Besides, dense voxel representations impose substantial computational overhead and struggle with capturing high-level contextual relationships attributed to excessive granularity. Recent self-supervised approaches [ 24 , 57 , 48 , 17 ] , inspired by RenderOcc [ 36 ] , have sought to alleviate the data constraints by leveraging temporal image reprojection consistency [ 18 ] to provide geometric supervision. However, their reliance on pre-computed 2D segmentation pseudo-labels restricts their ability to learn generalizable representations, leaving them prone to out-of-distribution scenarios in real-world applications. Drawing inspiration from the emerging 3D Gaussian splatting (GS) [ 27 ] for scene reconstruction, we propose GaussTR , a novel Gauss ian-based TR ansformer that integrates sparse 3D modeling with foundation models through Gaussian representations to facilitate self-supervised 3D spatial understanding. GaussTR models scenes as sparse, unstructured sets of Gaussians, predicted through a series of Transformer [ 46 ] layers in a feed-forward manner. Specifically, GaussTR aggregates multi-view VFM features for individual Gaussians with deformable cross-attention [ 62 ] , followed by global self-attention across Gaussian queries for effective 3D modeling. The resultant Gaussian parameters are regressed via MLPs within a dedicated Gaussian head. Exploiting the consistency of Gaussian splatting across 2D and 3D modalities, GaussTR renders Gaussians back to 2D views and enforces feature alignment with foundation models. GaussTR thus learns versatile 3D representations enriched with broad visual priors, enabling open-vocabulary occupancy prediction by similarity measurement with target categories without explicit annotations. Optionally, auxiliary segmentation supervision from Grounded SAM [ 28 , 39 ] is incorporated to further refine rendered Gaussian boundaries. Extensive experiments on the Occ3D-nuScenes [ 45 ] dataset demonstrate that GaussTR achieves state-of-the-art zero-shot performance of 12.27 mIoU—a 23% improvement over previous methods—while simultaneously reducing training time by 40%, as shown in Fig. 1 . Ablation studies further validate the efficacy of our design choices. These results underscore the synergistic benefits of our proposed Gaussian-based 3D modeling and foundation model alignment, charting a scalable pathway for generalizable 3D spatial understanding. In summary, our contributions are as follows: • Scene Representation as Sparse Gaussians. GaussTR introduces a novel Gaussian-based 3D modeling method through feed-forward Transformer prediction, eliminating dense voxel grids and alleviating computational overhead. • Foundation Model Alignment for Self-Supervised Learning. GaussTR learns generalizable 3D representations aligned with foundation models via differentiable splatting, facilitating self-supervised open-vocabulary occupancy prediction without requiring explicit labels. • State-of-the-Art Zero-Shot Performance. GaussTR achieves 12.27 mIoU on Occ3D-nuScenes, outperforming previous methods by 23% alongside a 40% reduction in training time, underscoring the efficacy of Gaussian-based 3D modeling and foundation model alignment. 2 Related Works 2.1 3D Semantic Occupancy Prediction 3D Semantic Occupancy Prediction, formerly known as 3D Semantic Scene Completion (SSC) [ 42 ] , aims to predict the occupancy and semantics for each voxel grid within a 3D scene volume. By providing spatial awareness, 3D occupancy is crucial for applications such as navigation and obstacle avoidance in the fields of autonomous driving and embodied agents. Prevalent approaches have explored 2D-to-3D projection mechanisms [ 5 , 29 , 30 , 22 ] , efficient scene representations [ 23 , 26 , 31 , 25 , 40 , 55 ] , and architectural advancements [ 59 , 29 , 45 , 32 , 33 , 10 ] . Despite their contributions, these supervised methods depend on detailed voxel-wise annotations, which are costly to obtain and constrain their scalability. RenderOcc [ 36 ] alleviates this constraint by leveraging volume rendering [ 34 ] to derive 3D supervision exclusively from 2D semantic and depth labels. Following RenderOcc, several self-supervised methods [ 24 , 57 , 20 ] have been proposed, treating occupancy volumes as radiance fields and using the temporal consistency of rendered or reprojected images [ 18 , 49 , 6 ] to provide geometric supervision. While reducing the need for voxel-wise annotations, these approaches still rely on open-vocabulary segmentation models to generate semantic pseudo-labels, making them struggle with generalizing to out-of-distribution scenarios. Besides, they suffer from the computational overhead of dense voxel-wise modeling and volume rendering. DistillNeRF [ 48 ] and OccFeat [ 41 ] enhance 3D representation learning by distilling volumetric radiance fields with VFMs including DINO [ 7 , 35 ] and CLIP [ 38 ] . GaussianOcc [ 17 ] improves training efficiency by replacing volume rendering with Gaussian splatting. POP3D [ 47 ] and LangOcc [ 3 ] align spatial representations with linguistic feature spaces for open-vocabulary predictions. Our proposed GaussTR diverges from prior works in two key aspects: (1) it adopts sparse, feed-forward 3D Gaussian splatting as an alternative to dense voxel-wise modeling for generalizable representation learning and computational efficiency, and (2) it enables knowledge alignment with foundation models for open-vocabulary 3D semantic occupancy prediction, mitigating the need for explicit labels. Figure 2 : Architectural overview of the GaussTR framework. The GaussTR framework initiates with extracting multi-view features with pre-trained foundation models.