SpatialSplat: Efficient Semantic 3D from Sparse Unposed Images
Yu Sheng, Jiajun Deng, Xinran Zhang, Yu Zhang, Bei Hua, Yanyong Zhang, Jianmin Ji · 2025
Read SpatialSplat: Efficient Semantic 3D from Sparse Unposed Images by Yu Sheng, Jiajun Deng, Xinran Zhang, Yu Zhang, Bei Hua, Yanyong Zhang, Jianmin Ji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23044.txt.
Introduction
Yu Sheng and co-authors presented SpatialSplat in 2025, a fast method for building semantic 3D scenes from a few photos. Feedforward systems can now turn sparse, unposed images into 3D Gaussian primitives in one pass. Adding meaning to each primitive is the hard part, since rich semantic features cost memory. Crude compression blurs fine detail, and overlapping views breed redundant primitives.
SpatialSplat splits semantics into two fields. A coarse field carries full-strength meaning on few primitives, since one object tends to share one meaning. A fine, low-dimensional field adds detail between neighboring instances. A selection step drops Gaussians the scene does not need. The result cut scene parameters by about sixty percent while beating rival methods. The authors aim to make semantic 3D practical for real applications.
Catalog record
| CREATOR | Yu Sheng, Jiajun Deng, Xinran Zhang, Yu Zhang, Bei Hua, Yanyong Zhang, Jianmin Ji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23044
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpatialSplat: Efficient Semantic 3D from Sparse Unposed Images Yu Sheng, Jiajun Deng, Xinran Zhang, Yu Zhang, Bei Hua, Yanyong Zhang, Jianmin Ji Abstract A major breakthrough in 3D reconstruction is the feedforward paradigm to generate pixel-wise 3D points or Gaussian primitives from sparse, unposed images. To further incorporate semantics while avoiding the significant memory and storage costs of high-dimensional semantic features, existing methods extend this paradigm by associating each primitive with a compressed semantic feature vector. However, these methods have two major limitations: (a) the naively compressed feature compromises expressiveness, affecting the model’s ability to capture fine-grained semantics, and (b) the pixel-wise primitive prediction introduces redundancy in overlapping areas, causing unnecessary memory overhead. To this end, we introduce SpatialSplat , a feedforward framework that produces redundancy-aware Gaussians and capitalizes on a dual-field semantic representation. Particularly, with the insight that primitives within the same instance exhibit high semantic consistency, we decompose the semantic representation into a coarse feature field that encodes uncompressed semantics with minimal primitives, and a fine-grained yet low-dimensional feature field that captures detailed inter-instance relationships. Moreover, we propose a selective Gaussian mechanism, which retains only essential Gaussians in the scene, effectively eliminating redundant primitives. Our proposed Spatialsplat learns accurate semantic information and detailed instances prior with more compact 3D Gaussians, making semantic 3D reconstruction more applicable. We conduct extensive experiments to evaluate our method, demonstrating a remarkable 60% reduction in scene representation parameters while achieving superior performance over state-of-the-art methods. The code will be made available for future investigation. Figure 1 : Comparison between previous methods and our SpatialSplat. (a): Previous methods predict pixel-wise Gaussians, associating each primitive with compressed semantic feature. (b): Our SpatialSplat avoids generating redundant Gaussian primitives for overlapping pixels, and it represents semantics with a dual-field architecture to better preserve the information. (c): SpatialSplat outperforms the state-of-the-art method LSM [ 15 ] for both novel-view rendering quality and semantic segmentation while being more efficient. 1 Introduction Reconstructing and understanding 3D scenes from 2D images [ 9 , 53 , 32 , 36 , 5 ] is a fundamental topic in computer vision, aiming to obtain semantic-aware 3D structure from low-cost devices, i.e. , RGB cameras. This technique is significant for various applications, such as robotics, autonomous driving, and VR/AR. Recently, with the emergence of the new 3D representation, i.e. , Neural Radiance Fields (NeRF) [ 29 ] and 3D Gaussian Splatting (3DGS) [ 19 ] , a popular paradigm of performing semantic-aware 3D reconstruction is to distill the feature field of NeRF or 3DGS with powerful 2D vision-language foundation models [ 20 , 52 , 31 , 33 ] . However, these methods typically rely on per-scene optimization and complex multi-step preprocessing, limiting their ability to generalize across multiple scenes within a single model. Recent breakthroughs [ 7 , 3 , 49 ] in 3D reconstruction have incorporated feed-forward networks to improve the generalization of 3DGS models and accelerate reconstruction. These methods follow a paradigm that generates pixel-wise Gaussian primitives from sparse posed images. Building on this, further efforts [ 46 , 38 ] have extended the paradigm to reconstruct scenes from unposed images. Moreover, methods [ 15 , 43 ] have advanced this approach by integrating semantic understanding, enabling semantic 3D reconstruction from sparse unposed images. Despite significant progress, these methods have two major limitations. First, as shown in Fig. 1 (a), pixel-wise Gaussian prediction introduces substantial redundancy in overlapping regions while providing minimal accuracy gains. Method [ 44 ] attempts to mitigate this issue by projecting primitives onto a plane and identifying overlaps, it depends on precise camera extrinsics, which are often difficult to obtain in real-world scenarios—especially when only a few sparse, textureless images are available. Second, methods [ 52 , 31 , 33 , 43 , 15 ] enforce per-primitive semantic learning to form a dense semantic feature field, as shown in Fig. 1 (a). This primitive-level approach incurs significant memory and storage costs due to high-dimensional (512 or more) language features. To mitigate this, the methods often compress features into lower-dimensional representations (e.g., 64-128), but this compression leads to irreversible information loss [ 52 ] , leading to suboptimal scene understanding performance. In this paper, we take a slightly different viewpoint. First, inspired by MASt3R [ 25 ] , which successfully extracts 3D geometry and pixel correspondences from images, we observe that redundant primitives often share similar geometry and appearance, allowing them to be identified directly from images without relying on complex geometric priors. Second, we find that per-primitive semantics are not essential for high-performance scene understanding. Instead, a coarse semantic representation, when combined with fine-grained instance information, is sufficient for strong performance, as Gaussian primitives within the same instance exhibit high semantic consistency. By consolidating this idea, we introduce SpatialSplat, a feed-forward network generating compact yet expressive semantic 3D representations from unposed images. As depicted in Fig. 1 (b), SpatialSplat introduces a dual-field architecture that decomposes the dense semantic feature field into two components: a coarse feature field encoding uncompressed instance-level semantics through minimal Gaussians, and a low-dimensional fine-grained feature field capturing inter-instance relationships. Guided by multiple 2D foundation models, our method learns both accurate semantic and instance priors, achieving superior performance while reducing representation parameters. Additionally, we introduce a Selective Gaussian Mechanism (SGM) to eliminate redundancy in overlapping areas caused by pixelwise representations, along with a novel loss function that jointly optimizes redundancy-aware Gaussians and scene fidelity. Extensive experiments demonstrate that our method achieves state-of-the-art performance on multiple downstream 3D tasks while using only 40 \% of the representation parameters required by the baseline, all without any 3D supervision. Our contributions are threefold: • A novel feed-forward 3DGS framework that, to the best of our knowledge, is the first to simultaneously learn semantic and instance priors with guidance from foundation models while maintaining a high-fidelity radiance field. • A dual-field semantic representation that significantly reduces storage consumption while enhancing open-vocabulary 3D segmentation performance. • A mechanism that identifies redundant primitives from images without relying on any geometric priors. 2 Related Work Figure 2 : Pipeline of SpatialSplat. The SpatialSplat processes unposed images along with their intrinsics through a 3D geometry transformer. The extracted features from the geometry transformer and the pretrained 2D model are then fed into separate Dense Prediction Transformer (DPT) heads to predict different Gaussian attributes, resulting in a fine-grained instance-aware radiance field \boldsymbol{\mathcal{F}_{I}} and a coarse semantic feature field \boldsymbol{\mathcal{F}_{S}} . This enables the synthesis of RGB and feature maps from novel viewpoints. 2.1 Feed-forward 3D Reconstruction The emergence of NeRF [ 29 ] and 3DGS [ 19 ] has significantly transformed the paradigm of 3D reconstruction from images. Despite achieving remarkable results in 3D reconstruction and novel view synthesis, these methods rely on time-consuming per-scene optimization, with even improved techniques [ 30 , 16 , 4 ] still requiring several minutes to hours. To overcome these limitations, recent approaches [ 7 , 3 , 48 , 41 , 45 , 18 , 11 , 49 ] have explored using a single feed-forward network to learn reconstruction priors from large datasets, enabling generalizable 3D reconstruction. DUSt3R serials [ 42 , 25 ] further simplifies the reconstruction pipeline, making 3D reconstruction possible from images without