Gen-LangSplat: Generalized Language Gaussian Splatting with Pre-Trained Feature Compression

Pranav Saxena · 2025

Read Gen-LangSplat: Generalized Language Gaussian Splatting with Pre-Trained Feature Compression by Pranav Saxena (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22930.txt.

Introduction

Pranav Saxena works on three-dimensional scenes that a person can query with plain words. Past systems built such language fields with 3D Gaussian splatting, which spreads soft blobs to model a scene. They first had to train a fresh compressor for each new scene, a slow step that blocked wider use.

His Gen-LangSplat swaps that step for one shared compressor trained in advance on the large ScanNet data set. A fixed compact code then works across new scenes with no extra training. The change makes language-linked 3D scenes faster to build for robots and other tools that work in real spaces.

Catalog record

CREATORPranav Saxena
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22930

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gen-LangSplat: Generalized Language Gaussian Splatting with Pre-Trained Feature Compression Pranav Saxena Abstract Modeling open-vocabulary language fields in 3D is essential for intuitive human-AI interaction and querying within physical environments. State-of-the-art approaches, such as LangSplat, leverage 3D Gaussian Splatting to efficiently construct these language fields, encoding features distilled from high-dimensional models like CLIP. However, this efficiency is currently offset by the requirement to train a scene-specific language autoencoder for feature compression, introducing a costly, per-scene optimization bottleneck that hinders deployment scalability. In this work, we introduce Gen-LangSplat, that eliminates this requirement by replacing the scene-wise autoencoder with a generalized autoencoder, pre-trained extensively on the large-scale ScanNet dataset. This architectural shift enables the use of a fixed, compact latent space for language features across any new scene without any scene-specific training. By removing this dependency, our entire language field construction process achieves a efficiency boost while delivering querying performance comparable to, or exceeding, the original LangSplat method. To validate our design choice, we perform a thorough ablation study empirically determining the optimal latent embedding dimension and quantifying representational fidelity using Mean Squared Error and cosine similarity between the original and reprojected 512-dimensional CLIP embeddings. Our results demonstrate that generalized embeddings can efficiently and accurately support open-vocabulary querying in novel 3D scenes, paving the way for scalable, real-time interactive 3D AI applications. The code can be found at https://github.com/Pranav-Saxena/Gen-LangSplat . Figure 1 : Visualization of learned 3D language features of the previous SOTA method, LangSplat, and our proposed approach. Without requiring per-scene training for the language-feature autoencoder, our method achieves comparable, or even superior, results while being more efficient. † † footnotetext: 1 Pranav Saxena is with Birla Institute of Technology and Science Pilani, K.K Birla Goa Campus, Goa, India. This work was done in collaboration with National University of Singapore and TU Delft, Netherlands. 1 Introduction Understanding and interacting with the 3D world through natural language is an emerging challenge at the intersection of 3D vision and multimodal learning. The ability to associate linguistic concepts with spatially grounded representations is essential for a range of applications in embodied AI, including open-vocabulary scene understanding, 3D semantic reasoning, and language-guided robotic manipulation. Earlier methods have primarily relied on Neural Radiance Fields (NeRFs) that distill vision-language model (VLM) features, such as those from CLIP radford2021learning , into volumetric neural fields. While these approaches enable open-ended language queries within reconstructed 3D scenes, the implicit nature of NeRFs introduces substantial inefficiencies, rendering is slow, optimization requires dense sampling, and the models are difficult to adapt or edit. These limitations have motivated the exploration of more explicit and computationally efficient 3D representations. Recent advances in 3D Gaussian Splatting (3DGS) kerbl20233d have demonstrated remarkable efficiency and visual fidelity by representing scenes as a collection of spatially distributed anisotropic Gaussians. Building on this representation, LangSplat qin2023langsplat introduced the concept of a 3D language field, where CLIP features are attached to individual Gaussians to support open-vocabulary querying in 3D. By replacing volumetric NeRF rendering with tile-based Gaussian rasterization, LangSplat achieves over two orders of magnitude faster performance while maintaining high-quality alignment between image and language spaces. However, a key limitation of LangSplat lies in its reliance on per-scene autoencoders, which are trained separately for every scene to compress high-dimensional CLIP features into a low-dimensional latent space. This per-scene feature compression not only adds significant computational overhead but also prevents generalization to unseen scenes, since each environment requires new optimization to adapt the language embeddings. In this work, we revisit this design bottleneck and propose a framework that eliminates the need for scene-specific autoencoders through the introduction of a generalized autoencoder trained across diverse scenes. Our method leverages a large-scale dataset, ScanNet dai2017scannet , to train a unified encoder–decoder network capable of learning a compact and transferable latent space for CLIP embeddings. Once trained, the generalized autoencoder can directly process CLIP features from new scenes without any fine-tuning, enabling rapid deployment across diverse environments. Integrating this generalized feature compression into a 3D Gaussian Splatting pipeline preserves the open-vocabulary grounding ability of LangSplat while improving overall efficiency by approximately 2\times . Each Gaussian in our model jointly encodes appearance, geometry, and low-dimensional language features derived from the generalized embedding, which are decoded back to CLIP space during rendering to ensure multi-view consistency. This design not only accelerates training and inference but also establishes a consistent latent manifold shared across different scenes, facilitating scalable and cross-scene language reasoning in 3D. To evaluate the representational strength of our generalized embedding, we perform a comprehensive ablation study that investigates both efficiency and reconstruction fidelity. We vary the latent dimensionality to analyze its effect on rendering quality, memory cost, and retrieval accuracy, and further quantify how closely reprojected CLIP embeddings approximate their original 512-dimensional counterparts using mean squared error (MSE) and cosine similarity metrics. Our findings reveal that a 16-dimensional embedding achieves the optimal balance between compactness and fidelity, outperforming the 3-dimensional latent space used in LangSplat in both reconstruction accuracy and semantic consistency. Despite removing the per-scene training step, our method achieves comparable localization precision and open-vocabulary retrieval accuracy to LangSplat, confirming that a single generalized autoencoder is sufficient for language-grounded 3D reconstruction across diverse real-world environments. In summary, our work establishes that cross-scene feature compression is a viable and efficient alternative to scene-specific optimization, making language-aware 3D Gaussian Splatting more practical and effective. By decoupling the language embedding process from per-scene adaptation, we demonstrate that robust open-vocabulary 3D understanding can be achieved at a fraction of the cost and time. We summarize the contributions of this paper as follows: • We introduce a cross-scene generalized autoencoder trained on the ScanNet dataset that learns a transferable latent representation for language features. Unlike prior per-scene approaches, our model eliminates the need for scene-specific retraining while maintaining strong semantic consistency across diverse environments. • Our method achieves nearly a 2× improvement in overall efficiency compared to LangSplat by removing the need to train a separate autoencoder for each scene. • We conduct a detailed ablation study on latent dimensionality to analyze its impact on feature reconstruction and semantic retention. The results show that a 16-dimensional latent embedding achieves high cosine similarity and low MSE when reprojected into the CLIP feature space, without compromising efficiency. 2 Related Work 2.1 3D Scene Representations and Neural Rendering Neural implicit representations have been widely adopted for high-quality 3D reconstruction and rendering. Early works such as DeepVoxels sitzmann2019deepvoxels and Neural Volumes Lombardi2019 introduced volumetric neural rendering pipelines that model scene appearance as continuous functions. Neural Radiance Fields (NeRFs) mildenhall2020nerf further revolutionized this field by representing radiance and density using multilayer perceptrons, achieving photorealistic view synthesis from sparse inputs. However, the implicit nature of NeRFs incurs heavy computational cost due to dense ray sampling and global optimization, limiting their real-time applicability. Later extensions such as Mip-NeRF barron2021mipnerf , Plenoxels yu_and_fridovichkeil2021plenoxels , and Instant-NGP muller2022instant improved speed and efficiency using mip-level sampling, voxel grids, and hash-based encodings, yet still rely on costly volume integration. 3D Gaussian Splatting (3DGS) kerbl20233d introduced a paradigm shift by representing scenes as explicit, spatially adaptive anisotropic Gaussians optimized through differentiable rasterization. Each Gaussian encapsulates geometry, color, opacity, and orientation, enabling real-time rendering with significantly reduced memory usage. Subsequent works have

Read the complete text