NRGS: Neural Regularization for Robust 3D Semantic Gaussian Splatting

Zaiyan Yang, Xinpeng Liu, Heng Guo, Jinglei Shi, Zhanyu Ma, Fumio Okura · 2026

Read NRGS: Neural Regularization for Robust 3D Semantic Gaussian Splatting by Zaiyan Yang, Xinpeng Liu, Heng Guo, Jinglei Shi, Zhanyu Ma, Fumio Okura (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22439.txt.

Introduction

Yang and five colleagues refine noisy 3D semantic fields for Gaussian splatting, a scene-rendering method using Gaussian blobs. Vision models give inconsistent multi-view features, and lifting them into 3D Gaussians yields a noisy field that hurts later tasks. Earlier methods either fix features first or improve optimisation, at the cost of time or compute. The authors instead use a variance-aware conditional MLP on the Gaussians, correcting semantic errors from geometric and appearance attributes. Experiments show their method enhances lifted semantics efficiently, giving robust semantic Gaussian splatting.

Catalog record

CREATORZaiyan Yang, Xinpeng Liu, Heng Guo, Jinglei Shi, Zhanyu Ma, Fumio Okura
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23984
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22439

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NRGS: Neural Regularization for Robust 3D Semantic Gaussian Splatting Zaiyan Yang, Xinpeng Liu, Heng Guo, Jinglei Shi, Zhanyu Ma, Fumio Okura Abstract We propose a neural regularization method that refines the noisy 3D semantic field produced by lifting multi-view inconsistent 2D features, in order to obtain an accurate and robust 3D semantic Gaussian Splatting. The 2D features extracted from vision foundation models suffer from multi-view inconsistency due to a lack of cross-view constraints. Lifting these inconsistent features directly into 3D Gaussians results in a noisy semantic field, which degrades the performance of downstream tasks. Previous methods either focus on obtaining consistent multi-view features in the preprocessing stage or aim to mitigate noise through improved optimization strategies, often at the cost of increased preprocessing time or expensive computational overhead. In contrast, we introduce a variance-aware conditional MLP that operates directly on the 3D Gaussians, leveraging their geometric and appearance attributes to correct semantic errors in 3D space. Experiments on different datasets show that our method enhances the accuracy of lifted semantics, providing an efficient and effective approach to robust 3D semantic Gaussian Splatting. Fig. 1: 3D semantic segmentation results on 3DGS point clouds. While the existing methods ( e.g. , Occam’s LGS [ 2 ] ) do not consider consistency among multi-view 2D features, our NGRS corrects the inconsistency errors simultaneously with the optimization of 3D Gaussians, resulting in accurate segmentation. † † † Equal contribution. ∗ Corresponding author. I Introduction Understanding 3D scenes is a key task for applications like robot navigation [ 3 ] , self-driving [ 19 ] , and augmented reality [ 4 , 8 ] . A major goal is to enable open-vocabulary understanding, where a system can recognize objects based on free-form language descriptions, not just a fixed list of categories. Accordingly, a desirable approach is to take 2D semantic features from vision-language models as input and output a semantic 3D field within the 3D Gaussian Splatting (3DGS) [ 5 ] representation. A unique challenge is the incorporation of high-dimensional 2D semantic features into 3D space. LangSplat [ 16 ] employs a scene-specific autoencoder for feature compression, then lifts the compressed features into 3D through optimization-based distillation; however, this can compromise training speed and semantic fidelity. In contrast, OccamLGS [ 2 ] directly aggregates uncompressed 2D features into 3D via a weighted average scheme, preserving both speed and accuracy. Another challenge remains the multi-view inconsistency of the extracted 2D features. Specifically, changes in viewpoint and occlusion can lead to significant semantic drift across different views of the same 3D region. When such inconsistent 2D features are lifted into 3D, they result in a noisy and unreliable semantic field. Several methods [ 15 , 11 ] have been proposed to address this problem. However, this often comes at the cost of either extensive pre-processing or high computational iterative optimization, limiting their practical applicability. To refine noisy semantic fields from multi-view inconsistencies, we propose NRGS. Our approach is motivated by the observation that the semantics of Gaussians should align with their intrinsic geometric and appearance attributes. We introduce a lightweight MLP to learn this mapping, thereby regularizing the semantic field. This process effectively corrects semantic errors. As optimization operates directly on the 3DGS representation, the method remains efficient. Following the training-free lifting strategy [ 2 ] for efficiency, we project 2D semantic features of multiple granularities into 3D. More importantly, we estimate the variance of each lifted semantic as a confidence measure for the noisy multi-view signals. Then we train a lightweight conditional MLP to learn a continuous mapping from the attributes of Gaussians to their corresponding semantics, acting as a direct regularizer on the Gaussian semantics. However, treating all Gaussians equally is suboptimal. Therefore, we employ a variance-weighted training objective that adaptively prioritizes the learning from more reliable (lower-variance) Gaussians, ensuring robust regularization. Furthermore, by employing a single, shared MLP to refine semantics across multiple granularities, we encourage the network to learn a more general and unified representation rather than overfitting to any single semantic granularity. Collectively, our method builds a robust 3D semantic field, leading to improved performance as shown in Fig. 1 . To summarize, the key contributions of this paper are: • We propose NRGS, a neural regularization method that refines noisy semantics for robust 3D semantic Gaussian Splatting. • We introduce a variance-weighted training objective. It uses the variance of the initial lifted semantics as a confidence measure, enabling the model to adaptively focus on reliable signals for robust regularization. • We condition a single, shared MLP on multiple semantic granularities, promoting the learning of a general and unified representation. II Related works In this section, we first review powerful 2D foundation models that provide the features essential for 3D understanding. We then discuss methods that lift these 2D features into 3D Gaussians, highlighting the key challenge of multi-view inconsistency and how our approach addresses this problem. II-A 2D Open-Vocabulary Scene Understanding Recent progress in 2D open-vocabulary scene understanding has been driven by large-scale vision-language foundation models. CLIP [ 17 ] learns a shared image-text embedding space via contrastive pre-training, enabling open-vocabulary classification and providing semantic priors for dense prediction. Building on this, methods such as LSeg [ 10 ] and SegCLIP [ 13 ] distill CLIP knowledge for open-vocabulary semantic segmentation. Meanwhile, SAM [ 7 ] enables high-quality zero-shot instance segmentation. However, both CLIP- and SAM-based approaches are limited to 2D, and their predictions often exhibit significant cross-view inconsistencies in 3D scenes. II-B 3D Open-Vocabulary Scene Understanding 3DGS [ 5 ] has demonstrated strong performance in real-time novel view synthesis by representing scenes explicitly via anisotropic Gaussians. To enable 3D open-vocabulary understanding, prior work integrates semantic features from 2D foundation models into this representation. Early methods such as LangSplat [ 16 ] and Feature 3DGS [ 20 ] rely on optimization-based feature distillation to lift 2D embeddings into 3D, while LangSplatV2 [ 12 ] improves efficiency via sparse coefficient representation for real-time querying. Later approaches, including OccamLGS [ 2 ] and LUDVIG [ 14 ] , adopt training-free schemes by aggregating 2D features through weighted averaging. A common limitation of these methods is that they overlook the multi-view inconsistency inherent in 2D feature maps, where directly integrating inconsistent features introduces noise and ambiguity into the 3D semantic field. To address this, GAGS [ 15 ] improves multi-view consistency by relating SAM prompt point density to camera distance and introducing an unsupervised granularity factor for selectively distilling consistent features. Similarly, EgoSplat [ 11 ] proposes a multi-view consistent instance feature aggregation strategy that mitigates occlusion-induced semantic conflicts through high-quality viewpoint selection and enhances representations via cross-view aggregation. However, these approaches either require extensive pre-processing to establish cross-view correspondence or rely on time-consuming optimization, limiting their efficiency and flexibility. In contrast, we introduce a neural regularization method that corrects the lifted 3D semantic field by learning a mapping from Gaussian attributes to semantics. This avoids complex pre-processing or costly optimization during lifting, offering a simpler yet effective path to robust 3D semantics. III Method Fig. 2: Details of our neural regularization. A shared conditional MLP takes Gaussian attributes and a granularity condition g as input, and is trained with a variance-weighted loss to

Read the complete text