Tackling View-Dependent Semantics in 3D Language Gaussian Splatting

Jiazhong Cen, Xudong Zhou, Jiemin Fang, Changsong Wen, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian · 2025

Read Tackling View-Dependent Semantics in 3D Language Gaussian Splatting by Jiazhong Cen, Xudong Zhou, Jiemin Fang, Changsong Wen, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24746.txt.

Introduction

Jiazhong Cen and seven colleagues wrote a 2025 computer vision paper on language-driven understanding of 3D scenes. The base technology, 3D Gaussian Splatting, rebuilds a scene from ordinary photographs as a cloud of soft 3D blobs. Adding language lets users query scenes in words, but most methods simply project 2D image features onto the 3D blobs.

That shortcut misses view-dependent semantics: the same object can present different meanings from different viewpoints, such as the front and back of a laptop. Their system, LaGa, first decomposes the scene into objects. It then builds cross-view semantic connections so each object's description is aggregated over viewpoints. The result is a 3D scene that answers language queries more consistently.

Catalog record

CREATORJiazhong Cen, Xudong Zhou, Jiemin Fang, Changsong Wen, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24746

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tackling View-Dependent Semantics in 3D Language Gaussian Splatting Jiazhong Cen, Xudong Zhou, Jiemin Fang, Changsong Wen, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian Abstract Recent advancements in 3D Gaussian Splatting (3D-GS) enable high-quality 3D scene reconstruction from RGB images. Many studies extend this paradigm for language-driven open-vocabulary scene understanding. However, most of them simply project 2D semantic features onto 3D Gaussians and overlook a fundamental gap between 2D and 3D understanding: a 3D object may exhibit various semantics from different viewpoints—a phenomenon we term view-dependent semantics . To address this challenge, we propose LaGa ( La nguage Ga ussians), which establishes cross-view semantic connections by decomposing the 3D scene into objects. Then, it constructs view-aggregated semantic representations by clustering semantic descriptors and reweighting them based on multi-view semantics. Extensive experiments demonstrate that LaGa effectively captures key information from view-dependent semantics, enabling a more comprehensive understanding of 3D scenes. Notably, under the same settings, LaGa achieves a significant improvement of +18.7% mIoU over the previous SOTA on the LERF-OVS dataset. Our code is available at: https://github.com/https://github.com/SJTU-DeepVisionLab/LaGa . † † affiliationnotice: Equal contribution 1 Introduction 3D Gaussian Splatting (3D-GS) ( Kerbl et al., 2023 ) , a recent breakthrough in radiance fields, offers superior rendering efficiency and quality. Its unstructured 3D point-cloud-like representation makes it a promising foundation for open-vocabulary scene understanding. To this end, recent studies ( Qin et al., 2024 ; Shi et al., 2024 ; Bhalgat et al., 2024 ; Peng et al., 2025 ; Cheng et al., 2024 ; Qu et al., 2024 ) extend 3D-GS by lifting multi-view 2D semantic features extracted from vision-language models like CLIP ( Radford et al., 2021 ) into the 3D feature representation. As shown in Figure 1 (a), these methods heavily rely on the differentiable rasterization mechanism of 3D-GS. During training, they optimize the 3D features by aligning rendered feature maps with 2D semantic features at corresponding viewpoints. At inference, they continue rendering the learned 3D semantic features into 2D feature maps, using them for pixel-wise semantic understanding with cross-view consistency. Figure 1: Pipeline comparison of existing 2D methods (a) and the direct 3D scene understanding paradigm (b). While 2D methods excel in pixel-wise understanding via rendered feature maps, they fail unexpectedly when 3D Gaussians are directly retrieved by matching learned 3D features with CLIP text embeddings. However, when applying their learned 3D features to direct 3D perception ( Wu et al., 2024b ; Lee et al., 2025 ) , as shown in Figure 1 (b), their effectiveness degrades significantly, limiting their applicability to downstream tasks such as 3D object editing, AI-driven interaction, and precise 3D localization. We identify a fundamental issue behind this limitation: the view-dependency of 3D semantics . As illustrated in Figure 2 , a passport viewed from the front clearly reveals its title, while from the back or side, it becomes unrecognizable. This phenomenon highlights a fundamental gap between 2D and 3D understanding. Simply projecting 2D semantics onto 3D Gaussians results in incomplete or inaccurate semantic assignments, as each Gaussian inherits semantics visible only from specific viewpoints. Specifically, this leads to false positives (noisy Gaussians) and false negatives (incomplete results), as depicted in Figure 1 (b). Figure 2: An illustration of view-dependent 3D semantics. The passport exhibits different semantics from different viewpoints. To quantify this issue, we conduct two analyses. First, a semantic similarity distribution analysis shows that multi-view semantic features of the same object often exhibit lower intra-object similarities than inter-object similarities, providing direct evidence of view-dependent semantics. Second, a semantic retrieval integrity analysis finds that about 50% of 2D semantic features fail to retrieve their corresponding 3D objects completely, further validating its negative affect. To address this challenge, we propose LaGa , a simple yet effective method for open-vocabulary 3D scene understanding with 3D-GS. LaGa first performs 3D scene decomposition by grouping multi-view 2D masks into coherent 3D objects, establishing cross-view semantic connections to explicitly capture view-dependent 3D semantics. Then, LaGa constructs view-aggregated semantic representations by extracting a representative set of semantic descriptors for each 3D object via adaptive clustering. To enhance robustness, LaGa assigns weights to descriptors based on two factors: (1) Global alignment, measuring the directional similarity between the descriptor and its global feature, i.e. , the average semantic embedding across all related 2D semantics; and (2) Internal compactness, reflecting the consistency of semantics within the descriptor’s feature cluster. During inference, LaGa selects the highest weighted response of descriptors as the object’s final output, thereby preserving critical information across viewpoints. Despite its simplicity, LaGa significantly bridges the performance gap between 3D and 2D perception on existing benchmarks, achieving +18.7% mIoU over previous 3D methods and surpassing state-of-the-art 2D results by +8.8% mIoU . Our approach offers a new perspective for advancing 3D semantic understanding within the 3D-GS framework. 2 Related Work 2.1 Radiance Fields and 3D Gaussian Splatting Neural Radiance Fields (NeRF) ( Mildenhall et al., 2020 ) pioneer using differentiable rendering for 3D scene reconstruction from multi-view images. Numerous works enhance its rendering quality ( Zhang et al., 2020 ; Martin-Brualla et al., 2021 ; Barron et al., 2021 ; Barron et al., 2022 ; Barron et al., 2023 ) and efficiency ( Sun et al., 2022 ; Chen et al., 2022 ; Lindell et al., 2021 ; Hedman et al., 2024 ; Reiser et al., 2021 ; Müller et al., 2022 ; Wizadwongsa et al., 2021 ; Neff et al., 2021 ; Fridovich-Keil et al., 2022 ) . Recently, 3D Gaussian Splatting (3D-GS) ( Kerbl et al., 2023 ) uses explicit 3D Gaussians as 3D representation and differentiable rasterization for real-time, high-quality rendering. It inspires applications in 3D generation ( Yi et al., 2024 ; Yang et al., 2024a ; Tang et al., 2025 ) , 3D scene editing ( Chen et al., 2024 ; Wang et al., 2024 ) , and 4D scene reconstruction ( Yang et al., 2024b ; Wu et al., 2024a ; Duan et al., 2024 ; Mihajlovic et al., 2025 ) . We build upon 3D-GS to achieve open-vocabulary 3D scene understanding. 2.2 3D Scene Understanding in Radiance Fields Semantic-Agnostic Scene Understanding. The growing popularity of radiance fields leads to research in interactive 3D segmentation and scene decomposition. NVOS ( Ren et al., 2022 ) introduces the first interactive method for object selection in NeRFs. N3F ( Tschernezki et al., 2022 ) , DFF ( Kobayashi et al., 2022 ) , and ISRF ( Goel et al., 2023 ) lift features from 2D self-supervised models ( Caron et al., 2021 ) into 3D via learned feature fields. NeRF-SOS ( Fan et al., 2023 ) and ContrastiveLift ( Bhalgat et al., 2023 ) distill 2D feature similarities into 3D. More recently, with the advent of the Segment Anything Model (SAM) ( Kirillov et al., 2023 ) , studies such as Garfield ( Kim et al., 2024 ) , OmniSeg3D ( Ying et al., 2024 ) , SA3D ( Cen et al., 2023 ) , GaGa ( Lyu et al., 2024 ) , and SAGA ( Cen et al., 2025a ) uses SAM-extracted segmentation masks for scene decomposition. Our approach builds upon this framework by using SAM-extracted masks to construct cross-view semantic connections and decompose the 3D

Read the complete text