Virtualized 3D Gaussians: Flexible Cluster-based Level-of-Detail System for Real-Time Rendering of Composed Scenes

Xijie Yang, Linning Xu, Lihan Jiang, Dahua Lin, Bo Dai · 2025

Read Virtualized 3D Gaussians: Flexible Cluster-based Level-of-Detail System for Real-Time Rendering of Composed Scenes by Xijie Yang, Linning Xu, Lihan Jiang, Dahua Lin, Bo Dai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06523.txt.

Introduction

Xijie Yang, Linning Xu, Lihan Jiang, Dahua Lin, and Bo Dai present a system for drawing huge three dimensional scenes in real time in a 2025 paper. A popular method builds scenes from millions of small Gaussian blobs, which is too slow when scenes grow to crowd scale. Their system groups blobs into clusters arranged in a hierarchy of detail levels.

An offline step builds the clusters so each level looks close to the next. At run time the system picks only clusters a viewer can notice, based on their screen footprint. Tests on synthetic and real scenes, from trees to buildings, show a good balance of speed and image quality.

Catalog record

CREATORXijie Yang, Linning Xu, Lihan Jiang, Dahua Lin, Bo Dai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06523

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Virtualized 3D Gaussians: Flexible Cluster-based Level-of-Detail System for Real-Time Rendering of Composed Scenes Xijie Yang, Linning Xu, Lihan Jiang, Dahua Lin, Bo Dai Abstract . 3D Gaussian Splatting (3DGS) enables the reconstruction of intricate digital 3D assets from multi-view images by leveraging a set of 3D Gaussian primitives for rendering. Its explicit and discrete representation facilitates the seamless composition of complex digital worlds, offering significant advantages over previous neural implicit methods. However, when applied to large-scale compositions, such as crowd-level scenes, it can encompass numerous 3D Gaussians, posing substantial challenges for real-time rendering. To address this, inspired by Unreal Engine 5’s Nanite system, we propose Virtualized 3D Gaussians (V3DG), a cluster-based LOD solution that constructs hierarchical 3D Gaussian clusters and dynamically selects only the necessary ones to accelerate rendering speed. Our approach consists of two stages: (1) Offline Build, where hierarchical clusters are generated using a local splatting method to minimize visual differences across granularities, and (2) Online Selection, where footprint evaluation determines perceptible clusters for efficient rasterization during rendering. We curate a dataset of synthetic and real-world scenes, including objects, trees, people, and buildings, each requiring 0.1 billion 3D Gaussians to capture fine details. Experiments show that our solution balances rendering efficiency and visual quality across user-defined tolerances, facilitating downstream interactive applications that compose extensive 3DGS assets for consistent rendering performance. Conference: Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers; August 10–14, 2025; Vancouver, BC, Canada Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH Conference Papers ’25), August 10–14, 2025, Vancouver, BC, Canada DOI: 10.1145/3721238.3730602 ISBN: 979-8-4007-1540-2/2025/08 CCS: Computing methodologies Rendering CCS: Computing methodologies Point-based models CCS: Computing methodologies Rasterization CCS: Computing methodologies Machine learning approaches Figure 1. Left : Collection of digital 3DGS assets . Right : Optimized 3D Gaussians ( Ours ) in composed scene Forest with adaptive LOD selection based on distance, resolution, and user-defined tolerance, minimizing overhead while maintaining visual quality. We report number (percentage) and FPS (acceleration rate) of selected 3D Gaussians at various distances ( Near , Middle , and Far ), showing consistent real-time performance compared to vanilla 3DGS composition. teaser 1. INTRODUCTION 3D Gaussian Splatting (3DGS) [ Kerbl et al., 2023 ] reconstructs objects and scenes with explicit 3D Gaussian primitives from multi-view images, achieving high visual fidelity and real-time rendering, offering multiple benefits compared to previous NeRF-based methods [ Mildenhall et al., 2021 ] . The reconstructed objects and scenes in 3D Gaussian representation, which we refer to as 3DGS assets , can be further used by digital artists to compose digital worlds . While individual 3DGS assets consisting of millions of explicit 3D Gaussians can be rendered in real-time with the GPU rasterizer [ Kerbl et al., 2023 ] , rendering a complex, large-scale digital world with numerous assets and 3D Gaussian in real time remains a significant challenge for interactive applications. This limitation arises because 3DGS-based methods rely solely on visibility-based filtering to select the visible primitives, assuming that these primitives share the same level of importance. As a composed scene grows with more 3DGS assets, primitives in the camera frustum become redundant and overlapping, causing a low rendering speed. To solve this problem, a Level-of-Detail (LOD) system is necessary to control the rendering overhead dynamically under given rendering resolution and camera distance. Inspired by the Nanite Virtualized Geometry system [ Karis et al., 2021 ] in Unreal Engine 5, which efficiently handles a large amount of geometric detail through dynamic LOD streaming, we propose Virtualized 3D Gaussians (V3DG), an LOD system for real-time rendering of a large number of 3D Gaussians in composed scenes. Specifically, our system comprises two stages: an offline build stage and an online selection stage . (1) The offline build stage acts as a one-time post-processing step for each asset. Each 3DGS asset is first split into clusters, forming the finest layer. Then, these clusters are simplified using a novel local splatting strategy to build coarser layers with minimal difference in a tree structure, establishing parent-child relationships across levels of detail for selection. The multi-level cluster data is stored as a bundle for the selection stage. (2) The online selection stage is embedded in each rendering process to lower rendering overhead. Before the rasterization of 3D Gaussians, clusters from all bundles are projected onto the 2D screen space to derive their footprints based on the camera view, measuring the level of detail for each cluster. Clusters at appropriate levels of detail are then selected to meet a predefined footprint tolerance , ensuring only necessary 3D Gaussians are rasterized further, which matches the detail of 3D Gaussians with the given rendering conditions, achieving faster rasterization and anti-aliasing effects. We further curate a dataset consisting of four composed scenes with thousands of detailed 3DGS assets, covering both synthetic and real-world data, to better evaluate our method and advance the field. Comprehensive experiments show that 3DGS struggles with real-time rendering of these challenging scenes, while our approach boosts rendering speeds under predefined tolerances, preserving visual quality and incorporating anti-aliasing for enhanced comfort. Users can also balance quality and speed by adjusting tolerances. To summarize, our contributions are as follows: • A post-processing cluster-based LOD system for rendering approximately 100 million 3D Gaussians in digital worlds in real-time, with a build stage constructing hierarchical clusters and a selection stage that filters necessary clusters. • A local splatting method that simplifies clusters across layers with minimal visual difference agnostic to training views. • A selection strategy based on footprint evaluation, achieving high-quality rendering with the minimally required number of 3D Gaussians. • A large-scale dataset with composed scenes containing various synthetic and real-world assets for evaluation of LOD system on 3D Gaussians. 2. RELATED WORK Figure 2. Framework of our virtualized LOD system , featuring the offline build stage to prepare clusters at various levels of detail, and the online selection stage to select clusters at appropriate levels of detail given rendering conditions. framework 2.1. 3D Gaussian Splatting for Reconstruction Recently, 3D Gaussian Splatting (3DGS) [ Kerbl et al., 2023 ] has made significant advancements in the Novel View Synthesis (NVS) field by employing explicit 3D Gaussian primitives for reconstruction from captured multi-view images, achieving state-of-the-art rendering quality and speed compared to Neural Radiance Field (NeRF) -based works [ Mildenhall et al., 2021 ; Barron et al., 2022 ; Müller et al., 2022 ; Fridovich-Keil et al., 2022 ] . Each 3D Gaussian primitive is parameterized by a set of attributes, including position \boldsymbol{p} , scale \boldsymbol{s} , rotation quaternion \boldsymbol{q} , opacity o , and view-dependent color \boldsymbol{c} . Rendering an image requires first splatting 3D Gaussians onto the image plane [ Zwicker et al., 2001 ] and then accumulating colors of 2D splats overlapping each pixel using alpha-blending. In order to capture fine details of objects or scenes being reconstructed, vanilla 3DGS, its variants [ Lu et al., 2024 ; Yu et al., 2024 ] to better the representation, and some compression works [ Niemeyer et al., 2024 ; Fan et al., 2024 ; Niedermayr et al., 2024 ; Papantonakis et al., 2024 ; Lee et al., 2024 ; Navaneet et al., 2023 ; Chen et al.,

Read the complete text