HRGS: Hierarchical Gaussian Splatting for Memory-Efficient High-Resolution 3D Reconstruction

Changbai Li, Haodong Zhu, Hanlin Chen, Juan Zhang, Tongfei Chen, Shuo Yang, Shuwei Shao, Wenhao Dong, Baochang Zhang · 2025

Read HRGS: Hierarchical Gaussian Splatting for Memory-Efficient High-Resolution 3D Reconstruction by Changbai Li, Haodong Zhu, Hanlin Chen, Juan Zhang, Tongfei Chen, Shuo Yang, Shuwei Shao, Wenhao Dong, Baochang Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14229.txt.

Introduction

Changbai Li and co-authors made high-resolution 3D scene rebuilding fit in limited memory in 2025. Gaussian splatting renders scenes in real time, but sharp, large scenes blow past memory limits. Their method, HRGS, works in a hierarchy.

A coarse set of Gaussians is first built from low-resolution data. The scene is then cut into blocks, each refined with sharp data and only the views that touch it. A pruning step scores each Gaussian's worth and drops the near-useless ones, speeding training and saving memory. Surface hints from a pretrained model sharpen the geometry. On three benchmarks the method matched the best quality scores while staying inside tight memory budgets.

Catalog record

CREATORChangbai Li, Haodong Zhu, Hanlin Chen, Juan Zhang, Tongfei Chen, Shuo Yang, Shuwei Shao, Wenhao Dong, Baochang Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-14229

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HRGS: Hierarchical Gaussian Splatting for Memory-Efficient High-Resolution 3D Reconstruction Changbai Li, Haodong Zhu, Hanlin Chen, Juan Zhang, Tongfei Chen, Shuo Yang, Shuwei Shao, Wenhao Dong, Baochang Zhang Abstract 3D Gaussian Splatting (3DGS) has achieved significant progress in real-time 3D scene reconstruction. However, its application in high-resolution reconstruction scenarios faces severe memory scalability bottlenecks. To address this issue, we propose Hierarchical Gaussian Splatting (HRGS), a memory-efficient framework with hierarchical block-level optimization from coarse to fine. Specifically, we first derive a global, coarse Gaussian representation from low-resolution data; we then partition the scene into multiple blocks and refine each block using high-resolution data. Scene partitioning comprises two steps: Gaussian partitioning and training data partitioning. In Gaussian partitioning, we contract irregular scenes into a normalized, bounded cubic space and employ a uniform grid to evenly distribute computational tasks among blocks; in training data partitioning, we retain only those observations that lie within their corresponding blocks or make significant contributions to the rendering results. By guiding each block’s refinement with the global coarse Gaussian prior, we ensure alignment and seamless fusion of Gaussians across adjacent blocks. To reduce computational resource demands, we introduce an Importance-Driven Gaussian Pruning (IDGP) strategy: during each block’s refinement, we compute an importance score for every Gaussian primitive and remove those with minimal rendering contribution, thereby accelerating convergence and reducing redundant computation and memory overhead. To further enhance surface reconstruction quality, we also incorporate normal priors from a pretrained model. Finally, even under memory-constrained conditions, our method enables high-quality, high-resolution 3D scene reconstruction. Extensive experiments on three public benchmarks demonstrate that our approach achieves state-of-the-art performance in high-resolution novel view synthesis (NVS) and surface reconstruction tasks. Figure 1: (a) High-resolution ( \sim 5K) renderings of the “bicycle” scene from the Mip-NeRF 360 dataset, with results from 3DGS, Mip-Splatting, and our method. Red dashed boxes highlight key details. (b) Performance on this scene: our method achieves the highest PSNR (25.6 dB) with significantly lower GPU memory (23 GB) and model size (567 MB) than 3DGS and Mip-Splatting. (c) Average resource usage across the full dataset shows our method maintains the smallest memory and model footprint. 1 Introduction 3D scence reconstruction remains a longstanding challenge in computer vision and graphics. A significant advancement in this domain is the Neural Radiance Field (NeRF) nerf , which effectively represents geometry and view-dependent appearance using multi-layer perceptrons (MLPs), demonstrating significant advancements in 3D reconstruction quality. Recently, 3D Gaussian Splatting (3DGS) 3dgs has gained considerable attention as a compelling alternative to MLP-based nerf and feature grid-based representations tensorf ; plenoxels ; Neural ; Instant_neural . 3DGS stands out for its impressive results in 3D scene reconstruction and novel view synthesis while achieving real-time rendering at 1K resolutions. This efficiency and effectiveness, combined with the potential integration into the standard GPU rasterization pipeline, marks a significant step toward the practical adoption of 3D reconstruction methods. In particular, 3DGS models complex scenes as a collection of 3D Gaussian distributions, which are projected onto screen space using splatting-based rasterization. The characteristics of each 3D Gaussian, including position, size, orientation, opacity, and color, are optimized using multi-view photometric loss. Although 3DGS has demonstrated impressive 3D reconstruction results, its application in high-resolution scenarios encounters critical memory scalability limitations. Specifically,when reconstructing outdoor scenes at ultra-high resolutions approaching 5K (e.g.4978 \times 3300 pixels) in standardized benchmark datasets like Mip-NeRF 360 Mip-NeRF360 , conventional 3DGS implementations demand excessive VRAM, exceeding the capacity of mainstream GPUs with limited memory, such as the NVIDIA A5000 (24GB VRAM). This computational bottleneck arises from the increasing resolution: higher resolutions demand more GPU memory, as illustrated in Fig. 1 . Such algorithmic behavior fundamentally conflicts with finite GPU memory resources, resulting in catastrophic memory overflow during optimization phases. To overcome these critical memory constraints while preserving reconstruction fidelity for high-resolution scene reconstruction, we present Hierarchically Gaussian Splatting (HRGS), a memory-efficient framework with hierarchical block optimization from coarse to fine. Specifically, we first obtain a coarse global Gaussian representation using low-resolution images. Subsequently, to minimize memory usage on a single GPU, we partition the scene into spatially adjacent blocks and parallelly refined. Each block is represented with fewer Gaussians and trained on reduced data, allowing further optimization with high-resolution images. The partitioning strategy operates at two levels: Gaussian primitives and training data. To achieve a more balanced partition of Gaussians and avoid blocks with sparse Gaussians, we begin by contracting unbounded Gaussians. In detail, we define a bounded cubic region and use its boundary to normalize the Gaussian positions. Within this region, Gaussians are contracted via a linear mapping, while those outside undergo nonlinear contraction, yielding a more compact Gaussian representation. We then apply a uniform grid subdivision strategy to this contracted space, ensuring an even distribution of computational tasks. During data partitioning for training, we compute the SSIM loss mss for each observation by comparing two renderings:One rendering is implemented with the complete global Gaussian representation, while the other is executed subsequent to the elimination of Gaussians within the target block. A more pronounced SSIM loss denotes that the observation exerts a more substantial contribution to the target block, so we set a threshold on SSIM loss and retain only observations whose values exceed it. To mitigate artifacts at the block boundaries, we further include observations that fall within the region of the considered block. Finally, to prevent overfitting, we employ a binary search algorithm during data partitioning to expand each block until the number of Gaussians it contains exceeds a specified threshold. This innovative strategy effectively reduces interference from irrelevant data while improving fidelity with decreased memory usage, as demonstrated in Tab. C . After partitioning the Gaussian primitives and data, we initialize each block in the original, uncontracted space using the coarse global Gaussian representation. To accelerate convergence and reduce computational overhead during block-level refinement with high-resolution data, we introduce an Importance-Driven Gaussian Pruning (IDGP) strategy. Specifically, we evaluate the interaction between each Gaussian and the multi-view training rays within the corresponding block, and discard those with negligible rendering contributions. All blocks are then refined in parallel, and subsequently integrated into a unified, high-resolution global Gaussian representation. In addition to novel view synthesis, we evaluate our method on another key subtask of 3D reconstruction: 3D surface reconstruction. To further enhance the quality of the reconstructed surfaces, we incorporate the View-Consistent Depth-Normal Regularizer vcR , which is applied both during the initialization of the coarse global Gaussian representation and throughout the subsequent block-level refinement. Finally, our method enables high-quality and high-resolution scene reconstruction even under constrained memory capacities (e.g., NVIDIA A5000 with 24GB VRAM). We validate our method on two sub-tasks of 3D reconstruction: high-resolution NVS and surface reconstruction, and demonstrate that it delivers superior high-resolution reconstruction performance. In summary, the main contributions of this paper are: • We propose HRGS, a memory-efficient coarse to fine framework that leverages low-resolution global Gaussians to guide high-resolution local Gaussians refinement, enabling high-resolution scene reconstruction with limited GPU memory. • A novel partitioning strategy for Gaussian primitives and data is introduced, optimizing memory usage, reducing irrelevant data interference, and enhancing reconstruction fidelity. • We propose a novel dynamic pruning strategy, Importance-Driven Gaussian Pruning (IDGP), which evaluates the contribution of each Gaussian

Read the complete text