Beyond Heuristics: Learnable Density Control for 3D Gaussian Splatting

Zhenhua Ning, Xin Li, Jun Yu, Guangming Lu, Yaowei Wang, Wenjie Pei · 2026

Read Beyond Heuristics: Learnable Density Control for 3D Gaussian Splatting by Zhenhua Ning, Xin Li, Jun Yu, Guangming Lu, Yaowei Wang, Wenjie Pei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00408.txt.

Introduction

Zhenhua Ning and 5 co-authors replace hand-written rules with a learned policy for a core chore in 3D Gaussian Splatting, the fast rendering method that builds scenes from millions of soft blobs. Deciding where to add or remove blobs has relied on heuristic density control. Refined rules still cannot adapt to every scene's geometry, the authors argue.

Their framework, LeGS, recasts density control as a policy network trained by reinforcement learning. A reward grounded in sensitivity analysis measures each blob's marginal contribution to reconstruction quality, and a closed-form shortcut keeps that reward cheap to compute even for huge scenes. On three standard datasets of indoor and outdoor scenes, LeGS beat leading methods, striking a better balance between image quality and efficiency. The work suggests learned control can retire a whole family of brittle hand-tuned rules.

Catalog record

CREATORZhenhua Ning, Xin Li, Jun Yu, Guangming Lu, Yaowei Wang, Wenjie Pei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00408

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Heuristics: Learnable Density Control for 3D Gaussian Splatting Zhenhua Ning, Xin Li, Jun Yu, Guangming Lu, Yaowei Wang, Wenjie Pei Abstract While 3D Gaussian Splatting (3DGS) has demonstrated impressive real-time rendering performance, its efficacy remains constrained by a reliance on heuristic density control. Despite numerous refinements to these handcrafted rules, such methods inherently lack the flexibility to adapt to diverse scenes with complex geometries. In this paper, we propose a paradigm shift for density control from rigid heuristics to fully learnable policies. Specifically, we introduce LeGS , a framework that reformulates density control as a parameterized policy network optimized via Reinforcement Learning (RL). Central to our approach is the tailored effective reward function grounded in sensitivity analysis, which precisely quantifies the marginal contribution of individual Gaussians to reconstruction quality. To maintain computational tractability, we derive a closed-form solution that reduces the complexity of reward calculation from O ⁡ ( N 2 ) O(N^{2}) to O ⁡ ( N ) O(N) . Extensive experiments on the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets demonstrate that LeGS significantly outperforms state-of-the-art methods, striking a superior balance between reconstruction quality and efficiency. The code will be released at https://github.com/AaronNZH/LeGS . Figure 1 : Comparison between our LeGS and FastGS. The heuristic paradigm, relying on a manually-designed score function with a fixed threshold, achieves suboptimal results. For instance, FastGS tends to under-densify Gaussians in wheel (red box), causing artifacts. In contrast, our learning-based paradigm effectively captures intricate local details. 1 Introduction 3D Gaussian Splatting (3DGS) ( 10 ) has emerged as a premier technique for 3D scene reconstruction, leveraging 3D Gaussians to achieve high-fidelity scene representation alongside real-time rendering speeds. A pivotal step of 3DGS is adaptive density control, a mechanism that dynamically refines the Gaussian distribution—specifically their quantity, spatial density, and geometric parameters—to optimize the scene representation. Most existing methods adopt a heuristic paradigm for adaptive density control centered on two fundamental operations: densification and pruning. Original 3DGS performs densification by cloning or splitting Gaussians that exhibit high positional gradients, while pruning ineffective Gaussians via thresholding of opacity values to maintain computational efficiency. Despite its simplicity, such rigid heuristics often fail to generalize across diverse and complex scenarios, frequently resulting in rendering artifacts such as blurriness and Gaussian redundancies. In light of this, recent works have investigated several refinements to this heuristic density control mechanism. Specifically, Pixel-GS ( 39 ) normalizes positional gradients based on pixel coverage to alleviate under-reconstructions, while FastGS ( 24 ) targets regions exhibiting high rendering errors. Taming-3DGS ( 19 ) and Perceptual-GS ( 40 ) incorporate saliency and perceptual priors to emphasize visually significant Gaussians. While these methods have demonstrated substantial improvements in both representation quality and efficiency, a significant limitation of the heuristic density control paradigm remains: the reliance on handcrafted rules and predefined hyperparameters. Such rigid designs inherently constrain generalizability, as the optimal thresholds for densification and pruning often vary significantly across diverse scenes with complex geometric conditions and varying scales. As evidenced by Figure 1 , the density control in FastGS is agnostic to regional geometry and texture, failing to adaptively allocate Gaussians according to the underlying geometric or textural complexity. To circumvent the aforementioned limitations of heuristic density control, we propose an entirely different paradigm: Learnable Density Control for Gaussian Splatting ( LeGS ). In contrast to existing heuristic-driven frameworks, LeGS formulates density control as a learned policy parameterized by a neural network. By determining optimal refinement operations at each iteration in a data-driven manner, our approach enables superior adaptation to diverse scenes with complex geometries and textures. Specifically, by characterizing the iterative adaptive density control operations as a sequence of discrete actions—specifically maintain , clone , split , and prune —and treating Gaussian attributes as interactive states, the optimization of LeGS can be naturally formulated as a Reinforcement Learning (RL) problem. The design of the reward function is essential to the efficacy of the Reinforcement Learning process in terms of both the reconstruction fidelity and convergence efficiency. We draw inspiration from sensitivity analysis ( 37 ) and design an effective reward function for our LeGS . The core idea is to evaluate the reconstruction impact of each Gaussian by measuring the sensitivity of the rendering quality to each Gaussian. Specifically, we quantify each Gaussian’s contribution by aggregating its impact on rendering quality across its entire projected footprint. Nevertheless, a naive implementation of reward calculation is computationally prohibitive. Since evaluating the marginal contribution of each Gaussian necessitates a full rendering pass, the resulting O(N^{2}) complexity—where N is the number of Gaussians—imposes an unsustainable overhead on the training process. To address this, we derive a closed-form solution for the rendering output that simulates the ablation of individual Gaussians, leveraging intermediate rendering states. This efficient-yet-exact formulation circumvents the need for actual re-rendering, allowing us to compute the sensitivity in constant time ( O(1) ) per Gaussian. Consequently, the overall complexity is reduced to O(N) . Given that this reward is computed periodically (e.g., every 100 iterations), the cumulative computational overhead remains negligible. To conclude, the contributions of this work are summarized as follows: • We identify the inherent limitations of heuristic-based density control in 3DGS and propose LeGS , a novel learnable paradigm that reformulates density control as a parameterized policy network within a reinforcement learning (RL) framework. By replacing rigid handcrafted rules with a data-driven and optimization-driven strategy, LeGS dynamically optimizes Gaussian densification and pruning in a scene-adaptive and context-aware manner. • We design a novel reward function grounded in sensitivity analysis to precisely quantify the marginal contribution of each Gaussian to reconstruction quality, thereby providing effective learning signals for reinforcement learning. Furthermore, we derive an exact and efficient formulation for reward computation, which avoids exhaustive re-rendering and reduces the computational complexity from quadratic O(N^{2}) to linear O(N) . • Extensive experiments demonstrate that our method achieves state-of-the-art performance on the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets. Qualitative results further show that our method yields reconstructions better aligned with scene textures. In addition, the efficiency analysis shows that our method offers a favorable trade-off between rendering efficiency and reconstruction quality. 2 Related Work 2.1 3D Gaussian Splatting for Novel View Synthesis Novel view synthesis aims to generate photorealistic images from arbitrary viewpoints given a set of calibrated input images. Neural radiance fields (NeRF) ( 20 ) and subsequent variants ( 1 ; 2 ; 31 ; 7 ; 30 ; 22 ; 23 ; 21 ; 34 ) have achieved remarkable rendering quality by modeling scenes with implicit neural representations and differentiable volume rendering. However, their reliance on dense ray marching and volumetric integration often leads to substantial computational overhead, limiting their practicality for real-time or large-scale applications. To overcome these limitations, 3D Gaussian Splatting (3DGS) ( 10 ) introduces an explicit representation based on anisotropic 3D Gaussians, which can be rendered efficiently through differentiable rasterization. Thanks to its high rendering quality and real-time performance, 3DGS has rapidly inspired a wide range of follow-up studies. Recent works have extended 3DGS to dynamic scene reconstruction ( 18 ; 33 ; 35 ; 5 ) , sparse-view and few-shot novel view synthesis ( 41 ; 3 ) , surface reconstruction and geometry

Read the complete text