Sketch&Patch++: Efficient Structure-Aware 3D Gaussian Representation

Yuang Shi, Géraldine Morin, Simone Gasparini, Wei Tsang Ooi · 2026

Read Sketch&Patch++: Efficient Structure-Aware 3D Gaussian Representation by Yuang Shi, Géraldine Morin, Simone Gasparini, Wei Tsang Ooi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05394.txt.

Introduction

Yuang Shi and 3 co-authors present a 2026 method for shrinking 3D Gaussian Splatting models, which store scenes as many small fuzzy blobs for realistic rendering. Their system sorts those blobs by role. Some mark sharp edges and outlines, while others fill smooth, flat areas.

Inside, a clustering step groups blobs by density and then refines the groups by quality, with no need for outside line drawings. The outline group can be sent first during streaming, so a scene skeleton appears before fine detail arrives. Tests on indoor and outdoor scenes report clearer images at the same file size, and indoor scenes kept good quality at a tiny fraction of the original size.

Catalog record

CREATORYuang Shi, Géraldine Morin, Simone Gasparini, Wei Tsang Ooi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE95 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05394

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sketch&Patch++: Efficient Structure-Aware 3D Gaussian Representation Yuang Shi, Géraldine Morin, Simone Gasparini, Wei Tsang Ooi Abstract 3D Gaussian Splatting (3DGS) has emerged as a promising representation for photorealistic rendering of 3D scenes. However, its high storage requirements pose significant challenges for practical applications such as remote visualization, collaborative 3D content sharing, and bandwidth-constrained streaming in extended reality systems. We observe that Gaussians exhibit distinct roles and characteristics analogous to traditional artistic techniques — like how artists first sketch outlines before filling in broader areas with color, some Gaussians capture high-frequency features such as edges and contours, while others represent broader, smoother regions analogous to brush strokes that add volume and depth. Based on this observation, we propose a hybrid representation that categorizes Gaussians into (i) Sketch Gaussians , which represent high-frequency, boundary-defining features, and (ii) Patch Gaussians , which cover low-frequency, smooth regions. This semantic separation naturally enables layered progressive streaming, where the compact Sketch Gaussians establish the structural skeleton before Patch Gaussians incrementally refine volumetric detail. In our previous work, we demonstrated the effectiveness of this categorization for man-made scenes by identifying Sketch Gaussians using 3D line segments extracted via Line3D++. However, this line-based approach was fundamentally limited to scenes with linear features. In this work, we extend our method to arbitrary 3D scenes by proposing a novel hierarchical adaptive categorization framework that operates directly on the 3DGS representation. Our approach employs multi-criteria density-based clustering, combined with adaptive quality-driven refinement. This method eliminates dependency on external 3D line primitives while ensuring optimal parametric encoding effectiveness. After categorization, Sketch Gaussians are efficiently encoded using parametric models leveraging their coherence, while Patch Gaussians undergo optimized pruning, retraining, and quantization. Our comprehensive evaluation across diverse scenes, including both man-made and natural environments, demonstrates that our method achieves up to 1.74 dB 1.74\text{\,}\mathrm{dB} improvement in PSNR, 6.7 % 6.7\text{\,}\mathrm{\%} in SSIM, and 41.4 % 41.4\text{\,}\mathrm{\%} in LPIPS at equivalent model sizes compared to uniform pruning baselines. For indoor scenes, our method can maintain visual quality with only 0.5 % 0.5\text{\,}\mathrm{\%} of the original model size, representing a 175 × \times compression ratio. This structure-aware representation enables efficient storage, adaptive streaming, and rendering of high-fidelity 3D content across bandwidth-constrained networks and resource-limited devices. Figure 1: We propose Sketch&Patch++, a hybrid Gaussian representation with 3D structure prior, significantly reducing the storage of 3DGS model by an order of magnitude while maintaining the visual quality. 1 Introduction 1.1 Background and Motivation The increasing demand for immersive experiences in extended reality (XR) applications, such as virtual reality (VR), augmented reality (AR), and cloud gaming, has driven significant progress in 3D scene representation technologies. These applications rely on accurately modeling complex 3D environments to deliver high-quality user experiences. Classical 3D representations such as meshes and point clouds face challenges in balancing storage efficiency with rendering quality, particularly for large-scale environments where point clouds can introduce visual artifacts and holes during rendering [ 24 , 52 ] . Recently, learning-based methods have given rise to alternative representations that seamlessly model geometry and appearance. 3D Gaussian Splatting (3DGS) [ 24 ] has emerged as a leading approach for learning more explicit and efficient point-based 3D representations while keeping the mix between geometry and appearance, leading to a very satisfying visual quality. Compared to point clouds, 3DGS represents 3D scenes with a collection of 3D ellipsoidal Gaussian splats (or Gaussians for short), which can mitigate the presence of holes with a much sparser distribution. By leveraging the properties of Gaussians and taking advantage of modern GPUs, 3DGS achieves comparable photorealistic quality to NeRF [ 38 ] but with significantly improved rendering speed, showing great promise as a foundational representation for future immersive multimedia systems and applications. While 3DGS achieves impressive visual fidelity, it still generates vast numbers of Gaussians to capture fine geometric and appearance details, with five attributes stored independently. For instance, Kerbl et al . [ 24 ] employ one to five million Gaussians to model static scenes, demanding up to 1\text{\,}\mathrm{GB} of storage per scene. Such a heavy representation, while suitable for local rendering on powerful hardware, poses significant challenges for remote rendering, sharing, and real-time transmission of 3DGS models across networks. The substantial storage requirements of 3DGS models have motivated continuous efforts in compression. Numerous recent works (see Section 2.3 ) have proposed various compression strategies, achieving significant size reductions while maintaining visual quality. However, compression alone is insufficient for practical deployment, as efficient delivery of 3DGS content requires adapting compressed representations to streaming pipelines that enable progressive transmission and adaptive quality based on network conditions and viewing requirements [ 54 , 29 ] . Common point-based streaming strategies, such as 3D downsampling or spatial tiling, are widely used to generate scalable and multi-resolution representations for 3D streaming [ 15 , 56 , 51 , 48 , 45 ] . Given that 3DGS is also a point-based representation ( i.e . the centers of the Gaussians), it is natural to consider adapting these techniques for Gaussians. However, these streaming techniques are difficult to apply to 3DGS due to the unique properties of its Gaussian-based representation and optimization. Unlike conventional point clouds, 3DGS employs heterogeneous Gaussians parameterized by ellipsoid shape and view-dependent color, with attributes optimized iteratively to minimize discrepancies between rendered and ground-truth images, leading to a non-uniform and highly interdependent distribution. This tightly coupled optimization means that spatial partitioning or downsampling techniques often result in substantial visual quality degradation [ 50 , 53 , 54 , 55 , 26 ] , as the interdependent Gaussians cannot be meaningfully decomposed without retraining. Furthermore, most existing compression methods treat 3DGS as a monolithic representation without semantic structure, making it challenging for progressive/scalable streaming. 1.2 Key Observations (a) 3DGS (Relaxed, \tau=0.01 ). (b) 3DGS (Medium, \tau=0.002 ). (c) 3DGS (Strict, \tau=0.0002 ). Figure 2: Impact of densification on 3DGS models. (a) Relaxed ( \tau=0.01 ), (b) medium ( \tau=0.002 ), and (c) strict ( \tau=0.0002 ) densification thresholds produce 10K, 27K, and 589K Gaussians respectively. Top row: rendered images with PSNR values. Middle row: per-pixel reconstruction error maps. Bright regions indicate higher reconstruction error, while darker areas correspond to well-reconstructed regions. Bottom row: Gaussian distributions. Note the correlation between the density of Gaussians and the sharp/smooth areas. 3DGS’s adaptive density control mechanism introduces storage inefficiencies. The adaptive density control aims to enhance visual fidelity by cloning Gaussians in under-reconstructed regions and splitting those in over-reconstructed areas, based on 2D positional gradients. And the reliance on 2D gradients often highlight the need for densification in high-frequency and boundary-defining areas, such as edges and contours [ 49 ] . As a result, 3DGS naturally forms dense clusters of Gaussians along structural boundaries and detailed areas, while smooth regions such as walls remain sparsely populated. This adaptive behavior is essential for capturing fine details and avoiding artifacts. However, it also leads to a highly non-uniform distribution where many Gaussians convey redundant information in locally dense regions [ 27 , 32 , 36 ] . For better understanding, we visualize the impact of adaptive density control in Figure 2 , which compares three 3DGS models trained with different densification thresholds \tau . Here, we vary this parameter to produce three levels of density:

Read the complete text