SD-GS: Structured Deformable 3D Gaussians for Efficient Dynamic Scene Reconstruction
Wei Yao 1⋆ , Shuzhao Xie 1⋆ , Letian Li 1 , Weixiang Zhang 1 , Zhixin Lai 2 , Shiqi Dai 3 , Ke Zhang 4 , Zhi Wang 1† 1 SIGS, Tsinghua University 2 Google 3 Department of CST, Tsinghua University 4 Soochow University {yaow21, xsz24, lilt24, zhang-wx22}@mails.tsinghua.edu.cn, [email protected], dai · 2025
Read SD-GS: Structured Deformable 3D Gaussians for Efficient Dynamic Scene Reconstruction by Wei Yao 1⋆ , Shuzhao Xie 1⋆ , Letian Li 1 , Weixiang Zhang 1 , Zhixin Lai 2 , Shiqi Dai 3 , Ke Zhang 4 , Zhi Wang 1† 1 SIGS, Tsinghua University 2 Google 3 Department of CST, Tsinghua University 4 Soochow University {yaow21, xsz24, lilt24, zhang-wx22}@mails.tsinghua.edu.cn, [email protected], dai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07465.txt.
Introduction
Wei Yao and seven co-authors present SD-GS for dynamic scene reconstruction in two thousand twenty-five. Four dimensional Gaussian methods render motion with high fidelity and speed. Storage cost and complex motion still limit their real use. Their frame shrinks the model while it tracks hard motion.
A deformable anchor grid holds the scene in a layered, lean form. Each anchor spawns several Gaussians in its local space and time. A densification step grows anchors where motion is under built. It trims anchors in static zones at the same time. Model size and frame rate both improved a lot in tests.
Catalog record
| CREATOR | Wei Yao 1⋆ , Shuzhao Xie 1⋆ , Letian Li 1 , Weixiang Zhang 1 , Zhixin Lai 2 , Shiqi Dai 3 , Ke Zhang 4 , Zhi Wang 1† 1 SIGS, Tsinghua University 2 Google 3 Department of CST, Tsinghua University 4 Soochow University {yaow21, xsz24, lilt24, zhang-wx22}@mails.tsinghua.edu.cn, [email protected], dai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07465
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SD-GS: Structured Deformable 3D Gaussians for Efficient Dynamic Scene Reconstruction Wei Yao 1⋆ , Shuzhao Xie 1⋆ , Letian Li 1 , Weixiang Zhang 1 , Zhixin Lai 2 , Shiqi Dai 3 , Ke Zhang 4 , Zhi Wang 1† 1 SIGS, Tsinghua University 2 Google 3 Department of CST, Tsinghua University 4 Soochow University {yaow21, xsz24, lilt24, zhang-wx22}@mails.tsinghua.edu.cn, [email protected], [email protected], [email protected], [email protected] Abstract Current 4D Gaussian frameworks for dynamic scene reconstruction deliver impressive visual fidelity and rendering speed, however, the inherent trade-off between storage costs and the ability to characterize complex physical motions significantly limits the practical application of these methods. To tackle these problems, we propose SD-GS, a compact and efficient dynamic Gaussian splatting framework for complex dynamic scene reconstruction, featuring two key contributions. First, we introduce a deformable anchor grid, a hierarchical and memory-efficient scene representation where each anchor point derives multiple 3D Gaussians in its local spatiotemporal region and serves as the geometric backbone of the 3D scene. Second, to enhance modeling capability for complex motions, we present a deformation-aware densification strategy that adaptively grows anchors in under-reconstructed high-dynamic regions while reducing redundancy in static areas, achieving superior visual quality with fewer anchors. Experimental results demonstrate that, compared to state-of-the-art methods, SD-GS achieves an average of 60% reduction in model size and an average of 100% improvement in FPS, significantly enhancing computational efficiency while maintaining or even surpassing visual quality. Figure 1 : Our method successfully achieves photorealistic quality and high resolution rendering in real time while maintaining a compact model size. (a) Our approach can be rendered at high resolution with 82 FPS on an Nvidia RTX 3090 GPU. (b) Quantitative comparisons of rendering quality, speed, and storage requirements with several state-of-the-art baselines on the N3DV Dataset. $\star$ $\star$ footnotetext: Equal contribution. $\dagger$ $\dagger$ footnotetext: Corresponding author. 1 Introduction Dynamic scene reconstruction from multi-view videos is an important task in 3D computer vision, with tremendous applications in AR, VR, and 3D content creation [ 31 ] . While Neural Radiance Fields (NeRFs) [ 17 , 32 , 1 , 26 , 24 ] have made notable progress in dynamic scene reconstruction, 3D Gaussian Splatting (3DGS)-based methods [ 14 ] have emerged as the dominant approach. This advantage stems from two key factors: first, 3DGS employs explicit geometric representations that naturally facilitate dynamic modeling; second, its highly optimized CUDA rasterization pipeline eliminates the need for intensive sampling and querying of neural fields [ 43 , 44 , 4 ] , significantly accelerating both rendering and training. Recent 3DGS-based dynamic scene representations fall into two categories: 1) Explicit approaches, which extend the 3D Gaussian to 4D Gaussian primitives by adding a temporal dimension to approximate the spatiotemporal 4D volume of dynamic scenes [ 39 , 9 ] . Although 4D Gaussians achieve higher visual quality and faster rendering speed, these works suffer from substantial storage requirements for numerous Gaussians and their 4D parameters as they cannot leverage inherent cross-spatiotemporal correlations. 2) Implicit approaches, which employ consistent deformable 3D Gaussians as the underlying structure to characterize scenes, interpreting motion at each timestamp as a deformation of the underlying structure of different magnitudes [ 35 , 38 , 2 ] . These deformation-based methods offer a more compact representation than explicit methods by leveraging intrinsic correlations across spatial and temporal conditions through deformations, enabling effective cross-spatiotemporal information sharing. However, these implicit representations struggle to handle complex real-world motions and generally exhibit slower rendering speeds. Motivated by these challenges, we introduce SD-GS for dynamic scene reconstruction to address the balance between storage efficiency and the capability to model complex real-time motions. Our framework extends the anchor-based scaffold representation from static scenes to deformable 3D Gaussian-based dynamic scene reconstruction. Since local 3D Gaussians in spatiotemporal domains typically possess similar feature information, we propose to use deformable 3D structured anchors initialized from a sparse grid of SfM points as the underlying scene representation, achieving a more compact structure compared to explicit methods. The attributes of local neural 3D Gaussians can be predicted from anchor features adapted to various timestamps and viewing angles. Furthermore, previous densification strategies in static scene reconstruction accumulated Gaussian gradients and applied growth in underfit regions based on predefined gradient thresholds. When applied to dynamic scene reconstruction, this approach tends to generate numerous redundant Gaussians in static regions, accompanied by increased deformation calculation overhead and lower FPS, while producing insufficient anchors in dynamic regions, leading to poor reconstruction quality in these areas. To overcome this limitation, we propose a deformation-aware densification strategy to achieve adaptive and efficient anchor allocation, directing new anchors toward poorly reconstructed high-dynamic regions instead of static scene parts. In summary, our contributions are as follows: 1) We introduce structured 3D Gaussians and a meticulously designed time-aware architecture to model dynamic scenes, significantly reducing the model size. 2) We propose a deformation-aware densification method to further enhance the representation ability of the deformation grid in complex dynamics, which also reduces storage costs. 3) Extensive experiments demonstrate that our approach significantly reduces storage requirements by 60% on average and achieves 2x faster rendering speed, while maintaining or even exceeding state-of-the-art visual quality. 2 Related Work 2.1 Dynamic 3D Gaussians Based on the formulation of the movement of the objects, the existing Dynamic 3D Gaussians can be divided into explicit and implicit methods based on the representation of time. The explicit methods [ 39 , 9 , 18 ] are built on the 4D Gaussians, with one more dimension representing the timestamp, which requires substantial memory for training and rendering. In contrast, the implicit methods [ 35 , 2 , 38 , 22 , 13 ] employ the deformation grid to model the movement, which greatly utilize the spatiotemporal correlations to reduce the memory and storage requirement. For instance, D-3DGS [ 22 ] models dynamic scenes by allowing the positions and rotation matrices of 3DGS to change over time. Deformable 3DGS [ 38 ] uses an MLP to model a deformation field based on time and the canonical Gaussian space. SC-GS [ 13 ] bounds dense 3DGS with sparse control points, calculating the movement of Gaussians in a coarse-to-fine manner. However, these implicit methods ignore the inner redundancy of the canonical model, 3D Gaussians. Thus, we introduce a scaffold representation to replace the 3D Gaussians, which further reduces the memory requirement. Besides, we carefully design a deformation grid to enable the high-quality reconstruction for each timestamp. Although contemporary works [ 7 , 16 ] also utilize scaffold representations, they employ a different deformation strategy from us. For example, Scaffold4D [ 7 ] still keeps the temporal dimension of the 4D Gaussians, while our method utilizes a memory-efficient deformation grid. 2.2 Gaussian Densification Gaussian densification plays a pivotal role in recovering accurate scene geometry. The vanilla 3DGS approach initializes sparse points from structure-from-motion (SfM) [ 29 ] and employs adaptive density control [ 14 ] , first selecting Gaussians based on image-space gradients and then cloning or splitting them according to their scale. Intuitively, a well-designed densification strategy can enhance optimization convergence, accelerating the overall training process. Most existing methods, with limited exceptions such as [ 10 , 15 , 23 ] , prioritize improving rendering quality, often