Swift4D: Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene

Jiahao Wu, Rui Peng, Zhiyan Wang, Lu Xiao, Jinbo Yan, Kaiqiang Xiong, Ronggang Wang, [email protected] · 2025

Read Swift4D: Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene by Jiahao Wu, Rui Peng, Zhiyan Wang, Lu Xiao, Jinbo Yan, Kaiqiang Xiong, Ronggang Wang, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12307.txt.

Introduction

Jiahao Wu and seven co-authors present Swift4D, a method that rebuilds moving 3D scenes from new viewpoints. Their paper from twenty twenty-five uses 3D Gaussian Splatting, a fast scene format. Past methods slow down on motion and eat storage. Most of a scene is still, the team notes, yet old tools pay the motion cost for all of it.

Swift4D splits the scene's blobs into still and moving sets. A learned tag sorts each blob. Only moving blobs get time-based warps, via a compact 4D hash map. The two sets then merge for the final view. This split speeds training and cuts storage waste. On real data the method matched top quality while training far faster. It needed only a small store, and code is public.

Catalog record

CREATORJiahao Wu, Rui Peng, Zhiyan Wang, Lu Xiao, Jinbo Yan, Kaiqiang Xiong, Ronggang Wang, [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-12307

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Swift4D: Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene Jiahao Wu, Rui Peng, Zhiyan Wang, Lu Xiao, Jinbo Yan, Kaiqiang Xiong, Ronggang Wang, [email protected] Abstract Novel view synthesis has long been a practical but challenging task, although the introduction of numerous methods to solve this problem, even combining advanced representations like 3D Gaussian Splatting, they still struggle to recover high-quality results and often consume too much storage memory and training time. In this paper we propose Swift4D, a divide-and-conquer 3D Gaussian Splatting method that can handle static and dynamic primitives separately, achieving a good trade-off between rendering quality and efficiency, motivated by the fact that most of the scene is the static primitive and does not require additional dynamic properties. Concretely, we focus on modeling dynamic transformations only for the dynamic primitives which benefits both efficiency and quality. We first employ a learnable decomposition strategy to separate the primitives, which relies on an additional parameter to classify primitives as static or dynamic. For the dynamic primitives, we employ a compact multi-resolution 4D Hash mapper to transform these primitives from canonical space into deformation space at each timestamp, and then mix the static and dynamic primitives to produce the final output. This divide-and-conquer method facilitates efficient training and reduces storage redundancy. Our method not only achieves state-of-the-art rendering quality while being 20× faster in training than previous SOTA methods with a minimum storage requirement of only 30MB on real-world datasets. Code is available at https://github.com/WuJH2001/swift4d . Figure 1: Our method demonstrates high-quality rendering, rapid convergence, and compact storage characteristics. It can achieve competitive result with just 5 minutes of training. Additionally, with increased training iterations, our method excels in handling finer details. 1 Introduction Figure 2: Illustration of different dynamic scene rendering methods. (a) Pumarola et al. (2021) ; Park et al. (2021) proposes mapping deformation field points to canonical space, a widely adopted practice in NeRF-based methods; (b) Wu et al. (2024) ; Yang et al. (2024) propose mapping canonical space points to the deformation field; (c) We propose dividing the points in canonical space into dynamic and static, and then mapping only the dynamic points to the deformation space. Novel view synthesis (NVS) is a crucial task in computer vision and graphics, with significant applications in areas such as augmented reality (AR), virtual reality (VR), and content production. The goal of NVS is to render photorealistic images from arbitrary viewpoints using 2D images or video inputs. While recent advancements have achieved considerable success in static scenes, this task becomes particularly challenging when applied to dynamic scenes, where complexities introduced by object motion and temporal changes make accurate rendering significantly difficult. Current NVS techniques can be broadly classified into two predominant approaches: neural rendering methods, exemplified by Neural Radiance Fields (NeRF) Mildenhall et al. (2021) , and point cloud-based rendering techniques, such as 3D Gaussian Splatting Kerbl et al. (2023) . NeRFs have recently made significant strides in achieving photorealistic rendering of static scenes, with subsequent works Barron et al. (2021) ; Barron et al. (2022) ; Barron et al. (2023) ; Reiser et al. (2023) further enhancing both quality and speed. Despite these advancements in static scene rendering, NeRFs face significant challenges when extended to dynamic scenes, primarily due to the substantial training time and storage requirements. To overcome these obstacles, various approaches have been proposed. As shown in Fig. 2 (a), Pumarola et al. (2021) and Park et al. (2021) leverage deformation fields to map deformation space at arbitrary timestamps to canonical space, effectively capturing dynamic scene changes. Li et al. (2021) and Gao et al. (2021) employ scene flow to model the motion trajectories within dynamic environments. Cao & Johnson (2023) and Fridovich-Keil et al. (2023) , decompose the 4D spacetime domain into multiple compact planes, thereby improving training and rendering speeds. Although these methods have achieved some degree of success, achieving high-quality real-time rendering remains challenging. Compared to NeRF, 3DGS offers significant advantages, including real-time rendering and substantially reduced training time. Within the scope of dynamic modeling, several notable methods have emerged. As shown in Fig. 2 (b), 4DGS Wu et al. (2024) , inspired by HexPlane, introduces a neural voxel encoder to model deformation relationships over time. 3DGStream Sun et al. (2024) utilizes a compact Neural Transformation Cache (NTC) to efficiently model the translation and rotation of 3D Gaussians between two adjacent frames. RTGS Yang et al. (2023) treats spacetime as an integrated whole by optimizing a set of 4D primitives, parameterized as anisotropic ellipses that capture both geometry and appearance. Additionally, STGS Li et al. (2024) enhances standard 3D Gaussians with temporal opacity and motion/rotation parameters, effectively capturing both static and dynamic elements to model dynamic deformation. While these approaches achieve higher-quality results with faster rendering times, they still face challenges related to long training time and heavy storage requirements. One potential limitation in their approach is the uniform treatment of all Gaussian points during the modeling process. However, we observe that static points, such as those in background regions, constitute the majority of the scene. These points exhibit minimal or no deformation and therefore do not require complex dynamic modeling. It is more efficient to partition the scene into static and dynamic points and model each separately. This strategy has the potential to significantly reduce computational overhead and storage requirements. Moreover, as demonstrated by Wang et al. (2023) , applying the same modeling technique to both dynamic and static points can cause blurring in dynamic regions due to the influence of static areas, ultimately compromising rendering quality. In this paper, we introduce Swift4D, a method that simultaneously achieves fast convergence, compact storage, and real-time high-quality rendering. Our approach starts by decomposing Gaussian points into dynamic and static groups based on 2D multi-view images, incorporating an additional parameter d for differentiation. For temporal modeling, we employ a deformation field approach using a compact multi-resolution 4DHash and MLPs as the deformer, which maps dynamic Gaussian points from canonical space to deformation space at arbitrary timestamps. Notably, as shown in Fig. 2 (c), temporal modeling is applied exclusively to the dynamic points, while static points are treated as temporally invariant, significantly reducing computational demands. This reduction in the number of dynamic points enables the 4DHash to concentrate on deformation, leading to faster convergence and improved rendering quality. Finally, we combine the static and dynamic Gaussian points to render the final output. This approach also addresses the issue of blurring caused by static elements interfering with the time-aware multi-resolution 4DHash. Our method achieves SOTA performance in terms of training and rendering speed, storage efficiency, and rendering quality. Furthermore, our supplement videos ( basketball 1 and 2) demonstrate that our approach remains effective even in scenarios involving large movements. We will release our code and pre-trained models upon acceptance. In summary, the key contributions of our work are: 1) We propose a novel method for decomposing dynamic 3D scenes into dynamic and static components based on 2D images, effectively reducing computational complexity. This method can be seamlessly integrated into existing dynamic approaches as a plug-and-play module to enhance quality. 2) We introduce a compact multi-resolution 4DHash, with a footprint as small as 8MB, to

Read the complete text