4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives

Zeyu Yang, Zijie Pan, Xiatian Zhu, Li Zhang, Jianfeng Feng, Yu-Gang Jiang, Philip H.S. Torr · 2024

Read 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives by Zeyu Yang, Zijie Pan, Xiatian Zhu, Li Zhang, Jianfeng Feng, Yu-Gang Jiang, Philip H.S. Torr (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20720.txt.

Description

A 2024 arXiv research paper by Zeyu Yang, Zijie Pan, Xiatian Zhu, Li Zhang, Jianfeng Feng, Yu-Gang Jiang, Philip H.S. Torr. Dynamic 3D scene representation and novel view synthesis are crucial for enabling immersive experiences required by AR/VR and metaverse applications. It is a challenging task due to the complexity of unconstrained real-world scenes and their temporal dynamics. In this paper, we reformulate the reconstruction of a time-varying 3D scene as approximating its underlying spatiotemporal 4D volume by optimizing a collection of native 4D primitives, i.e.

Catalog record

CREATORZeyu Yang, Zijie Pan, Xiatian Zhu, Li Zhang, Jianfeng Feng, Yu-Gang Jiang, Philip H.S. Torr
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20720

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives Zeyu Yang, Zijie Pan, Xiatian Zhu, Li Zhang, Jianfeng Feng, Yu-Gang Jiang, Philip H.S. Torr Abstract Dynamic 3D scene representation and novel view synthesis are crucial for enabling immersive experiences required by AR/VR and metaverse applications. It is a challenging task due to the complexity of unconstrained real-world scenes and their temporal dynamics. In this paper, we reformulate the reconstruction of a time-varying 3D scene as approximating its underlying spatiotemporal 4D volume by optimizing a collection of native 4D primitives, i.e. , 4D Gaussians, with explicit geometry and appearance modeling. Equipped with a tailored rendering pipeline, our representation can be end-to-end optimized using only photometric supervision while free viewpoint viewing at interactive frame rate, making it suitable for representing real world scene with complex dynamic. This approach has been the first solution to achieve real-time rendering of high-resolution, photorealistic novel views for complex dynamic scenes. To facilitate real-world applications, we derive several compact variants that effectively reduce the memory footprint to address its storage bottleneck. Extensive experiments validate the superiority of 4DGS in terms of visual quality and efficiency across a range of dynamic scene-related tasks (e.g., novel view synthesis, 4D generation, scene understanding) and scenarios (e.g., single object, indoor scenes, driving environments, synthetic and real data). keywords Dynamic scenes, Novel view synthesis, Gaussian splatting, Neural video synthesis † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction Modeling dynamic scenes and rendering novel views is crucial in computer vision and graphics, which has received increasing attention from both industry and academia because of its potential value in a wide range of AR/VR applications. Whilst recent breakthroughs (e.g., NeRF Mildenhall et al. (2020) ) have achieved photorealistic rendering for static scenes Barron et al. (2021) ; Verbin et al. (2022) , handling dynamic scenes remains a significant challenge. The temporal dynamics of scenes introduce additional complexity. The central challenge lies in preserving intrinsic correlations and sharing relevant information across different time steps to create a compact representation, while minimizing interference between unrelated spacetime locations. Figure 1: Schematic illustration of (a) our proposed spatio-temporal 4D volume learning framework for dynamic scenes, corresponding to (b) the breakdown process of our 4D Gaussian Splatting model, which transitions from 4D Gaussian representations to 2D Planar Gaussian forms in space and time. Existing novel view synthesis methods for dynamic scenes can be categorized into two groups. The first group directly learns parametric 6D plenoptic functions using implicit neural models, employing structures such as MLPs Li et al. (2022) , grids Wang et al. (2023) , or their low-rank decompositions Fridovich-Keil et al. (2023) ; Cao and Johnson (2023) ; Attal et al. (2023) , without explicitly modeling scene motion. Their ability to capture correlations across varying spatiotemporal locations relies on the intrinsic properties of the selected data structure. As a result, these methods may experience interference due to shared parameters across spatiotemporal locations or struggle to leverage the inherent correlations arising from object motion effectively. The second group posits that scene dynamics underlie the motion and deformation of a canonical 3D model Pumarola et al. (2021) ; Song et al. (2023) ; Abou-Chakra et al. (2022) ; Luiten et al. (2024) . These approaches explicitly model scene motion, potentially improving the utilization of spatial and temporal correlations. However, global tracking of object motion involves inherent ambiguities and risks singularities when dealing with complex motions in real-world scenes, such as sudden appearances and disappearances, which limits their capability and scalability in unconstrained environments. To address these fundamental limitations, in this work we reformulate the modeling of dynamic scenes as a generic, spatio-temporal 4D volume learning problem. This novel perspective not only introduces an explicit representation , but also makes minimal assumptions about how motion is composed , paving the way for a versatile dynamic scene learning framework . Concretely, we propose to use a native 4D primitive 4D Gaussian that models the geometry and motion of a time-varying 3D scene at the same time. The adopted 4D Gaussian parameterized by anisotropic ellipses that can rotate arbitrarily in space and time fits naturally within the intrinsic spatio-temporal manifold. Furthermore, we introduce Spherindrical Harmonics, a generalized form of Spherical Harmonics, to represent the time evolution of appearance in dynamic scenes, along with a dedicated splatting-based differentiable rendering pipeline. Our 4DGS model is the first to support end-to-end training and real-time rendering of high-resolution, photorealistic novel views of complex dynamic scenes. We make the following contributions : (i) We propose to use a set of anisotropic 4D Gaussians equipped with 4D Spherindrical Harmonics to represent the dynamic 3D scene, which is the first native 4D primitive. (ii) We cast the novel view synthesis of the dynamic scene as to optimize the proposed 4D Gaussian primitives for direct fitting the underlying spatio-temporal volume, and introduce the necessary differentiable rasterization pipeline. (iii) Extensive experiments validate the superiority of 4DGS over previous alternatives in terms of visual quality and efficiency across a variety of dynamic scene-based tasks ( e.g. , novel view synthesis, content generation, scene understanding) and scenarios ( e.g. , single object, indoor and outdoor scenes, synthetic and real data). A preliminary version of this work has been presented Yang et al. (2024) . Moving beyond our initial focus on novel view synthesis, here we significantly extend and explore the proposed 4DGS from a versatile dynamic scene representation perspective and manage to address the storage bottleneck. In comparison, we summarize several important improvements: (i) Introducing more compact variants: There is a trade-off between compactness and versatility governed by how much assumption of the scene’s dynamic hold by the representation. We prioritize the latter while leaving room to inject motion priors via regularization. As a result, the size of vanilla 4DGS may increase rapidly with the inherent large 4D volume. However, this is not a fundamental limitation. We introduce several model compression strategies across multiple dimensions, including parameter size, the number of Gaussian points, that substantially alleviate storage demands and reduce memory footprint. (ii) Application extension: We first extend our model to handle large-scale dynamic urban scenes, which present additional challenges such as sparse viewpoints and unbounded environments. We also enhance the ability to create 4D content from monocular video via a generative approach. Finally, we demonstrate the proposed representation can enhance time-consistent scene understanding, providing object segmentation across both space and time. (iii) Extensive evaluation: We conduct comprehensive evaluations of model compression and the newly added applications on more benchmarks. 2 Related work 2.1 Novel view synthesis Novel view synthesis for static scenes The field of novel view synthesis has received widespread attention initiated by NeRF Mildenhall et al. (2020) . Querying a MLP for hundreds of points along each ray, NeRF is significantly limited in training and rendering. Subsequent works attempted to improve the speed Chen et al. (2022) ; Sun et al. (2022) ; Hu et al. (2022) ; Chen et al. (2023) ; Fridovich-Keil et al. (2022) ; Müller et al. (2022) and enhance rendering quality Zhang et al. (2020) ; Verbin et al. (2022) ; Barron et al. (2021) ; Barron et al. (2022)

Read the complete text