FreeTimeGS: Free Gaussian Primitives at Anytime and Anywhere for Dynamic Scene Reconstruction

Yifan Wang, Peishan Yang, Zhen Xu, Jiaming Sun, Zhanhua Zhang, [2pt] Yong Chen, Hujun Bao, Sida Peng, Xiaowei Zhou, [5pt] Zhejiang University · 2025

Read FreeTimeGS: Free Gaussian Primitives at Anytime and Anywhere for Dynamic Scene Reconstruction by Yifan Wang, Peishan Yang, Zhen Xu, Jiaming Sun, Zhanhua Zhang, [2pt] Yong Chen, Hujun Bao, Sida Peng, Xiaowei Zhou, [5pt] Zhejiang University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05348.txt.

Introduction

Wang and co-authors let their virtual building blocks be born anywhere, at any time. Rebuilding moving 3D scenes for novel-view video usually starts from a fixed set of Gaussian blobs in a canonical pose, warped by a deformation field; complex motion breaks that scheme. Their method, FreeTimeGS, instead allows each primitive to appear at an arbitrary moment and place.

Each blob also carries its own motion function, drifting into neighboring regions as time passes, which trims redundancy. The result is a flexible four-dimensional representation that bends with the scene instead of fighting it. On several benchmark data sets, the rendering quality beats recent methods by a wide margin. The authors provide a project page with examples.

Catalog record

CREATORYifan Wang, Peishan Yang, Zhen Xu, Jiaming Sun, Zhanhua Zhang, [2pt] Yong Chen, Hujun Bao, Sida Peng, Xiaowei Zhou, [5pt] Zhejiang University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05348

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FreeTimeGS: Free Gaussian Primitives at Anytime and Anywhere for Dynamic Scene Reconstruction Yifan Wang, Peishan Yang, Zhen Xu, Jiaming Sun, Zhanhua Zhang, [2pt] Yong Chen, Hujun Bao, Sida Peng, Xiaowei Zhou, [5pt] Zhejiang University Abstract This paper addresses the challenge of reconstructing dynamic 3D scenes with complex motions. Some recent works define 3D Gaussian primitives in the canonical space and use deformation fields to map canonical primitives to observation spaces, achieving real-time dynamic view synthesis. However, these methods often struggle to handle scenes with complex motions due to the difficulty of optimizing deformation fields. To overcome this problem, we propose FreeTimeGS, a novel 4D representation that allows Gaussian primitives to appear at arbitrary time and locations. In contrast to canonical Gaussian primitives, our representation possesses the strong flexibility, thus improving the ability to model dynamic 3D scenes. In addition, we endow each Gaussian primitive with an motion function, allowing it to move to neighboring regions over time, which reduces the temporal redundancy. Experiments results on several datasets show that the rendering quality of our method outperforms recent methods by a large margin. Project page: https://zju3dv.github.io/freetimegs/ Figure 1 : Photorealistic and real-time rendering of dynamic 3D scenes. Our method achieves the best performance on challenging dynamic scenes with fast and complex motion. Compared with current state-of-the-art methods 4DGS Yang et al. 2023b and STGS Li et al. 2023 , our PSNR is improved by 2.4dB and 1.4dB on the SelfCap dataset. For dynamic regions, our PSNR is improved by 4.1dB and 2.6dB. What’s more, our method supports real-time rendering at 1080p resolution with a speed of 450 FPS using a single RTX 4090 GPU. † † ∗ Equal contribution. † Corresponding author: Xiaowei Zhou. 1 Introduction Dynamic view synthesis aims to produce novel views of a dynamic 3D scene from captured multi-view videos, which has a wide range of applications, such as movie production, video games, and virtual reality. Traditional approaches Hilsmann et al. 2020 ; Casas et al. 2014 ; Collet et al. 2015 ; Newcombe et al. 2015 ; Dou et al. 2016 ; Orts-Escolano et al. 2016 ; Yu et al. 2018 use sequences of textured meshes to represent dynamic 3D scenes. Such representation requires complicated hardware setups for high-quality reconstruction, thus making traditional approaches limited to controlled environments. NeRF-based methods Mildenhall et al. 2021 ; Fridovich-Keil et al. 2023 ; Li et al. 2022b have achieved impressive results in dynamic view synthesis by modeling scenes as neural implicit representations. However, these representations are computationally expensive, leading to slow rendering speed and hindering practical applications. Recently, a popular paradigm for dynamic view synthesis is to equip 3D Gaussian primitives Yang et al. 2023a ; Wu et al. 2023 ; Shaw et al. 2023 ; Guo et al. 2024 ; Bae et al. 2024 ; Liang et al. 2023 ; Lu et al. 2024 ; Labe et al. 2024 ; Liu et al. 2024 ; Zhu et al. 2024 with deformation fields to model dynamic scenes. These methods model the geometry and appearance of the scene based on Gaussian primitives in canonical space and then use MLP networks to model scene motions for deforming the canonical-space scene to observation-space scenes at particular moments. Although these methods achieve real-time and high-quality rendering performance on scenes with small motions, they often struggle to handle scenes with complex motions. A plausible reason is that when objects move much in the scene, these methods need to build long-range correspondences between canonical space and observation spaces, which is difficult to be recovered from RGB observations, as discussed in Park et al. 2021a ; Li et al. 2021 . In this paper, we propose a novel 4D representation, named FreeTimeGS, for reconstructing dynamic 3D scenes with complex motions. In contrast to previous methods Yang et al. 2023a ; Wu et al. 2023 that define Gaussian primitives at canonical space only, FreeTimeGS allows Gaussian primitives to appear at arbitrary positions and time steps, possessing the strong flexibility. In addition, we assign an explicit motion function to each Gaussian primitive, allowing it to move to neighboring regions over time, which facilitates the reuse of Gaussian primitives along the temporal dimension and reduces the representation redundancy. By endowing Gaussian primitives high degrees of freedom, our representation has two advantages. First, this significantly improves the ability to model dynamic 3D scenes and the rendering quality, as demonstrated in our experiments. Second, we only need to model short-range motions between Gaussian primitives and observed scenes, compared with deformation-based methods Yang et al. 2023a ; Wu et al. 2023 . Therefore, our motion function can be implemented as a linear function, alleviating the ill-posed optimization problem. During experiments, we find that optimizing FreeTimeGS with only rendering loss tend to get trapped in local minima on fast-moving regions, leading to poor rendering quality. To tackle this issue, we examine the opacity distribution of Gaussian primitives and discover that a considerable part of it approaches 1. The result suggests that high opacity of some Gaussian primitives may prevent the gradient from back-propagating to all Gaussian primitives and block the optimization process. Motivated by this observation, we design a simple regularization strategy to penalize the high opacity of Gaussian primitives in the early stage of optimization, which effectively mitigates the local minima problem and improves our rendering quality. To validate the effectiveness of our method, we evaluate FreeTimeGS on multiple widely used datasets for multiview dynamic novel view synthesis, including Neural3DV Li et al. 2022a and ENeRF-Outdoor Lin et al. 2022 . Our method achieves highest quality on these public datasets compared with existing state-of-the-art methods. To further evaluate and demonstrate the capability of our method on challenging scenes, we also collect a dataset with much faster and more complex motions compared with Neural3DV Li et al. 2022a . Our method achieves best quality on it compared with the SOTA methods by a large margin on both quality and efficiency. Figure 2 : Pipeline Overview. We represent a dynamic scene using Gaussian primitives that can appear anytime anywhere. Each Gaussian is assigned with a motion function to to model its movement. And its opacity is modulated by the temporal opacity function which control the impact of the Gaussian primitive over time. With this 4D representation, we further regularize the Gaussians with a 4D regularization loss and optimize the rasterization result with rendering loss for reconstructing a dynamic 3D scene from multi-view videos. 2 Related Work Dynamic scene reconstruction with RGB-D cameras In the early days, dynamic scene reconstruction leverages RGB-D cameras to capture the dynamic geometry and appearance of a scene. Methods pioneered by DynamicFusion Newcombe et al. 2015 use depth sensors to build a TSDF model in canonical space. Each frame applies non-rigid tracking to create a deformation graph, enabling TSDF fusion in canonical space to integrate reconstruction results across frames. Fusion4D Dou et al. 2016 and Holoportation Orts-Escolano et al. 2016 further refined this system, achieving highly impressive dynamic reconstruction results. Subsequent works, such as DoubleFusion Yu et al. 2018 and BodyFusion Yu et al. 2017 , introduced human body priors, enhancing the robustness of non-rigid tracking. The main bottleneck of the aforementioned methods lies in the lack of robustness and inaccuracy

Read the complete text