RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos

Junmyeong Lee * Hoseung Choi * Yoonwoo Jeong *, Minsu ChoPohang University of Science and Technology (POSTECH) · 2024

Read RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos by Junmyeong Lee * Hoseung Choi * Yoonwoo Jeong *, Minsu ChoPohang University of Science and Technology (POSTECH) (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03077.txt.

Introduction

Junmyeong Lee, Hoseung Choi, Yoonwoo Jeong, and Minsu Cho rebuild moving scenes from casual phone videos. Their paper, from twenty twenty-four, presents RoDyGS. A single moving camera leaves the true 3D motion unclear. Their method paints the scene with soft 3D blobs that shift over time, a method called Gaussian splatting.

Static and moving parts are kept apart by design. Rules on space and time keep the motion physically sane. The team also built a new test set with rich camera and object motion and true multi-view capture. Their method beat past pose-free dynamic methods by a clear margin. Still scenes render at a level close to top static methods.

Catalog record

CREATORJunmyeong Lee * Hoseung Choi * Yoonwoo Jeong *, Minsu ChoPohang University of Science and Technology (POSTECH)
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03077

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos Junmyeong Lee * Hoseung Choi * Yoonwoo Jeong *, Minsu ChoPohang University of Science and Technology (POSTECH) Abstract 4D reconstruction from casually captured monocular videos is challenging due to inherent ambiguity in reconstructing dynamic 3D geometry. To address this challenge, we introduce Robust Dynamic Gaussian Splatting (RoDyGS), a method that reconstructs dynamic scene representation from casual monocular videos. RoDyGS explicitly separates static and dynamic scene elements, and applies spatiotemporal regularization to enforce physically plausible geometry and temporally consistent motion. Furthermore, we propose a comprehensive benchmark, Kubric-MRig, which provides extensive camera and object motion along with simultaneous multi-view capture, features that are absent in previous benchmarks. Experiments demonstrate that RoDyGS significantly outperforms previous pose-free dynamic novel view synthesis approaches and achieves competitive rendering quality compared to existing pose-free static novel view synthesis approaches. Our proejct page is available at https://rodygs.github.io * * footnotetext: Equal contribution. 1 Introduction Figure 1 : Robust Dynamic Gaussian Splatting (RoDyGS). RoDyGS reconstructs the spatiotemporally consistent geometry of objects and background, rendering high-fidelity dynamic scenes from casually captured monocular videos. Everyday environments are filled with complex and dynamic objects, yet we typically capture them using monocular handheld videos. Such videos provide rich visual appearance but lack 3D structure and temporal information, thereby limiting downstream tasks requiring spatial and dynamic scene understanding, such as AR, VR, and robotics. Accurate 4D reconstruction is therefore essential for recovering complete scene geometry and motion from casual monocular videos. In this paper, we propose a novel end-to-end framework that performs 4D reconstruction directly from monocular videos, without relying on external sensors or manual calibration. Despite recent advances in 4D reconstruction [ 46 , 44 , 45 , 40 , 62 ] , reconstructing accurate scene geometry and motion from videos remains challenging due to two major problems. First, the ambiguity of camera-object motion complicates the joint estimation of scene geometry and camera trajectory. Second, occlusions result in missing observations, disrupting spatiotemporal consistency in scene elements. To address these issues, recent work has relied on specialized capture setups [ 16 , 40 ] , limited scene conditions [ 46 , 45 , 62 ] , or domain-specific assumptions [ 53 , 18 ] . While these methods show promising results, they still face limitations when applied to general dynamic scenes casually captured with a monocular camera. In this work, we propose RoDyGS , a dynamic scene reconstruction method for casually captured monocular videos that jointly optimizes dynamic Gaussian splatting and camera motion. A key aspect of RoDyGS is reducing inherent ambiguity in dynamic-scene reconstruction from monocular videos. We design a part-separation method that explicitly divides static and dynamic scene elements, ensuring stability and enabling part-aware regularization. For dynamic elements, we propose spatiotemporal regularization to preserve geometric consistency and natural motion. Moreover, we jointly optimize camera poses with static regions, ensuring temporally coherent scene representation. Additionally, existing benchmarks face challenges in evaluating both pose estimation and rendering quality due to the lack of ground truth (GT) camera poses or concurrent multi-view captures. To overcome these limitations, we introduce a challenging benchmark, Kubric-MRig , which includes photorealistic scenes with a variety of simultaneously captured viewpoints and extensive camera and object movement. The experiments show that RoDyGS outperforms prior pose-free dynamic novel view synthesis methods on Kubric-MRig dataset and achieves competitive performance on iPhone dataset [ 15 ] . Furthermore, it achieves rendering quality comparable to pose-free static novel view synthesis methods on Tanks and Temples dataset [ 30 ] . In summary, our contributions are as follows: 1. We propose RoDyGS, a robust reconstruction method for casually captured monocular videos that achieves high-fidelity rendering through part separation and spatiotemporal regularization. 2. We present Kubric-MRig, a challenging dataset designed to evaluate both pose estimation and rendering quality in dynamic scenes, addressing limitations found in existing benchmarks. 3. Experiments demonstrate that RoDyGS achieves state-of-the-art performance and competitive results across novel view synthesis benchmarks. 2 Related work 2.1 Static Novel View Synthesis Novel View Synthesis (NVS) is a task of generating novel viewpoints from a set of observations. Pioneering work in NVS leverages point clouds [ 31 , 73 , 64 ] , meshes [ 48 , 49 ] , and planes [ 22 ] for geometrically convincing view synthesis. Recently, NeRF [ 41 ] has achieved ground-breaking rendering quality by representing volumetric scene functions via MLPs. To accelerate the training and inference speed, subsequent research focuses on baking trained NeRFs [ 21 ] or directly optimizing explicit representations [ 12 , 55 , 43 ] . More recently, 3D Gaussian Splatting (3DGS) [ 27 ] has generated novel views by rasterizing anisotropic 3D Gaussians onto image planes. By employing a tile-based alpha-blending CUDA technique, 3DGS achieves real-time rendering with cutting-edge quality on NVS benchmarks. Building upon 3DGS, several methods have been introduced to enhance fidelity [ 28 , 69 ] , enable training with sparse views [ 63 , 72 ] , and support editing [ 5 , 7 ] . However, these methods rely on pre-calibrated camera poses, limiting their applicability to in-the-wild captures. To address pose-free scenarios and enhance scalability, iNeRF [ 70 ] estimates camera poses from a pre-trained NeRF by minimizing the discrepancy between the query and rendered views. In subsequent research, [ 61 , 24 , 38 , 6 , 2 ] optimize both camera poses and NeRF parameters using photometric loss and geometric regularization. Additionally, CF-3DGS [ 13 ] introduces a progressively growing 3DGS for camera pose estimation, while InstantSplat [ 9 ] leverages DUSt3R [ 60 ] to initialize camera poses and point clouds in 3DGS. Yet, these approaches are based on static scenes, which struggle to reconstruct scenes containing typical moving objects found in casual videos. 2.2 Dynamic Novel View Synthesis Following the success of NVS in static scenes, researchers have extended neural fields to capture both the underlying geometry and motion of dynamic scene. Building on NeRF, [ 46 , 44 , 45 ] learn additional time-varying deformation fields. In parallel, [ 37 , 11 , 4 ] have taken a different approach by focusing on learning multi-dimensional feature fields. These methods encode scene dynamics without relying on explicit motion representations. With the emergence of 3DGS, [ 40 , 62 ] propose learning the trajectories of individual Gaussians over time. Subsequent research has introduced more efficient representations, such as factorized motion basis [ 32 ] and sparse control points [ 23 ] . Another line of work [ 67 ] extends spherical harmonics to a 4D spherindrical harmonics function, integrating both time and view-dependent components. Despite their reliable performance, these methods assume a limited data distribution for constrained optimization, requiring pre-calibrated poses and functioning under specific conditions, such as camera teleportation or ambient dynamics, thus restricting their applicability to general scenarios in casual videos. RoDynRF [ 39 ] introduces a pose-free optimization method for dynamic radiance fields using a voxelized feature space, enabling reconstruction from casually captured videos. However, it remains limited to forward-facing scenes or small viewpoint changes, restricting its applicability to arbitrary videos. To address these limitations, we propose RoDyGS, which extends applicability to a wider range of videos through separated optimization and dynamic part-aware regularization techniques. Moreover,

Read the complete text