High-Speed Dynamic 3D Imaging with Sensor Fusion Splatting

Zihao Zou, Ziyuan Qu, Xi Peng, Vivek Boominathan, Adithya Pediredla, Praneeth Chakravarthula · 2025

Read High-Speed Dynamic 3D Imaging with Sensor Fusion Splatting by Zihao Zou, Ziyuan Qu, Xi Peng, Vivek Boominathan, Adithya Pediredla, Praneeth Chakravarthula (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04630.txt.

Introduction

Zihao Zou and co-authors fuse three camera types to image fast moving three dimensional scenes in a 2025 paper. Capturing deforming scenes at high speed serves graphics, robotics, aerodynamics, and biology. A single sensor type falls short, since color cameras are slow, have limited exposure, and offer narrow baselines for depth.

Their method joins color, depth, and event cameras, which add fine color, true geometry, and microsecond change sensing. The scene is held as deformable three dimensional Gaussians, a splat based form. All sensor data jointly tune the Gaussian parameters and their motion fields over time. The fusion yields efficient, high quality imaging even in low light or rapid motion.

Catalog record

CREATORZihao Zou, Ziyuan Qu, Xi Peng, Vivek Boominathan, Adithya Pediredla, Praneeth Chakravarthula
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-04630

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

High-Speed Dynamic 3D Imaging with Sensor Fusion Splatting Zihao Zou, Ziyuan Qu, Xi Peng, Vivek Boominathan, Adithya Pediredla, Praneeth Chakravarthula Abstract . Capturing and reconstructing high-speed dynamic 3D scenes has numerous applications in computer graphics, vision, and interdisciplinary fields such as robotics, aerodynamics, and evolutionary biology. However, achieving this using a single imaging modality remains challenging. For instance, traditional RGB cameras suffer from low frame rates, limited exposure times, and narrow baselines. To address this, we propose a novel sensor fusion approach using Gaussian splatting, which combines RGB, depth, and event cameras to capture and reconstruct deforming scenes at high speeds. The key insight of our method lies in leveraging the complementary strengths of these imaging modalities: RGB cameras capture detailed color information, event cameras record rapid scene changes with microsecond resolution, and depth cameras provide 3D scene geometry. To unify the underlying scene representation across these modalities, we represent the scene using deformable 3D Gaussians. To handle rapid scene movements, we jointly optimize the 3D Gaussian parameters and their temporal deformation fields by integrating data from all three sensor modalities. This fusion enables efficient, high-quality imaging of fast and complex scenes, even under challenging conditions such as low light, narrow baselines, or rapid motion. Experiments on synthetic and real datasets captured with our prototype sensor fusion setup demonstrate that our method significantly outperforms state-of-the-art techniques, achieving noticeable improvements in both rendering fidelity and structural accuracy. (a) (b) Figure 1 . High-speed Imaging with Sensor Fusion Splatting. We propose a method to reconstruct high-speed, deforming 3D scene geometries by integrating data from three complementary imaging modalities—RGB, event, and depth cameras, as shown in (a). Using these inputs, we train a set of deformable Gaussians to dynamically represent the scene. This approach enables detailed, time-resolved reconstructions of the scene in challenging conditions such as low light, high dynamic range or rapid scene motion and from a limited baseline (see (b)), allowing novel view synthesis at any timestamp. teaser 1. Introduction Capturing high-speed 3D phenomena—such as the flapping wings of a hummingbird, explosions, or industrial processes—has wide-ranging applications in science and engineering. However, designing imaging systems for these scenarios is challenging due to their extremely high bandwidth demands and limited baseline geometries. Ultra-high-speed cameras (e.g., Phantom, Photron, IDT), although can achieve remarkable frame rates, are prohibitively expensive (over $100k), offer short capture durations (only a few seconds), lack depth information, and generate massive data volumes. Currently, no single imaging modality can capture dynamic scenes with high spatial, temporal, and depth resolutions. In this work, we address this limitation by integrating three complementary imaging modalities—RGB cameras, event cameras, and depth cameras—which collectively provide spatial, temporal, and depth information. RGB cameras are inexpensive and provide high 2D spatial resolution along with color information about the scene. However multiple RGB cameras struggle to achieve high-resolution depth information in limited baseline scenarios ( Qadri et al., 2024 ; Qu et al., 2024 ) , due to the missing cone problem. Additionally, RGB cameras face challenges in low-light conditions and are constrained by bandwidth limitations, resulting in low frame rates. Event cameras, in contrast, offer ultra-high temporal resolutions (down to microseconds) by capturing only pixel-level changes in brightness. They transmit asynchronous sparse data, known as events, which significantly reduces data redundancy and storage demands. Furthermore, their high dynamic range (HDR) enables them to capture scenes with extreme lighting contrasts, which would otherwise appear washed out or darkened with conventional RGB cameras. These features make event cameras an excellent complement to RGB cameras for capturing high-speed scenes. However, neither of these cameras provide depth information, which complicates reconstruction in small baseline scenarios. Depth sensors are shown to sample data within the missing cone, capturing essential information that RGB and event cameras cannot ( Qu et al., 2024 ) . Active depth cameras, such as SPADs ( Gupta et al., 2019b ; Folden et al., 2024 ; Gupta et al., 2019a ; Jungerman et al., 2022 ; Po et al., 2022 ) or CWToF sensors ( Keetha et al., 2024 ; Friday et al., 2024 ; He and Chen, 2019 ; Shrestha et al., 2016 ) , measure the distance between the camera and the scene, providing necessary depth information for accurate reconstruction, especially in small baseline conditions. In this work, we integrate information from all three complementary sensory modalities to enable reconstruction of high-speed 3D scenes in small baseline scenarios. As shown in Figure 1(a) , the image representation of each modality differ significantly: RGB cameras provide color images, event cameras provide pixel-level intensity changes at extremely high speed, and depth cameras generate depth maps. Fusing this sensory information requires a shared scene representation. To achieve this, we represent the underlying scene using deformable 3D Gaussians that are shared across all imaging modalities. We render data from RGB, event and depth cameras through a Gaussian splatting framework, and jointly optimize the 3D Gaussian parameters and their temporal deformation model by minimizing the differences between the rendered splatting outputs and the observed multi-view, multi-sensory data. This approach enables reconstruction of high-resolution multiview images and depth maps, achieving enhanced spatial, temporal, and angular fidelity. We rigorously evaluate our approach on diverse synthetic scenes and real-world scenes captured with our sensor fusion prototype (shown in Figure 2 ), including challenging scenarios with rapid motion and low-light conditions—scenarios where traditional cameras or single modalities often fail (see Figure 1(b) ). Our results demonstrate that the proposed multi-sensor fusion approach significantly improves reconstructions, delivering both fidelity and robustness in capturing high-speed, dynamic scenes. In summary, we make the following contributions: (1) We propose Sensor Fusion Splatting, a novel deformable 3D Gaussian splatting framework that integrates RGB, event, and depth data streams to enable robust, high-speed 3D scene reconstruction and consistent photorealistic novel view synthesis from multi-sensor inputs. (2) We develop a hardware sensor fusion prototype that combines RGB, event, and depth cameras to capture high-speed deformable 3D scenes, supporting our proposed method. (3) We rigorously evaluate our approach on diverse synthetic and real-world datasets featuring high-speed, deforming 3D scenes, demonstrating significant performance improvements in scene reconstruction. Our code and data will be made public upon acceptance of the manuscript. 2. Related Works We propose a multi-sensor fusion of RGB, event, and depth data using shared deformable Gaussian splatting representation. Our technique builds on recent advances in neural rendering, sensor fusion, and high-speed imaging. We outline these advancements and how our method builds upon or differs from them. For further details, we refer readers to resources on neural rendering ( Xie et al., 2022 ; Tewari et al., 2022 ; Fei et al., 2024 ) , event camera applications ( Gallego et al., 2020 ) , and depth sensing ( Horaud et al., 2016 ; Szeliski, 2022 ) . 2.1. Neural Rendering Neural radiance field (NeRF) introduced by Mildenhall et al. (2021) and its extensions ( Barron et al., 2021 ; Zhang et al., 2020 ) represent 3D scenes implicitly using multilayer perceptrons (MLPs) for inverse differentiable rendering via analysis-by-synthesis. This method has been adapted to dynamic scenes ( Pumarola et al., 2020 ; Liu et al., 2022 ) by conditioning NeRF on time ( Li et al., 2021 ; Du et al., 2021

Read the complete text