Geometric-Photometric Event-based 3D Gaussian Ray Tracing
Kai Kohyama, Yoshimitsu Aoki, Guillermo Gallego, Shintaro Shiba · 2025
Read Geometric-Photometric Event-based 3D Gaussian Ray Tracing by Kai Kohyama, Yoshimitsu Aoki, Guillermo Gallego, Shintaro Shiba (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18640.txt.
Introduction
Kai Kohyama and three coauthors present a 2025 computer vision method for rebuilding 3D scenes from event cameras. Event cameras record brightness changes with very fine timing, but earlier 3D Gaussian Splatting methods could not use that timing well. Their framework splits rendering into two branches: event-by-event depth rendering by ray tracing, and snapshot-based brightness rendering, linked through the image of warped events.
Tests show state-of-the-art results on real-world datasets and competitive results on synthetic data. The method needs no pretrained reconstruction models and no COLMAP initialization. It stays flexible in how many events it processes per sample, trains fast, and keeps scene edges sharp. The work clarifies how sparse event streams can drive full 3D reconstruction.
Catalog record
| CREATOR | Kai Kohyama, Yoshimitsu Aoki, Guillermo Gallego, Shintaro Shiba |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-18640
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Geometric-Photometric Event-based 3D Gaussian Ray Tracing Kai Kohyama, Yoshimitsu Aoki, Guillermo Gallego, Shintaro Shiba Abstract Event cameras offer a high temporal resolution over traditional frame-based cameras, which makes them suitable for motion and structure estimation. However, it has been unclear how event-based 3D Gaussian Splatting (3DGS) approaches could leverage fine-grained temporal information of sparse events. This work proposes a framework to address the trade-off between accuracy and temporal resolution in event-based 3DGS. Our key idea is to decouple the rendering into two branches: event-by-event geometry (depth) rendering and snapshot-based radiance (intensity) rendering, by using ray-tracing and the image of warped events. The extensive evaluation shows that our method achieves state-of-the-art performance on the real-world datasets and competitive performance on the synthetic dataset. Also, the proposed method works without prior information (e.g., pretrained image reconstruction models) or COLMAP-based initialization, is more flexible in the event selection number, and achieves sharp reconstruction on scene edges with fast training time. We hope that this work deepens our understanding of the sparse nature of events for 3D reconstruction. The code will be released. 1 Introduction Figure 1 : Overview of the proposed method , which takes raw events and poses as input. During the optimization of the 3D Gaussians, rendering is decoupled into two pathways: event-by-event (temporally dense) depth rendering and spatially dense intensity rendering. We use the image of warped events to connect these two pathways to compute both geometric and photometric losses. The color results are from a synthetic dataset, and the monochrome results are from two standard, real-world datasets. Event cameras have attracted increasing attention in computer vision and robotics due to their advantages and capabilities for various tasks Gallego20pami ; AliAkbarpour2024survey . Unlike conventional cameras that record synchronous frames at fixed time intervals, event cameras respond asynchronously to per-pixel brightness changes with \mu s resolution Lichtsteiner08ssc ; Posch14ieee ; Finateu20isscc . This working principle makes them highly sensitive to motion, since changes are due to scene contrast and relative motion. In parallel, Gaussian Splatting (GS) Kerbl23tog has emerged as a state-of-the-art representation for photometric 3D reconstruction and novel view synthesis (NVS). Since 3D structure and motion are tightly connected in the generation of event data, it is paramount to develop 3DGS algorithms that leverage the event camera’s fine-grained temporal information. Also, thanks to the event camera’s minimal motion blur and high dynamic range (DR), event-based GS methods have the potential to overcome some of the limitations of frame-based GS, such as motion blur and low DR. Previous approaches in event-based photometric 3D reconstruction (e.g., NeRF Mildenhall21nerf and GS) typically perform two dense renderings per sample (e.g., Yura25cvpr ). The difference between these two rendered images is compared with the edge-like image obtained by pixel-wise aggregation of the event data, resulting in a photometric loss that drives the 3D-Gaussian optimization Huang25cvpr ; Yura25cvpr . However, as this approach requires dense (all-pixel) scene rendering twice, it not only slows down the training, but also introduces a fundamental limitation: the trade-off between accuracy and temporal window selection. A short time interval between the two renderings fails to capture subtle intensity variations that generate only a few events. Contrarily, a large interval makes the predicted edge image blurry and discards fine-grained temporal information in the observed edge. Similar observations have been made in Rudnev23cvpr ; Xiong24corl : a trade-off between capturing global lighting and local details. In this work, we fundamentally address these limitations and introduce the first framework for event-based GS that renders the dense intensity (radiance) once per sample (i.e., a batch of events), while keeping the rendering efficient and leveraging the high temporal resolution. Our key idea is to consider dedicated structure and appearance updates in the GS framework ( Fig. 1 ). A “structure” pathway defines a geometric loss built on top of Contrast Maximization Gallego18cvpr by leveraging the known camera motion and the event-by-event (i.e., sparse) rendered depth. An “appearance” pathway defines a photometric loss between the instantaneous brightness change modeled by the rendered dense intensity and the change measured by the event data. Our method showcases several advantages through extensive evaluations. First, it does not rely on any prior knowledge (e.g., frames or pretrained models for depth and intensity) or COLMAP Schoenberger16cvpr for initialization. Second, it achieves state-of-the-art rendering quality performance on real-world datasets, where poses and raw events are noisy, with fast training time (it is faster than the latest methods that we benchmark Huang25cvpr ; Low23iccv ; Yura25cvpr ). Third, it shows robustness with respect to the number of events processed per sample, without compromising accuracy. In summary, this work presents several distinctive contributions in event-only Gaussian Splatting: 1. The proposed method decouples two different quantities in 3DGS rendering: the continuous-time spatially sparse depth, and the instantaneous dense intensity. It addresses the trade-off between accuracy and temporal resolution in existing event-based 3D reconstruction methods. 2. The comprehensive evaluation shows state-of-the-art results on real-world datasets and competitive results in simulation without relying on any prior knowledge, as opposed to existing event-based 3DGS methods. 3. The proposed framework connects event-by-event depth estimation and 3DGS, which is enabled by an efficient event-by-event ray-tracing implementation. 4. The method achieves the fastest training time among other tested state-of-the-art methods (e.g., Huang25cvpr ; Yura25cvpr ; Low23iccv ). We hope that this work unblocks the potential of high temporal resolution event data in 3D reconstruction. 2 Related Work Figure 2 : Method overview . Using ray-tracing renderer, we estimate depth for each event and compute the flow with the interpolated poses (i.e., motion field). Performing event warping produces the image of warped events at t_{\text{mid}} and computes the contrast loss. We render the dense intensity (radiance) at t_{\text{mid}} and compute the instantaneous brightness increment image, which we use for the photometric loss. 3D Gaussian Splatting (3DGS) Kerbl23tog represents the scene as a collection of anisotropic Gaussian ellipsoids and renders via differentiable splatting Kerbl23tog . These methods achieve high‐fidelity NVS with faster rendering and more scalable training in static and dynamic scenes Huang24siggraph ; Wu24cvpr ; Kulhanek24wildgaussians than Neural Radiance Fields (NeRFs) Mildenhall21nerf . While most works focus on rasterization-based splatting, recently, ray-tracing approaches have emerged, which inspire our work. Notably, 3D Gaussian Ray Tracing (3DGRT) Moenne24tog casts rays against volumetric Gaussian particles, and 3DGUT Wu25cvpr replaces Elliptical weighted-average raster splatting with an Unscented Transform projection of Gaussian particles. Event cameras have inspired substantial work in reconstructing 3D geometry and scene appearance, leveraging motion information from their data modality of high temporal resolution. Several approaches propose enhanced frame-based NVS aided by the complementary information in the event data, e.g., in the presence of fast motion (images with motion blur) or deformations Deguchi24icip ; Xiong24corl ; Cannici24cvpr ; He25tvcg ; Deng25arxiv . Such approaches may still suffer from bottlenecks of the frame-based cameras in the system, such as low DR, and inaccuracies in sensor fusion calibration. Instead, event-only methods like our work focus on unlocking the potential of event cameras. The key question of event-based NVS methods (NeRF and GS) lies in the measurement model: how to compare the modeled scenes, which typically display absolute intensity rendered at concrete viewpoints and times, to the acquired event data stream, which measures sparse intensity differences asynchronously at