Neural Inverse Rendering from Propagating Light
Anagh Malik Benjamin Attal Andrew Xie Matthew O’Toole David B. Lindell, University of Toronto Vector Institute Carnegie Mellon University · 2025
Read Neural Inverse Rendering from Propagating Light by Anagh Malik Benjamin Attal Andrew Xie Matthew O’Toole David B. Lindell, University of Toronto Vector Institute Carnegie Mellon University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05347.txt.
Introduction
Anagh Malik and co-authors presented a system in 2025 that rebuilds scenes from videos of light itself in flight. Their tool works from many viewpoints of a light pulse spreading through a room, caught by a flash lidar device. Normal rendering tools cannot easily run that movie backward to recover the scene.
The system speeds the math with a neural store of light that has bounced any number of times, stretched to track time. It separates direct light from bounced light on its own. In tests, it rebuilt three-dimensional scenes well even where bounced light was strong. It can also replay the pulse from new angles and relight the scene as if the flash had moved. The authors call it a first for this kind of capture.
Catalog record
| CREATOR | Anagh Malik Benjamin Attal Andrew Xie Matthew O’Toole David B. Lindell, University of Toronto Vector Institute Carnegie Mellon University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05347
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural Inverse Rendering from Propagating Light Anagh Malik Benjamin Attal Andrew Xie Matthew O’Toole David B. Lindell, University of Toronto Vector Institute Carnegie Mellon University Abstract We present the first system for physically based, neural inverse rendering from multi-viewpoint videos of propagating light. Our approach relies on a time-resolved extension of neural radiance caching — a technique that accelerates inverse rendering by storing infinite-bounce radiance arriving at any point from any direction. The resulting model accurately accounts for direct and indirect light transport effects and, when applied to captured measurements from a flash lidar system, enables state-of-the-art 3D reconstruction in the presence of strong indirect light. Further, we demonstrate view synthesis of propagating light, automatic decomposition of captured measurements into direct and indirect components, as well as novel capabilities such as multi-view time-resolved relighting of captured scenes. Figure 1 : We introduce a method to model and invert multi-view, time-resolved measurements of propagating light from a flash lidar system. (row 1) Our method accurately recovers the geometry of this scene and enables rendering of time-resolved lidar measurements that reveal light propagation from novel views. ( row 2 ) Physically-based modeling enables novel applications, such as time-resolved relighting and automatic decomposition of light transport into direct and indirect components. 1 Introduction Ultrafast imaging systems such as lidar illuminate a scene with a pulse of light and capture the backscattered “echoes” from the propagating wavefront Schwarz 2010 . Precisely measuring the speed-of-light time delay of backscattered light enables 3D reconstruction, and so lidar systems based on this principle are popular in applications from autonomous driving Li and Ibanez-Guzman 2020 to augmented reality Apple Inc. 2024 and remote sensing Dong and Chen 2017 . Lidar relies on time-resolved measurements of direct light transport, or light that reflects directly from a surface back to the sensor. Measurements of indirect light transport (i.e., light that scatters multiple times before reaching the sensor) are typically ignored or discarded because modeling indirect light requires computationally expensive (and often intractable) inverse rendering using procedures such as recursive path tracing Pharr et al. 2023 ; Jakob et al. 2022 . Still, measurements of indirect light are a rich source of information about material properties, appearance, and geometry Naik et al. 2011 ; Wu et al. 2014 ; Zhang et al. 2022 ; O’Toole et al. 2018 ; Klinghoffer et al. 2024 . Our work seeks to model and invert multi-viewpoint, time-resolved measurements of propagating light from a lidar system to recover scene geometry and to render videos of propagating light from novel viewpoints and under novel illumination conditions (see Figure 1 ). Conventional lidar systems pre-process captured time-resolved measurements into a 3D point cloud, which represents an estimate of scene geometry based on direct light transport. While recent work leverages lidar measurements captured from multiple viewpoints to perform 3D reconstruction and novel view synthesis, existing methods use a point cloud representation Rematas et al. 2022 ; Deng et al. 2022 ; Zhang et al. 2024 ; Huang et al. 2023 or direct-only time-resolved measurements Malik et al. 2023 ; Luo et al. 2025 ; Ramazzina et al. 2024 ; Mu et al. 2024 , and thus ignore indirect light. Other methods for multi-viewpoint reconstruction with active imaging use time-of-flight sensors Attal et al. 2021 ; Okunev et al. 2024 , or structured light Shandilya et al. 2023 ; Ichimaru et al. 2024 , but similarly fail to explicitly model indirect light. Our work is close to that of Malik et al. Malik et al. 2024 , which uses lidar measurements and a representation based on neural radiance fields (NeRFs) Mildenhall et al. 2021 to render videos of propagating light from novel viewpoints. However, while their representation is effective for view synthesis, it does not use a physically-based model, and so cannot reconstruct accurate geometry or render the scene under novel illumination conditions. Accurately modeling indirect light transport requires solving the rendering equation Kajiya 1986 . Many renderers use path tracing algorithms based on Monte Carlo sampling, and differentiable versions of these methods have shown promise for inverse rendering of conventional (i.e., steady state) images Li et al. 2018 ; Loubet et al. 2019 ; Nimier-David et al. 2019 and time-resolved measurements Wu et al. 2021 ; Yi et al. 2021 , such as those from a lidar system. However, while these methods work in constrained problem settings, such as non-line-of-sight imaging Tsai et al. 2019 ; Iseringhausen and Hullin 2020 , they are difficult to deploy on captured multi-viewpoint experiments due to their computational expense and sensitivity to noise and local minima. To address this issue, recent methods approximate the rendering equation in a hybrid fashion using a volumetric representation of geometry and physics-based appearance models Zhang et al. 2021 ; Srinivasan et al. 2021 . This type of approach has proven especially effective for inverse rendering of appearance, geometry, and material parameters from conventional intensity images Zhang et al. 2022 ; Jin et al. 2023 ; Liu et al. 2023 ; Attal et al. 2024 . Still, no previous technique performs inverse rendering of time-resolved direct and indirect light transport in the same fashion. Here, we propose a method for inverse rendering from multi-viewpoint, time-resolved measurements of propagating light from a flash lidar system — a type of lidar that flood-illuminates the entire scene with a pulse of light. Our approach is based on a hybrid neural representation, where we model geometry using volume rendering, and appearance using a physically-based model that simulates global illumination effects using a radiance cache Ward et al. 1988 . Specifically, instead of integrating light paths using path tracing, our radiance cache stores a representation of time-resolved radiance arriving at any point in a volume from any direction. The representation is optimized in an amortized fashion, thereby removing the need to evaluate the rendering integral recursively; instead, we need only render direct lighting from the lidar and query the radiance cache to evaluate indirect light at any surface point. Our model builds on previous neural rendering frameworks for multi-viewpoint, time-resolved rendering of the direct Malik et al. 2023 and indirect components Malik et al. 2024 of propagating light, and grounds them in a framework for physically-based rendering. Overall, we make the following contributions. • We propose a method for neural inverse rendering of propagating light using a physically-based model with a time-resolved radiance cache. • We capture a new dataset of multi-viewpoint, time-resolved flash lidar measurements with calibrated light source and camera positions. • We demonstrate our method in simulation and on captured data; we show state-of-the-art results in geometry reconstruction under strong indirect light transport, and we render videos of propagating light from novel viewpoints and under novel illumination conditions in scenes with varying reflectance properties and significant indirect light transport effects. 2 Related Work Our approach brings together the areas of time-resolved imaging and rendering, as well as inverse rendering. Time-resolved imaging. Time-of-flight systems such as lidar measure the time of flight by marking the arrival time of a backscattered pulse of light Koechner 1968 . These systems usually combine nanosecond or picosecond pulsed lasers with fast photodiodes Huntington 2020 or single-photon avalanche diodes (SPADs) Rapp et al. 2020b ; Kirmani