Blind Time-of-Flight Imaging: Sparse Deconvolution on the Continuum with Unknown Kernels
Ruiming Guo, Ayush Bhandari · 2024
Read Blind Time-of-Flight Imaging: Sparse Deconvolution on the Continuum with Unknown Kernels by Ruiming Guo, Ayush Bhandari (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00893.txt.
Description
A 2024 arXiv research paper by Ruiming Guo, Ayush Bhandari. In recent years, computational Time-of-Flight (ToF) imaging has emerged as an exciting and a novel imaging modality that offers new and powerful interpretations of natural scenes, with applications extending to 3D, light-in-flight, and non-line-of-sight imaging. Mathematically, ToF imaging relies on algorithmic super-resolution, as the back-scattered sparse light echoes lie on a finer time resolution than what digital devices can capture. Traditional methods necessitate knowledge of the emitted light pulses or kernels and employ sparse deconvolution to recover scenes.
Catalog record
| CREATOR | Ruiming Guo, Ayush Bhandari |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00893
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Blind Time-of-Flight Imaging: Sparse Deconvolution on the Continuum with Unknown Kernels Thanks: Article in press. Ruiming Guo, Ayush Bhandari Abstract In recent years, computational Time-of-Flight (ToF) imaging has emerged as an exciting and a novel imaging modality that offers new and powerful interpretations of natural scenes, with applications extending to 3D, light-in-flight, and non-line-of-sight imaging. Mathematically, ToF imaging relies on algorithmic super-resolution, as the back-scattered sparse light echoes lie on a finer time resolution than what digital devices can capture. Traditional methods necessitate knowledge of the emitted light pulses or kernels and employ sparse deconvolution to recover scenes. Unlike previous approaches, this paper introduces a novel, blind ToF imaging technique that does not require kernel calibration and recovers sparse spikes on a continuum, rather than a discrete grid. By studying the shared characteristics of various ToF modalities, we capitalize on the fact that most physical pulses approximately satisfy the Strang-Fix conditions from approximation theory. This leads to a new mathematical formulation for sparse super-resolution. Our recovery approach uses an optimization method that is pivoted on an alternating minimization strategy. We benchmark our blind ToF method against traditional kernel calibration methods, which serve as the baseline. Extensive hardware experiments across different ToF modalities demonstrate the algorithmic advantages, flexibility and empirical robustness of our approach. We show that our work facilitates super-resolution in scenarios where distinguishing between closely spaced objects is challenging, while maintaining performance comparable to known kernel situations. Examples of light-in-flight imaging and light-sweep videos highlight the practical benefits of our blind super-resolution method in enhancing the understanding of natural scenes. keywords Computational imaging, inverse problems, blind deconvolution, super-resolution and time-of-flight imaging. Funding. The work of the authors is supported by the UK Research and Innovation council’s FLF Program “Sensing Beyond Barriers via Non-Linearities” (MRC Fellowship award no. MR/Y003926/1). † † runningheads: SIAM Journal on Imaging Sciences / Ruiming Guo and Ayush Bhandari MSC 62H35, 68U99, 78A46, 94A12. 1 Introduction to Time-of-Flight Imaging The emerging theme of Computational Sensing and Imaging or CoSI [ 10 , 16 ] has catalyzed never-seen-before capabilities in the context of imaging and vision. Some noteworthy examples single-pixel imaging [ 48 , 25 ] , non-line-of-sight imaging [ 55 , 60 ] , ultrafast imaging at a trillion [ 55 ] and a billion [ 28 ] frames per second, and imaging of black hole [ 51 ] . In all such examples and beyond, the fundamental difference from the traditional viewpoint is the new mindset that altering the forward model of an imaging system enables new capabilities—one can co-design hardware and algorithms in pursuit of new advantages that can not be harnessed by optimizing hardware or algorithms alone. Figure 1: Diffusive Imaging. (a) Experimental setup [ 35 , 13 ] . A “TIME OF FLIGHT” placard is hidden by a diffusive semi-translucent sheet between the placard and the ToF sensor. (b) Depth estimation from pixel measurements via sparse deconvolution techniques [ 17 , 13 , 14 ] . The ToF kernel ( i.e. \varphi_{\mathbf{r}} in ( 2 )) is calibrated and known prior to experiments. (c) 3 D scene reconstruction from estimated depth parameters in (b). The main goal of this paper is to recover 3 D scene without pre-known kernel information (refer to Fig. 8 and Fig. 10 ), aiming to eliminate the necessity for experimental calibration. In the context of CoSI , the hardware-software co-design approach has been particularly fruitful for time-of-flight (ToF) imaging. A notable characteristic of ToF imaging is that each pixel of the imaging sensor captures a scene-dependent time profile at a specific time resolution, usually on a scale ranging from nanoseconds (ns) to picoseconds (ps). This technique, therefore, is also referred to as Time-Resolved Imaging, highlighting its capability to capture detailed temporal information within a scene (see Chapter 5, [ 10 ] ), as illustrated in Fig. 1 . In contrast to conventional imaging paradigms, ToF imaging offers numerous advantages that were previously unimaginable. This is due to the fact that each ToF measurement comprises two types of images: the conventional 2 D photograph or amplitude image, and the unconventional time-resolved, depth image. The advent of ToF imaging technology paves the way for novel methods and applications in various domains, including computer vision [ 34 , 52 , 21 ] , graphics [ 61 , 35 , 32 ] , autonomous vehicles, and bio-medical imaging [ 5 , 46 ] . Two concrete examples that also serve as an experimental validation of our work (in Section 5 ) are as follows. 1) Diffuse Imaging [ 35 , 32 , 13 , 8 ] . As illustrated in Fig. 1 , here ToF imaging enables recovery of objects hidden or obfuscated via diffusive materials, also see Section 5.1.1 . 2) Light-in-Flight (LIF) Imaging [ 1 , 32 , 56 , 35 , 29 ] . In 1978 [ 1 ] , N. Abramson pioneered LIF using a holographic method to record the wavefront of a light pulse as it traveled and scattered off a white screen. In a similar spirit, ToF measurements also enable LIF imaging that involves capturing and reconstructing the trajectory of light as it traverses and interacts with various objects in the scene, leading to completely new pathways for scene understanding. This is another ToF imaging capability, covered in this paper in Section 5.2 ( https://youtu.be/ffkc_z8ogE8 ). Depending on the technology and temporal resolution, ToF imaging setups can be broadly classified into three categories: i) Lock-in sensors [ 27 ] operating at nanosecond resolution ii) Single Photon Avalanche Diode (SPAD) detectors [ 37 , 29 ] operating at picosecond resolution iii) Streak tubes [ 55 ] operating at femtosecond resolution. Among them, lock-in sensors are notably popular for ToF imaging, largely because they are available as consumer-grade technology, exemplified by photonic mixer devices (PMD) and the Microsoft Xbox One’s Kinect, making them widely used and accessible. Figure 2: Pipeline for LIF Imaging (see https://youtu.be/ffkc_z8ogE8 ). Top Row: experimental setup capturing light’s interaction with a scene, consisting of a mannequin head positioned between a diffusive surface and a backdrop wall. Bottom Row: Using a ToF camera, we visualize the light’s progression across different time instances marked by “ \bullet ”, highlighted by visible light’s spectrum colors coding the ascending time sequence. Initially, light encounters the diffusive sheet in the early time-slots, i.e. (i) (blue) and (j) (green). Light then moves over the mannequin, and ultimately reaches the back wall, as shown in frames (q) (yellow) to (t) (red). The LIF accurately maps the 3 D geometry of the scene. The full LIF time slices are shown in Fig. 14 . 1.1 Fundamental Role of Sparsity and Super-Resolution in ToF Imaging In conventional 2D photography the scene is rich in its natural representation and described by low-level image features, such as edges, corners and ridges [ 26 ] . The resulting images are sparse in transform domain e.g. DCT or wavelet basis. However, when the same scene is interpreted from the perspective time-scales [ 61 , 35 , 32 ] , the features are fundamentally different and take the form of global and direct delays, inter-reflections and sub-surface scattering. Mathematically, this translates to the fact that scenes are