FoveaSPAD: Exploiting Depth Priors for Adaptive and Efficient Single-Photon 3D Imaging
Justin Folden, Atul Ingle *, Sanjeev J. Koppal * · 2024
Read FoveaSPAD: Exploiting Depth Priors for Adaptive and Efficient Single-Photon 3D Imaging by Justin Folden, Atul Ingle *, Sanjeev J. Koppal * (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02052.txt.
Introduction
Justin Folden, Atul Ingle, and Sanjeev Koppal improve single-photon depth sensing in 2024. Such sensors are very sensitive and precise, which suits self-driving cars. Two flaws block wide use: stray light spoils readings, and the raw photon data are huge.
Their method guides the sensor with outside signals, focusing effort where the signal of interest lies. This focus cuts the data that must be stored and moved, and it hardens readings against stray light. Tests in simulation and with real hardware showed a vast cut in memory use, over a thousand-fold in one setup. The methods fit current and future sensor arrays.
Catalog record
| CREATOR | Justin Folden, Atul Ingle *, Sanjeev J. Koppal * |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-02052
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FoveaSPAD: Exploiting Depth Priors for Adaptive and Efficient Single-Photon 3D Imaging Justin Folden, Atul Ingle *, Sanjeev J. Koppal * Abstract Fast, efficient, and accurate depth-sensing is important for safety-critical applications such as autonomous vehicles. Direct time-of-flight LiDAR has the potential to fulfill these demands, thanks to its ability to provide high-precision depth measurements at long standoff distances. While conventional LiDAR relies on avalanche photodiodes (APDs), single-photon avalanche diodes (SPADs) are an emerging image-sensing technology that offer many advantages such as extreme sensitivity and time resolution. In this paper, we remove the key challenges to widespread adoption of SPAD-based LiDARs: their susceptibility to ambient light and the large amount of raw photon data that must be processed to obtain in-pixel depth estimates. We propose new algorithms and sensing policies that improve signal-to-noise ratio (SNR) and increase computing and memory efficiency for SPAD-based LiDARs. During capture, we use external signals to foveate , i.e., guide how the SPAD system estimates scene depths. This foveated approach allows our method to “zoom into” the signal of interest, reducing the amount of raw photon data that needs to be stored and transferred from the SPAD sensor, while also improving resilience to ambient light. We show results both in simulation and also with real hardware emulation, with specific implementations achieving a 1548-fold reduction in memory usage, and our algorithms can be applied to newly available and future SPAD arrays. I Introduction Biological vision systems have the remarkable ability to foveate — i.e. redistribute cognitive resources towards “salient” features or objects in a scene, depending on context. Unfortunately, most conventional cameras and computer vision systems today capture scene information in a non-adaptive fashion, spending power and bandwidth on sensing scene components that may not help the overall imaging task. In fact, the current framework for deep learning-based systems assumes uniform sampling of the scene and overcomes these limitations through data-driven pipelines that focus on interesting regions of the scene [ 1 ] [ 2 ] in the input RGB images. Fig. 1: Depth Prior Driven SPAD Depth Foveation: SPAD sensors suffer from a data bottleneck, since thousands of histogram bins are used to generate depth as shown in the top left. If fewer bins are used, this reduces depth resolution, as shown in the limited bins depth result. Our idea is to use additional information, such as a color image (Sec. IV , VII ) or optical flow (Sec. VI ), to foveate the SPAD bins. Therefore, for the same memory cost we can place the bins near where the histogram peak should be, results in accurate depth, as shown in the depth foveation result. The insets show that our method achieves the accuracy and resolution of ground truth, with fewer bins. They also show that the depth prior, in this case monocular estimation, by itself cannot provide the correct depth, and foveation is required. While this inefficient but popular framework for conventional RGB sensors may be difficult to change, our proposed method, called FoveaSPAD, can impact the next wave of single-photon avalanche diode (SPAD) sensor technology. SPADs can capture scene information at the granularity of individual photons, at timescales as small as 10’s of picoseconds. Recent advances in CMOS-compatible SPAD pixel designs has enabled real-time in-pixel processing of these photon timestamp streams. Thus, SPADs are a natural candidate for designing efficient depth cameras — individual pixels can be reprogrammed on-the-fly to adaptively accept or reject a spatio-temporal subset of the photon stream. Our FoveaSPAD algorithms enable capturing scene information at higher granularity in regions that are most relevant to a downstream vision task. In this sense, we generalize the term “foveation” in the context of adaptive SPAD spatio-temporal sampling to allow both depth and memory efficiencies. For robots, remote sensor nodes, and other resource-constrained systems, foveation for SPAD sensors can allow accurate depth sensing under constraints on power and bandwidth (see Fig. 1 ). The raw data captured by an array of SPAD pixels can be thought of as a spatio-temporal photon stream. Each photon detection is represented as an (x,y,t) coordinate, where the x-y coordinates denote the pixel location and the t coordinate denotes the photon detection timestamp. Each SPAD pixel captures the round trip time of a laser pulse to and from a given scene point, constructing a photon timing histogram which records the number of photons captured at various time delays with respect to the time the laser pulse was transmitted. Each pixel must construct one such histogram, typically with 1000’s of bins, which causes a severe data bottleneck for today’s SPAD cameras. To illustrate the severity of the bottleneck, consider a 1-megapixel SPAD array with a 1000-bin histogram per pixel, storing 1 byte per bin. At 30 frames per second, this setup generates a staggering 30 GB of data every second Our algorithms foveate across the spatio-temporal histogram space to efficiently recover the peak, providing the time-delay t for depth computation. We adaptively capture subranges to locate laser photons, rejecting ambient photons. Note that our proposed algorithms are not exhaustive; rather, we aim to define a class of algorithms that rely on a depth prior. In this work, we propose three methods for acquiring priors, though many other methods exist, such as depth from stereo, depth from defocus [ 3 ] , or non-vision-based methods such as sonar. Each method has trade-offs, and it is up to the user to determine which method best suits their use case. Our contributions in this work are as follows: • We present a theoretical model for expected gains (in terms of increased signal-to-noise-ratio and depth resolution) from foveation with SPADs. • We explore the question of how to foveate in space and time at a single time instant by leveraging monocular depth estimates, which can come either from the SPAD-generated image or a cheap, external color camera. We propose different flavors of practical FoveaSPAD designs that optimize for memory/bandwidth and depth resolution. • For images of moving scenes, we demonstrate how to use optical flow cues to direct SPAD foveation. • We show results both in simulation and using recently available real SPAD datasets. I-A Hardware Emulation Time-correlated single photon counting is the technique that enables SPAD cameras to build histograms and control binning on-sensor. Our work is limited to simulation experiments and hardware emulation of existing SPAD LiDAR data. Hardware emulation refers to leveraging real-world data captured using a single SPAD or line arrays [ 4 , 5 ] , which we then use to emulate the performance of larger SPAD arrays. While SPAD sensor arrays with native support for foveation are not yet available commercially, we believe our proposed techniques could be implemented at the pixel or camera level. This is supported by recent proofs-of-concept in kilopixel-resolution reconfigurable SPAD arrays with in-pixel timestamping, gating, and histogramming capabilities [ 6 , 7 ] . We anticipate that this work will inspire future hardware designs, leading to more efficient and versatile SPAD sensor arrays. I-B Scope: Simulation and Emulations In this work, we anticipate future hardware advancements that will enhance SPAD-based depth sensing. Our simulations and emulations are intended to project the performance of emerging SPAD sensor technologies, focusing on adaptive and efficient bin