Transforming Single Photon Camera Images to Color High Dynamic Range Images
Sumit Sharma, Girish Rongali, Kaushik Mitra · 2024
Read Transforming Single Photon Camera Images to Color High Dynamic Range Images by Sumit Sharma, Girish Rongali, Kaushik Mitra (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12942.txt.
Introduction
Sumit Sharma, Girish Rongali, and Kaushik Mitra turn single-photon camera images into full color images with a wide range of light. Standard sensors fail at both light ends, with noise in the dark and white-out in glare. Fusion tricks fix the range yet leave ghosts in moving scenes. Their 2024 study takes a newer sensor kind, which counts single photons with fine time marks. Its bent response curve fits both light ends at once.
Raw images from it run noisy and plain. The team grades leading network designs on the path from gray single-photon images to color wide-range results, at a spread of sizes. Each stage of the path gets its own grade, by eye and by number. Where each design wins and where it wilts is the map they draw.
Catalog record
| CREATOR | Sumit Sharma, Girish Rongali, Kaushik Mitra |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12942
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transforming Single Photon Camera Images to Color High Dynamic Range Images Sumit Sharma, Girish Rongali, Kaushik Mitra Abstract Traditional CMOS sensors suffer from restricted dynamic range and sub-optimal performance under extreme lighting conditions. They are affected by electronic noise in low light conditions and pixel saturation while capturing high illumination. Recent High Dynamic Range (HDR) imaging methods, often designed for CMOS sensors, attempt to address these issues by fusing multiple exposures. However, they frequently introduce artifacts like ghosting and light flickering in dynamic scenarios and non-uniform signal-to-noise ratios in extreme dynamic range conditions. Recently, Single Photon Avalanche Diode (SPAD) sensors, also known as Single Photon Camera (SPC) sensors, have surpassed CMOS sensors due to their capability to capture individual photons with high timing precision. Unlike traditional digital cameras that first convert light energy into analog electrical currents and then digitize them, SPAD sensors perform direct photon detection, making them less susceptible to extreme illumination conditions. Their distinctive non-linear response curve aids in capturing photons across both low-light and high-illumination environments, making them particularly effective for high-dynamic range imaging. Despite their advantages, images from SPAD sensors are often noisy and visually unappealing. To address these challenges, our paper evaluates state-of-the-art architectures for converting monochromatic SPAD images into color HDR images at various resolutions. We present a two-stage approach: first, transforming monochromatic SPAD images into color images via Image2Image translation networks, and second, enhancing these color images to achieve HDR via Single Image HDR methods. Our evaluation involves both qualitative and quantitative assessments of these architectures, focusing on their effectiveness in each stage of the conversion process. 1 Introduction Traditional imaging systems often struggle with scenes featuring high levels of illumination and intricate details, such as bright light sources and highlights. Conventional image sensors have limited dynamic range because of limited full-well capacity. This leads to saturation at high brightness levels. To overcome the constraints of traditional sensors, several computational and hardware approaches have been developed [ 1 ] for capturing High Dynamic Range (HDR) images. One common method is exposure bracketing, which combines multiple images taken at different exposure levels into a single HDR image [ 2 ] , [ 3 ] . However, this method can introduce ghosting artifacts [ 4 ] and light flicker in dynamic scenes [ 7 ] , [ 6 ] , [ 5 ] . To mitigate these issues, commercial HDR sensors typically use a limited number of exposures—usually between two and four—either sequentially or through dual-pixel architectures [ 6 ] , [ 12 ] . Despite these improvements, HDR cameras using CMOS sensors still face challenges such as motion artifacts and a reduced signal-to-noise ratio (SNR) in scenes with extreme dynamic ranges. The process of recovering an HDR image from a few exposures can lead to uneven SNR artifacts, making it difficult to preserve fine details and effective denoising of the image [ 14 ] , [ 13 ] , [ 7 ] . In contrast, Single Photon avalanche Diode (SPAD) sensors offer a promising solution by measuring brightness through precise timing of individual photons based on Poisson statistics. This approach allows SPAD sensors to surpass the saturation limits of traditional pixels and capture timing information with high accuracy [ 8 ] . Recent advancements in Single-Photon Cameras (SPC) have demonstrated their superior dynamic range compared to conventional CMOS sensors [ 9 ] . SPAD sensors excel in both low and high light conditions within a single exposure, enabling them to produce detailed images across a wide range of brightness levels [ 10 ] , [ 8 ] . Despite these advantages, SPC images are not visually appealing, are noisy, and have limited resolutions. We propose two-stage as well as single stage approaches for converting monochromatic SPC images captured by single-photon cameras into high-resolution color HDR images. In the two-stage approach, we first utilize existing Image-to-Image translation networks to convert monochromatic SPC images to color LDR images and then convert them to color HDR images based on single image HDR reconstruction techniques. We evaluate various two-stage architectures through comprehensive quantitative and qualitative analyses. We also evaluate a direct image2image translation method for converting monochromatic SPC images to color HDR images. Our main contributions are: • Transforming SPC images to color HDR images. • Detailed analysis of various two-stage architectures for achieving the above goal. • Single-stage Image2Image translation from monochromatic SPC images to color HDR images at high resolution produces comparable results. 2 Related Work 2.1 Image-to-Image Translation Given that Single Photon Camera (SPC) images lack color information, employing image-to-image translation methods presents a viable avenue for colorizing SPC images. Among these methods, Generative Adversarial Networks (GANs) stand out as a prominent framework for estimating generative models through an adversarial process. In this process, two models are concurrently trained: a Generative model ’G’ that captures the data distribution, and a Discriminative model ’D’ that estimates the probability that a sample originates from the training data rather than ’G’ [ 15 ] . Building upon this concept [ 15 ] , researchers have delved into conditional adversarial networks as a versatile solution to various image-to-image translationI2I challenges. These networks not only learn the mapping from input images to output images but also acquire a loss function to facilitate this mapping training [ 16 ] . Defining and optimizing perceptual loss functions based on high-level features extracted from pre-trained networks makes it feasible to generate high-quality images [ 17 ] . Consequently, the fusion of training feed-forward convolutional neural networks with per-pixel loss and perceptual loss functions has emerged as a favored approach for numerous image-to-image translation tasks [ 18 ] , [ 19 ] , [ 20 ] . Furthermore, various unsupervised methods have been proposed for tackling the image-to-image translation task [ 21 ] , [ 22 ] , [ 23 , 24 ] .In contrast, one approach [ 25 ] focuses on producing images with a photographic appearance that aligns with the input layout. This approach operates as a rendering engine, translating a two-dimensional semantic specification of the scene into a photographic image without relying on adversarial training, which may suffer from training instability and optimization issues. To circumvent these challenges, this approach [ 25 ] adopts a direct regression objective based on computing distances between image features extracted by deep neural networks [ 26 ] , [ 27 ] , [ 17 ] . Consequently, although this approach [ 25 ] yields the first model capable of synthesizing 2048 x 1024 images, it may lack finer details and textures. Alternatively, another approach [ 28 ] achieves visually appealing results at a resolution of 2048x1024 by incorporating a novel adversarial loss function, along with multi-scale generator and discriminator architectures. Unpaired image-to-image translation often relies on maximizing mutual information between source and translated images across different domains to preserve source content and prevent unnecessary modifications. Self-supervised contrastive learning has proven effective in this context, where it constrains features to ensure fidelity to the source. However, prior methods often use features from random locations to impose constraints, potentially lacking sufficient information from the source domain. 2.2 Single Image HDR As a result of the non-linear response curve inherent in the SPAD pixel, characterized by an