Zero-Shot Low Light Image Enhancement with Diffusion Prior

Joshua Cho, Sara Aghajanzadeh, Zhen Zhu, D. A. Forsyth · 2024

Read Zero-Shot Low Light Image Enhancement with Diffusion Prior by Joshua Cho, Sara Aghajanzadeh, Zhen Zhu, D. A. Forsyth (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13401.txt.

Introduction

Dark photos lose detail, yet fixing them often means long training runs and careful tuning. Joshua Cho and three co-authors offer a shortcut in 2024 that needs no training at all. Their method borrows a large image model that was trained to make pictures from text, and uses traits inside that model to guide the repair.

Rather than add custom rules, the team lets the borrowed knowledge steer each step of the fix. Tests on standard data sets show results that beat leading methods on score, with truer colours and small colour shifts set right. A side test shows the same method, unchanged, can also handle white balance at a level close to the best tools built for that job alone.

Catalog record

CREATORJoshua Cho, Sara Aghajanzadeh, Zhen Zhu, D. A. Forsyth
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13401

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zero-Shot Low Light Image Enhancement with Diffusion Prior Joshua Cho, Sara Aghajanzadeh, Zhen Zhu, D. A. Forsyth Abstract In this paper, we present a simple yet highly effective “free lunch” solution for low-light image enhancement (LLIE), which aims to restore low-light images as if acquired in well-illuminated environments. Our method necessitates no optimization, training, fine-tuning, text conditioning, or hyperparameter adjustments, yet it consistently reconstructs low-light images with superior fidelity. Specifically, we leverage a pre-trained text-to-image diffusion prior, learned from training on a large collection of natural images, and the features present in the model itself to guide the inference, in contrast to existing methods that depend on customized constraints. Comprehensive quantitative evaluations demonstrate that our approach outperforms SOTA methods on established datasets, while qualitative analyses indicate enhanced color accuracy and the rectification of subtle chromatic deviations. Furthermore, additional experiments reveal that our method, without any modifications, achieves SOTA-comparable performance in the auto white balance (AWB) task. Figure 1 : In low-light image enhancement, an ideal method should achieve color constancy [ 21 , 38 ] by accurately recovering the intrinsic color (reflectance) of a scene, ensuring consistency across images taken under varying illumination conditions. However, suboptimal illumination introduces noise and hampers the accurate capture of all wavelengths, leading to color distortions. For instance, recent zero-shot diffusion-based methods such as GDP [ 20 ] suffer from hallucinations , introducing non-existent elements (row 1), while FourierDiff [ 49 ] is compromised by the inherent noise sensitivity of frequency-domain representations (rows 1 and 2). In contrast, our approach demonstrates superior color constancy and fidelity across images of the same scene, effectively mitigating the challenges posed by lighting variations. 1 Introduction Figure 2 : Hallucination. While diffusion prior is effective for image restoration, improper application can lead to unintended hallucinations, where the model generates nonexistent structures or alters scene semantics. For example, GDP [ 20 ] , a robust and versatile image restoration method, often hallucinates in the presence of substantial noise and darkness in input images. As shown in row 1, a blue-colored cabinet is inaccurately reconstructed as a sky, a pink cabinet as a building, and the entire scene resembles a battle. For less noisy inputs, our method produces clean and sharp outputs and effectively attenuates noise even in challenging cases involving severe darkness and pronounced noise levels. Low-light image enhancement aims to enhance images captured in suboptimal lighting conditions into their natural, well-lit counterpart. Its relevance spans from photography [ 44 ] to various downstream tasks such as autonomous driving [ 42 ] , underwater image enhancement [ 58 , 40 ] , and video surveillance [ 81 ] . Yet, it is a challenging task because of the presence of shot noise and color quantization effects, which undermine the applicability of elementary solutions such as uniform intensity scaling as evidenced in Figure 1 (first row, Scaled column). Despite advancements in prior studies (discussed in Section 2 ), the inherent dataset dependencies in both supervised and unsupervised learning paradigms continue to introduce unintended artifacts, as evidenced in Figures 5 and 6 . To address this limitation, prior zero-shot LLIE methods [ 25 , 20 , 49 ] optimize the auxiliary network or parameters alongside a frozen pre-trained diffusion model at test time by relying on custom loss formulation or degradation assumption (Figure 3 ). Our method, however, leverages internal signals within the model, specifically self-attention features, extending their applicability beyond previous uses in image editing tasks [ 72 , 9 , 67 , 54 , 30 , 16 , 19 ] by following four simple steps: (1) preprocessing; (2) inverting the input image; (3) adjusting the resulting noised latent with Adaptive Instance Normalization (AdaIN) to match standard normal distributions \mathcal{N}(0,I) ; and (4) denoising the inverted representation with self-attention features extracted during the inversion process. Figure 3 : LLIE Method Taxonomy. (category a) Image regression methods often produce results that are heavily dependent on the dataset, because real-world datasets are small in size. (category b) Zero-shot methods w/o a pre-trained model dynamically adjust model weights per image based on a predefined loss function. However, they require per-image tuning and may suffer from convergence instability. (category c) Zero-shot method w/ a pre-trained model and an auxiliary trainable network or parameters that learn on a per-image basis. While this enhances adaptability, it still requires per-image tuning and remains susceptible to convergence instability. (category d) In contrast, our method leverages the self-attention features of pre-trained diffusion models from the input to guide inference from any data source without any assumption about degradation and without test time tuning. Qualitative analysis suggests that the robustness of our method arises from the method’s ability to precisely correct subtle color shifts, a significant benefit in low-light images where color degradation is prevalent as shown in Figure 5 . Notably, as a secondary outcome, our method also proves effective for white balancing, achieving performance on par with the SOTA approaches. Our key contributions are as follows : 1. We propose a simple yet highly effective zero-shot method for low-light image enhancement (LLIE) that requires no training, fine-tuning, or optimization, yet surpasses state-of-the-art (SOTA) performance on established benchmarks using standard evaluation metrics. 2. To the best of our knowledge, without any modification, our method is also the first zero-shot auto white balance method and achieves results comparable to SOTA in this task. 2 Related Work Traditional Methods. Conventional image enhancement methods, including histogram equalization [ 29 , 56 ] and gamma correction [ 60 ] , rely on global adjustments to enhance image contrast. Despite their computational efficiency, these methods are inherently limited by their inability to account for varying scene-specific lighting conditions. Moreover, their global adjustment inadvertently amplifies dark noise in low-light areas, diminishing fine details and introducing artifacts. Supervised Methods. Convolutional Neural Networks (CNNs) are adept at learning transformations from under-exposed to well-lit images, effectively capturing local textures and patterns [ 48 , 79 , 83 , 86 ] . However, their limitation in capturing long-range dependencies has led to alternative approaches, such as ensemble methods [ 4 ] , transformer-based method [ 8 ] , and synthetic data augmentation [ 3 , 52 ] . Recently, generative methods have exhibited promising results in low-light image enhancement, with diffusion models [ 17 , 70 , 62 , 68 , 31 , 71 ] demonstrating particular efficacy because of their strong generative ability, being free from the instability and mode-collapse that are prevalent in previous generative models. However, standard Gaussian noise assumptions of diffusion models do not model the complex noise of low-light images. In response, new training strategies have been proposed for raw and RGB low-light image enhancement [ 53 , 78 , 35 , 33 , 84 , 36 , 46 ] . However, these methods remain dependent on supervised learning, without leveraging the generative priors of the pre-trained diffusion models. Unsupervised Methods. Unsupervised learning methods [ 61 , 43 , 26 , 41 , 37 , 51 , 43 , 82 , 22 , 66 , 76 , 90 , 65 ] have emerged as a promising direction for low-light image enhancement, because they do not rely on paired datasets. EnlightenGAN [ 37 ] and NeRCo

Read the complete text