Event-assisted 12-stop HDR Imaging of Dynamic Scene
Shi Guo, Zixuan Chen, Ziran Zhang, Yutian Chen, Gangwei Xu, Tianfan Xue · 2024
Read Event-assisted 12-stop HDR Imaging of Dynamic Scene by Shi Guo, Zixuan Chen, Ziran Zhang, Yutian Chen, Gangwei Xu, Tianfan Xue (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14705.txt.
Introduction
Shi Guo and co-authors widen camera range for moving scenes in a 2024 paper. High dynamic range imaging keeps detail in both bright and dark zones of a shot. Classic methods merge several frames at unlike exposures. In moving scenes with huge gaps in brightness, lining those frames up fails and ghosts appear. The team pairs a normal camera with an event camera, which reports pixel changes with fine timing.
The event stream helps line up frames across a 12-stop range, a span far past the usual. A diffusion-based fusion step then cleans high-contrast zones using learned image habits. The team built a new data set, ESHDR, with synced event data for such scenes. Tests on fake and real data put their method at the top.
Catalog record
| CREATOR | Shi Guo, Zixuan Chen, Ziran Zhang, Yutian Chen, Gangwei Xu, Tianfan Xue |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14705
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Event-assisted 12-stop HDR Imaging of Dynamic Scene Shi Guo, Zixuan Chen, Ziran Zhang, Yutian Chen, Gangwei Xu, Tianfan Xue Abstract High dynamic range (HDR) imaging is a crucial task in computational photography, which captures details across diverse lighting conditions. Traditional HDR fusion methods face limitations in dynamic scenes with extreme exposure differences, as aligning low dynamic range (LDR) frames becomes challenging due to motion and brightness variation. In this work, we propose a novel 12-stop HDR imaging approach for dynamic scenes, leveraging a dual-camera system with an event camera and an RGB camera. The event camera provides temporally dense, high dynamic range signals that improve alignment between LDR frames with large exposure differences, reducing ghosting artifacts caused by motion. Also, a real-world finetuning strategy is proposed to increase the generalization of alignment module on real-world events. Additionally, we introduce a diffusion-based fusion module that incorporates image priors from pre-trained diffusion models to address artifacts in high-contrast regions and minimize errors from the alignment process. To support this work, we developed the ESHDR dataset, the first dataset for 12-stop HDR imaging with synchronized event signals, and validated our approach on both simulated and real-world data. Extensive experiments demonstrate that our method achieves state-of-the-art performance, successfully extending HDR imaging to 12 stops in dynamic scenes. Figure 1 : Visual comparisons of different HDR imaging methods on real-captured data. We utilize Vibrant tone mapping style in commercial HDR software (Photomatix) to better visualization. 1 Introduction High dynamic range (HDR) imaging is a fundamental task in computational photography, aiming to accurately capture the full range of visual details in both bright and dark regions of a scene. Since the typical dynamic range of an image sensor is much lower than real-world luminance ranges, which is often about 80 to 100dB [ 56 , 41 ] , post-capture processing is often required to increase sensor dynamic range. The most common solution is to merge multiple low dynamic range (LDR) images captured at varying exposure levels, known as HDR fusion. Although HDR fusion is widely used, it still faces challenges when applied to very high dynamic range scenes, like a night street with very bright light. These scenes typically require 7 to 12 exposure stops to fully recover the dynamic range [ 1 , 14 , 18 ] . Majority of HDR fusion algorithms may fail, as aligning images with very different exposures becomes challenging, considering potential variation between images due to either object motion or exposure difference. As a result, previous dynamic HDR methods are restricted to handling only 2–3 exposure images within a 6-stop range ( e.g . , -3EV to +3EV), leading to information loss in high-contrast scenes (see Fig. 1 (d)). Given that, this raises an important question: Is it possible to extend HDR imaging to 12 stops in dynamic scenes? To solve the alignment challenges, we propose an event-RGB dual-camera HDR imaging system. The additional event camera [ 47 , 2 , 7 ] can perfectly solve the alignment problem in large exposure differences, because event cameras can capture temporally dense event streams for accurate alignment, and also have a much higher intrinsic dynamic range compared to normal RGB cameras (normally 60-80dB). As shown in Fig. 1 (b), event signals provide consistent, fine-grained motion information across both bright (+6EV LDRs) and dark (-6EV LDRs) regions. Moreover, unlike previous approaches [ 38 , 48 ] that implicitly use event data to assist HDR fusion, our approach directly estimates motion between RGB frames from event streams. This design avoids ghosting artifacts as shown in Fig. 1 (e), and allows us to handle much higher exposure differences (12 stops vs. 4-6 stops in previous approaches). Another challenge is to robustly fuse LDR images with large exposure differences. Current learning-based HDR imaging methods [ 21 , 30 , 5 , 29 , 58 ] employ \mu -law tone mapping as a loss function. However, the \mu -law function alone is insufficient for effectively managing both bright and dark regions in 12-stop HDR. Moreover, in the presence of alignment error, the fusion module must also introduce artifacts, particularly at occlusion boundaries. To tackle these challenges, we propose a diffusion-based fusion module. It integrates pre-trained image priors to reduce error accumulation from the alignment module and mitigate artifacts in regions where texture details are highly compressed, as shown in Fig. 1 (f). At last, there is no 12-stop HDR imaging with event signals. Therefore, we developed the ESHDR dataset, the first dataset for 12-stop HDR imaging with synchronized event signals, using a novel simulation pipeline. While previous approaches [ 21 , 30 , 5 , 29 , 58 ] simulate multiple exposure LDRs from LDR images, our simulation pipeline generates LDR images from -6 to +6EV using HDR scenes to ensure high-quality simulation. Furthermore, to validate our methods in real-world scenarios, we built a RGB-event dual-camera system capable of capturing synchronized Event and RGB sequences across a -6EV to 6EV range. Extensive experiments on both simulated and real-world datasets demonstrate that our framework achieves state-of-the-art performance across challenging scenes, successfully pushing HDR imaging to 12 stops. Our key contributions are summarized as follows: • To the best of our knowledge, we present the first attempt at 12-stop HDR imaging for dynamic scenes. • We propose an event-assisted diffusion-based HDR method that focuses on solving two major challenges in dynamic-scene 12-stop HDR imaging: image alignment and fusion for LDRs with extreme exposure differences. • We develop the first dataset, ESHDR, for 12-stop HDR imaging with event signals, and build a dual-camera system to validate our approach on both simulated and real-world datasets, demonstrating the practical feasibility and effectiveness of our method. Figure 2 : Illustration of the proposed framework for 12-stop HDR Imaging in Dynamic Scenes. The framework consists of two main components: (a) an event-assisted alignment module and (b) a diffusion-based fusion module. 2 Related Work 2.1 RGB-based HDR imaging To capture high dynamic range (HDR) images, the most common approach is to merge multiple low dynamic range (LDR) images taken at various exposure levels. This approach, however, presents challenges in dynamic scenes, as motion often leads to ghosting artifacts [ 19 , 40 ] . Previous HDR methods for dynamic scenes [ 19 , 33 , 40 , 46 , 36 , 55 , 59 , 60 , 31 , 6 , 30 , 5 , 29 , 61 , 9 , 21 , 58 ] have focused on mitigating these artifacts through advanced image alignment techniques. Specifically, Kalantari et al. [22] introduced a patch-based synthesis technique to improve temporal continuity in HDR images by enforcing similarity across adjacent frames. [ 59 , 60 ] proposed a context-aware transformer to enhance model performance in complex dynamic scenes. Additionally, methods using deformable convolutions [ 30 , 5 ] and more sophisticated models like the pyramid cross-attention module [ 29 ] and content alignment model [ 61 ] with ghost attention and patch aggregation have been developed. [ 9 ] explored frame alignment through a luminance-based attention score, while [ 21 , 58 ] used motion flow estimation from LDRs to facilitate alignment through warping. Despite these advancements, RGB-based HDR methods still face limitations in aligning LDRs with