GN-FR: Generalizable Neural Radiance Fields for Flare Removal
Gopi Raju Matta ∗, Rahul Siddartha, Rongali Simhachala Venkata Girish, Sumit Sharma, Kaushik Mitra · 2024
Read GN-FR: Generalizable Neural Radiance Fields for Flare Removal by Gopi Raju Matta ∗, Rahul Siddartha, Rongali Simhachala Venkata Girish, Sumit Sharma, Kaushik Mitra (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08200.txt.
Introduction
Gopi Raju Matta and four co-authors remove lens flare from photographs by treating the problem as three-dimensional. Flare comes from stray light scattering inside a lens, and it takes many forms, from halos and streaks to haze. Single-image cleanup methods struggle because the task is underdetermined. Flare, though, changes with viewpoint, so neighboring views of the same scene hold clues about what the flare hides.
Their system, GN-FR, builds on neural radiance fields that render a scene from sparse photos. Added modules estimate where flare sits, sample views and points, and learn to render clean images without ever seeing clean ground truth, guided by a masking loss. The team also releases a dataset of 17 real flare scenes with 782 images and mask annotations. They claim the first flare removal inside this framework.
Catalog record
| CREATOR | Gopi Raju Matta ∗, Rahul Siddartha, Rongali Simhachala Venkata Girish, Sumit Sharma, Kaushik Mitra |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08200
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GN-FR: Generalizable Neural Radiance Fields for Flare Removal Gopi Raju Matta ∗, Rahul Siddartha, Rongali Simhachala Venkata Girish, Sumit Sharma, Kaushik Mitra Abstract Flare, an optical phenomenon resulting from unwanted scattering and reflections within a lens system, presents a significant challenge in imaging. The diverse patterns of flares—such as halos, streaks, color bleeding, and haze—complicate the flare removal process. Existing traditional and learning-based methods have exhibited limited efficacy due to their reliance on single-image approaches, where flare removal is highly ill-posed. We address this by framing flare removal as a multi-view image problem, taking advantage of the view-dependent nature of flare artifacts. This approach leverages information from neighboring views to recover details obscured by flare in individual images. Our proposed framework, GN-FR (Generalizable Neural Radiance Fields for Flare Removal), can render flare-free views from a sparse set of input images affected by lens flare and generalizes across different scenes in an unsupervised manner. GN-FR incorporates several modules within the Generalizable NeRF Transformer (GNT) framework: Flare-occupancy Mask Generation (FMG), View Sampler (VS), and Point Sampler (PS). To overcome the impracticality of capturing both flare-corrupted and flare-free data, we introduce a masking loss function that utilizes mask information in an unsupervised setting. Additionally, we present the first-of-its-kind 3D multi-view flare dataset, comprising 17 real flare scenes with 782 images, 80 real flare patterns, and their corresponding annotated flare-occupancy masks. To our knowledge, this is the first work to address flare removal within a Neural Radiance Fields (NeRF) framework. GN-FR † † email: [email protected] † † email: [email protected] † † email: [email protected] † † email: [email protected] † † email: [email protected] † † affiliation: Computational Imaging Lab, Dept. of Electrical Engineering, Indian Institute of Technology Madras, Chennai, India † † footnotetext: * Corresponding author 1 Introduction The existence of flare artifacts can degrade image quality, potentially obscuring crucial object or scene details. Variations in lens characteristics, light source placement, and the camera’s orientation towards the light source can result in diverse flare types. This diversity makes flare removal an ill-posed problem. Recent works [ Wu et al.(2021)Wu, He, Xue, Garg, Chen, Veeraraghavan, and Barron , Dai et al.(2023a)Dai, Li, Zhou, Feng, Luo, and Loy , Dai et al.(2023b)Dai, Luo, Zhou, Li, and Loy , Zhang et al.(2023)Zhang, Ouyang, Liu, Wang, Kong, and Jin ] attempted to remove flare by utilizing the flare mask as a guiding cue in conjunction with paired data by using Unet and transformer-based architectures. Qiao et al. [ Qiao et al.(2021)Qiao, Hancke, and Lau ] utilized adversarial networks to learn from both flare and flare-free unpaired data by acknowledging the position of flare and light source. Lately, kotp et al. [ Kotp and Torki(2024) ] leveraged depth information as a guide to remove flare. However, none of the aforementioned papers exploited the directional dependence of flare. We propose to utilize multi-view settings for flare removal, leveraging information lost due to flare in the current view from adjacent views. With the current framework, we can achieve novel view synthesis from images corrupted by flare and simultaneously eliminate the flare artifacts. Figure 1 : Our method: GN-FR is built on significant modifications to the GNT framework with flare masking loss, an effective view sampler and a Point Sampler. Our method consistently removes flare across the views on synthetic flare-imposed scenes and real scenes with flare. Neural scene representations such as NeRF [ Mildenhall et al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng ] can render realistic novel views, but they are limited by a tedious optimization process for each new scene and cannot generalize to new scenes. Recent works on generalizable neural rendering like IBRNet [ Wang et al.(2021)Wang, Wang, Genova, Srinivasan, Zhou, Barron, Martin-Brualla, Snavely, and Funkhouser ] , PixelNerf [ Yu et al.(2021)Yu, Ye, Tancik, and Kanazawa ] , NeuRay [ Liu et al.(2022)Liu, Peng, Liu, Wang, Wang, Theobalt, Zhou, and Wang ] address these limitations by synthesizing a generalizable 3D representation through aggregation of image features extracted from seen views. We leverage GNT [ T et al.(2023)T, Wang, Chen, Chen, Venugopalan, and Wang ] , a recent powerful method that utilizes transformers as aggregation functions to extract features from source views and to predict the target pixel color, eliminating the usage of the volumetric rendering equation. This makes GNT a reliable framework for flare removal alongside generating novel views. However, vanilla GNT cannot remove the flare as the network cannot learn to identify the flare region without sufficient cues. By leveraging flare occupancy masks as cues, we direct the network’s attention to the flare-affected regions, enabling it to learn how to remove flare and recover lost information from neighboring views effectively. This underscores our central concept that information obscured in one view can be extracted from adjacent views. To this end, we propose our model GN-FR: Generalizable Neural Radiance Fields for Flare Removal, which is built on effective modifications to the GNT framework. Our model includes three sub-modules: 1) Flare-occupancy Mask Generation network (FMG): For each input view, it generates a binary mask to detect the regions w/ and w/o flare. 2) View Sampler (VS): Samples the views that are minimally affected by flare using the flare occupancy mask as a cue. 3) Point Sampler (PS): An effective attention masking-based sampling mechanism in the view transformer to guide the network to attend only to points that are valid in the flare occupancy mask. As ground truth is not available for real scenes, we propose Masking loss , this will only attend to the pixels where there is no flare while calculating loss. Hence it ensures the rendering of the regions where the flare is prominent by extracting information from corresponding nearby views’ regions where there is no flare. In summary, our contributions include: • A generalizable neural rendering network for flare removal that can render novel views and effectively remove the flare in complex scenarios. • Masking loss, Flare-occupancy Mask Generator (FMG), View Sampler (VS) and Point Sampler (PS) modules to intelligently sample only flare-free regions in the source views for target view rendering. • A first-of-its-kind 3D flare dataset of 17 scenes, together with real flare pattern dataset of 80 different flare patterns of real flare to enable our network to detect flare affected regions well and then to effectively de-flare them. To the best of our knowledge, this is the first work that does flare removal and also produces novel views in a single framework. 2 Related Work 2.1 Flare Removal Flare removal remained a challenging problem for a longer period due to its inherent complexity and diverse reasons causing flare. Despite these challenges, recent traditional [ Zhou et al.(2023)Zhou, Liang, Chen, Huang, Yang, and Li ] and deep learning [ Dai et al.(2023a)Dai, Li, Zhou, Feng, Luo, and Loy , Dai et al.(2023b)Dai, Luo, Zhou, Li, and Loy ] based methods have shown decent progress. Wu et al. [ Wu et al.(2021)Wu, He, Xue, Garg, Chen, Veeraraghavan, and Barron ] proposed Unet-based flare removal using paired data and flare occupancy mask information, where the mask information is used to blend the input light source back into the prediction where the light source is removed during training.Qiao et al. [ Qiao et al.(2021)Qiao, Hancke, and Lau ] utilized