Automatic Spectral Calibration of Hyperspectral Images: Method, Dataset and Benchmark

Zhuoran Du, Shaodi You, Cheng Cheng, Shikui Wei · 2024

Read Automatic Spectral Calibration of Hyperspectral Images: Method, Dataset and Benchmark by Zhuoran Du, Shaodi You, Cheng Cheng, Shikui Wei (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14925.txt.

Introduction

Zhuoran Du and co-authors automate the color calibration of hyperspectral cameras. Their 2024 paper replaces the physical reference cards normally used, which block views and tie cameras down, with a learned method that infers the lighting from the image itself.

They built a large calibration data set of seven hundred sixty-five scene pairs under varied natural light, then expanded it tenfold with measured illuminations. Their model, a spectral illumination transformer with an attention module, estimates the light and corrects the image. Benchmarks put it ahead of rivals, with low-light scenes proving hardest. Data and code are public. The authors aim to free hyperspectral imaging for fieldwork where carrying reference gear is impractical.

Catalog record

CREATORZhuoran Du, Shaodi You, Cheng Cheng, Shikui Wei
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14925

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automatic Spectral Calibration of Hyperspectral Images: Method, Dataset and Benchmark Zhuoran Du, Shaodi You, Cheng Cheng, Shikui Wei Abstract Hyperspectral image (HSI) densely samples the world in both the space and frequency domain and therefore is more distinctive than RGB images. Usually, HSI needs to be calibrated to minimize the impact of various illumination conditions. The traditional way to calibrate HSI utilizes a physical reference, which involves manual operations, occlusions, and/or limits camera mobility. These limitations inspire this paper to automatically calibrate HSIs using a learning-based method. Towards this goal, a large-scale HSI calibration dataset is created, which has 765 high-quality HSI pairs covering diversified natural scenes and illuminations. The dataset is further expanded to 7650 pairs by combining with 10 different physically measured illuminations. A spectral illumination transformer (SIT) together with an illumination attention module is proposed. Extensive benchmarks demonstrate the SoTA performance of the proposed SIT. The benchmarks also indicate that low-light conditions are more challenging than normal conditions. The dataset and codes are available online: https://github.com/duranze/Automatic-spectral-calibration-of-HSI 1 Introduction Figure 1 : Comparison of HSI Calibration Methods: (a) Synchronous, (b) Asynchronous, and (c) Automatic. Hyperspectral image (HSI) densely samples the world in both spatial and frequency domains. It is more informative than RGB images for recognizing material. And therefore it is widely used in remote sensing and laboratory analysis [ 21 ] [ 4 ] [ 6 ] [ 17 ] . However, HSI appearance is highly dependent on global illumination. While in remote sensing and lab environments, global illuminations are highly uniform in a large spatial and temporal range or can be well controlled; global illuminations in natural scenes are highly varied spatially and temporally. Therefore, to capture HSI in natural scenes, global illumination needs to be measured simultaneously. There are two traditional ways to measure the global illumination. The synchronous method [ 3 ] (Fig. 1 .a) utilizes a reference such as a barium sulfate panel (an ideal white panel) [ 16 ] in the scene. The disadvantage is the panel partly occludes the scene. The asynchronous method [ 8 , 2 ] (Fig. 1 .b) takes two HSIs of the same scenes quickly with and without the reference. The disadvantage is that this method requires double shots, manual operations, and an unmoved camera. These disadvantages inspire this paper: can we learn to automatically calibrate the HSI without a physical reference? If the accuracy is close to the physical methods, then the learning-based method has the advantages of less manual operations, no occlusion, and more mobility. To enable this research, we have created the first HSI auto calibration dataset BJTU-UVA Dataset . The dataset is captured using a 204 bands 400-1000nm hyperspectral camera with a 512 \times 512 spatial resolution, and 12bits quantization. The dataset has 765 calibrated and uncalibrated image pairs which are obtained using the asynchronous method. The dataset covers diversified scenes and illuminations. To further study the illumination calibration, an expansion dataset BJTU-UVA-E is created, which composite calibrated HSIs with 10 different physically measured illuminations. In such a way, the BJTU-UVA-E has 7650 uncalibrated HSIs. It is worth noticing that BJTU-UVA is not only the first HSI calibration dataset, it is also one of the largest public HSI datasets. After that, we propose the Spectral Illumination Transformer (SIT), which is the first learning-based HSI calibration method to the best of our knowledge. SIT is based on a U-shape transformer. To allow SIT to learn illumination properly, we propose the illumination attention module which is inspired by the classic Gray-World [ 5 ] algorithm. We consider SIT the baseline of learning-based methods which may inspire future research in this new topic. Detailed benchmarks are provided using the BJTU-UVA dataset. Given that there are no other learning-based methods for this novel task, methods for HSI denoising and reconstruction are compared. And classic method of Gray-World is compared. It is shown that SIT is the state-of-the-art. It is interesting to find that Gray-World performs comparable to other learning-based methods on real illuminations. It is also noticed that low-light scenes/areas are likely to have larger errors; which can be a future research topic. The novelty of the paper can be summarized as: • We propose a novel research topic automatic HSI calibration, which has no occlusion, less manual operations, and more mobility than traditional methods. • We propose the large-scale HSI dataset BJTU-UVA, which is the first dataset for HSI auto-calibration. • We propose the spectral illumination transformer (SIT) with the illumination attention module, which is the very first transformer-based method on this topic. 2 Related Work 2.1 Hyperspectral Image Calibration In the literature, several spectral calibrations using physical references are developed for remote sensing and indoor optics. In remote sensing, calibration is done using a physical reference (Fig. 1 .a). Pan et al. in 2001 [ 25 ] used a spectroradiometer calibrated against a barium sulfate panel (the physical reference), assuming constant solar irradiance; Ferreira et al. in 2004 [ 11 ] used linear interpolation of pre- and post-flight reference panel readings; and Miura et al. in 2008 [ 22 ] used continuous ground-based panel readings during flight to adjust data. The limitation of applying this method to natural scenes is that there always exists occlusion from the white reference. In laboratory settings, where illumination can be precisely controlled, calibration can be performed asynchronously (Fig. 1 .b). Under these conditions, illumination only needs to be measured once and can then be applied in subsequent measurements. Recently, Moghadam et al. [ 23 ] proposed a data-driven method to enable indoor hyperspectral imaging using affordable and commonly available LED and fluorescent lighting sources. However, in the natural outdoor scenes, controlling illumination is infeasible. Therefore, to apply asynchronous methods, after measuring the illumination, the HSI has to be taken as soon as possible. This requires manual operations and a stable camera. 2.2 RGB White Balancing RGB white balancing is a similar topic but with major differences. First, each of the RGB channels is a wide-band channel whereas in HSI the channel is a narrow band. Second, RGB bands usually overlap with each other whereas HSI bands do not or have limited overlaps. Third, many RGB-based methods aim for optimizing human perception while HSI calibration aims for physical accuracy. Recent advances in white balancing for sRGB images have primarily leveraged deep learning techniques to enhance color correction. The two-stage framework by Farghaly et al. [ 10 ] combines global color mapping with local adjustments, achieving perceptually improved results. Stability across varying color temperatures has been addressed by Li et al. [ 18 ] with their SWBNet, which learns color temperature-insensitive features and applies a novel contrastive loss for consistent outcomes. Addressing the complexity of mixed-illuminant scenes, Afifi et al. [ 1 ] blend images with predefined white-balance settings using learned weighting maps, while Kinli et al. [ 15 ] model lighting variations as style factors, enhancing correction capabilities without explicit illuminant detection. Lastly, efficiency in processing is significantly enhanced by [ 14 ] through deterministic illumination mapping, facilitating real-time applications. 2.3 HSI Restoration and Reconstruction Learning-based research on HSI is mainly focusing on denoising and band reconstruction. This research explores many new deep-learning architectures for HSI because it is not

Read the complete text