Multi-Scale Spectral Attention Module-based Hyperspectral Segmentation in Autonomous Driving Scenarios

Imad Ali Shah 1,2, Jiarong Li 1,2, Tim Brophy 1,2, Martin Glavin 1,2, Edward Jones 1,2, Enda Ward 3, and Brian Deegan 1,2 · 2025

Read Multi-Scale Spectral Attention Module-based Hyperspectral Segmentation in Autonomous Driving Scenarios by Imad Ali Shah 1,2, Jiarong Li 1,2, Tim Brophy 1,2, Martin Glavin 1,2, Edward Jones 1,2, Enda Ward 3, and Brian Deegan 1,2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18682.txt.

Introduction

A self-driving car that sees in hundreds of colors, not three, could read wet roads and fog more surely, but the data flood is heavy. Imad Ali Shah and co-authors lighten the reading with an attention module tuned to spectra, in a 2025 paper on hyperspectral image segmentation for driving scenes.

Their module listens at three scales at once, running parallel one-dimensional convolutions with kernel sizes from one to eleven along the spectrum, then merges what each scale hears. Plugged into the skip connections of a UNet, it lifted overlap scores by about two point three percent and F1 by about two point nine over the baseline, at competitive graphics cost. Ablations showed the best scale trio depends on the dataset. The authors offer the module as a practical step toward hyperspectral perception in cars.

Catalog record

CREATORImad Ali Shah 1,2, Jiarong Li 1,2, Tim Brophy 1,2, Martin Glavin 1,2, Edward Jones 1,2, Enda Ward 3, and Brian Deegan 1,2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-18682

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Scale Spectral Attention Module-based Hyperspectral Segmentation in Autonomous Driving Scenarios Imad Ali Shah 1,2, Jiarong Li 1,2, Tim Brophy 1,2, Martin Glavin 1,2, Edward Jones 1,2, Enda Ward 3, and Brian Deegan 1,2 Abstract Recent advances in autonomous driving (AD) have highlighted the potential of Hyperspectral Imaging (HSI) for enhanced environmental perception, particularly in challenging weather and lighting conditions. However, efficiently processing its high-dimensional spectral data remains a significant challenge. This paper introduces a Multi-scale Spectral Attention Module (MSAM) that enhances spectral feature extraction through three parallel 1D convolutions with varying kernel sizes between 1 to 11, coupled with an adaptive feature aggregation mechanism. By integrating MSAM into UNet’s skip connections (UNet-SC), our proposed UNet-MSAM achieves significant improvements in semantic segmentation performance across multiple HSI datasets: HyKo-VIS v2, HSI-Drive v2, and Hyperspectral City v2. Our comprehensive experiments demonstrate that with minimal computational overhead (on average 0.02% in parameters and 0.82% GFLOPS), UNet-MSAM consistently outperforms UNet-SC, achieving average improvements of 3.61% in mean IoU and 3.80% in mF1 across the three datasets. Through extensive ablation studies, we have established that multi-scale kernel combinations perform better than single-scale configurations. These findings demonstrate the potential of HSI processing for AD and provide valuable insights into designing robust, multi-scale spectral feature extractors for real-world applications. \receiveddate 12 June, 2025 \reviseddate XX Month, XXXX \accepteddate XX Month, XXXX \publisheddate XX Month, XXXX \currentdate 12 June, 2025 \doiinfo OJVT.2025.XXXXXXX \authornote This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. {IEEEkeywords} Autonomous Driving, Hyperspectral Image Segmentation, U-Net, HSI-Drive, Hyperspectral City, HyKo 1 INTRODUCTION \IEEEPARstart Autonomous driving (AD) systems require robust and reliable perception capabilities to ensure safe navigation in diverse environmental conditions. While traditional RGB imaging has been the cornerstone of computer vision in Advanced Driver Assistance Systems (ADAS), Hyperspectral Imaging (HSI) has recently emerged as a promising technology to address critical limitations in current perception systems. Unlike traditional RGB cameras that capture just three spectral bands (i.e. Red, Green, and Blue), HSI cameras can acquire tens to hundreds of bands across the electromagnetic spectrum as shown in Fig. 1 with sample images taken from the HSI-Drive v2 (HSI-Drive) [ 1 , 2 ] , and Hyperspectral City v2 (H-City) [ 3 ] datasets. The additional spectral information embedded in HSI can provide detailed material characterization and relevant spectral information about the observed scene and object [ 4 ] . These HSI capabilities have already demonstrated significant value in established fields such as remote sensing [ 5 ] , precision agriculture [ 6 ] , and medicine [ 7 ] . Figure 1: Comparison of RGB plane and HSI channels: (left) HSI-Drive’s pseudo-RGB [ 2 ] vs (right) H-City’s RGB images [ 3 ] and planes along with HSI channels of 25 and 128, respectively. Manual colour mapping has been performed on the HSI channels for representation purposes. One of the key potential benefits of HSI in AD is its ability to address metamerism [ 8 ] , a phenomenon where objects with different material compositions appear identical under specific lighting conditions with standard RGB imaging, but they can be distinguished through their unique spectral signatures. This capability has the potential to improve road hazard identification, material differentiation including water, snow, and ice on the road [ 9 ] , and reliable object segmentation under varying lighting conditions. However, the integration of HSI into ADAS presents unique challenges distinct from its traditional applications such as in remote sensing and medicine, where the relatively static image acquisition-based scenarios can accommodate typical line-scan and slow frame rate HSI cameras. However, the dynamic nature of AD scenarios demands portable "snapshot-based" HSI cameras capable of real-time video capture. Recent technological advances have led to the development of compact and cost-effective snapshot HSI cameras [ 10 ] , enabling the collection of annotated HSI datasets specifically for ADAS scenarios. However, achieving broad spectral and spatial coverage with affordable snapshot cameras remains challenging, particularly while maintaining real-time processing capabilities. This has limited the development of perception systems that leverage the capabilities of HSI sensors and consequently, the majority of HSI studies in the ADAS/AD field have concentrated on gathering data and presenting initial findings largely focusing on individual datasets. Despite the inherent challenges of HSI in the ADAS/AD domain, researchers have made efforts to utilize HSI data and have established performance benchmarks comparing HSI with RGB [ 10 , 11 ] and also evaluating baseline semantic segmentation models [ 12 ] on HSI data, using the available annotated datasets and utilising deep learning architectures. These studies have demonstrated the potential of HSI while highlighting its unique challenges. However, Shen et al. [ 3 ] indicate that existing deep learning architectures are primarily optimized for RGB, which often fails to fully utilize the HSI’s rich spectral information. This limitation is mainly due to two key factors: (1) These models are originally designed to extract spatial domain features from RGB images, and the use of pre-trained models (e.g., on ImageNet [ 13 ] ) primarily benefits RGB images. (2) When applied to HSI data, researchers typically reduce the high-dimensional spectral information to fit RGB-oriented models through dimensionality reduction techniques (such as single convolutional layers, principal component analysis, band subselection [ 14 ] etc), resulting in potential loss of the distinctive spectral characteristics within HSI [ 15 ] . This limitation highlights the need for specialized deep learning architectures that can effectively utilize dense spectral information of HSI while maintaining computational efficiency. Drawing intuition from the extensive research in remote sensing [ 16 ] and channel-wise feature extraction for HSI in ADAS/AD scenarios [ 12 ] where spectral feature extraction has proven beneficial for object classification, we propose a multi-scale spectral attention module (MSAM) for more effective feature extraction from HSI data in ADAS/AD applications. Our approach employs parallel kernels for multi-scale feature extraction using 1D convolution and a fusion-based attention mechanism (AM). The proposed MSAM, integrated into the Skip Connections (SC) variant [ 17 ] of the UNet [ 18 ] architecture, demonstrates superior performance in semantic segmentation compared to the baseline UNet-SC. This work establishes a foundation for further studies to explore better extraction and fusion of spectral and spatial data to improve the overall segmentation for ADAS/AD scenarios. The main contributions of this paper are as follows: • Development of MSAM, an attention mechanism specifically designed for HSI feature extraction, by implementing parallel multi-scale kernels and efficiently integrating it into UNet-SC • Comprehensive evaluation across three HSI datasets (HyKo-VIS, HSI-Drive, H-City) with detailed ablation studies on kernel size combinations, establishing a framework for multi-scale spectral kernels based AM integration into SSMs • With minimal additional computational overhead (0.02% in parameters and 0.82% GFLOPS), UNet-MSAM provides significant performance gains of over 3.61% in mean IoU and 3.80% in mean F1 across the three HSI datasets in comparison to baseline UNet-SC The remainder of this paper is organized as follows: Section 2 reviews the current state of HSI based perception for ADAS/AD, Section 3 presents the methodology for the proposed MSAM, Section 4 presents experiments and results, and Section 5 concludes the study with a summary of contributions

Read the complete text