Hyperspectral vs. RGB for Pedestrian Segmentation in Urban Driving Scenes: A Comparative Study

Jiarong Li 1,2,†, Imad Ali Shah 1,2,†, Enda Ward 3, Martin Glavin 1,2, Edward Jones 1,2, Brian Deegan 1,2 · 2025

Read Hyperspectral vs. RGB for Pedestrian Segmentation in Urban Driving Scenes: A Comparative Study by Jiarong Li 1,2,†, Imad Ali Shah 1,2,†, Enda Ward 3, Martin Glavin 1,2, Edward Jones 1,2, Brian Deegan 1,2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11301.txt.

Introduction

Jiarong Li and co-authors test whether richer light data helps cars spot people on foot. Cheap color cameras can fail when a person and the background share the same tint. Hyperspectral imaging reads many narrow bands of light, giving each surface a fuller signature. The team used a city data set with such images of street scenes.

They cut the 128 bands down to three channels in two ways and trained three vision models. One cut picks bands by a contrast score and shared facts between bands. That cut beat plain color images on overlap and F-score for people on foot. Gains also held for riders, since feet and wheels blur less with band choice. False alarms fell, a gain that matters for safe driving.

Catalog record

CREATORJiarong Li 1,2,†, Imad Ali Shah 1,2,†, Enda Ward 3, Martin Glavin 1,2, Edward Jones 1,2, Brian Deegan 1,2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11301

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hyperspectral vs. RGB for Pedestrian Segmentation in Urban Driving Scenes: A Comparative Study Thanks: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. 1 School of Engineering, University of Galway, Ireland 2 Ryan Institute, University of Galway, Ireland 3 Valeo Vision Systems, Tuam, Ireland † Equal Contribution and correspondance ([email protected]) Jiarong Li 1,2,†, Imad Ali Shah 1,2,†, Enda Ward 3, Martin Glavin 1,2, Edward Jones 1,2, Brian Deegan 1,2 Abstract Pedestrian segmentation in automotive perception systems faces critical safety challenges due to metamerism in RGB imaging, where pedestrians and backgrounds appear visually indistinguishable.. This study investigates the potential of hyperspectral imaging (HSI) for enhanced pedestrian segmentation in urban driving scenarios using the Hyperspectral City v2 (H-City) dataset. We compared standard RGB against two dimensionality-reduction approaches by converting 128-channel HSI data into three-channel representations: Principal Component Analysis (PCA) and optimal band selection using Contrast Signal-to-Noise Ratio with Joint Mutual Information Maximization (CSNR-JMIM). Three semantic segmentation models were evaluated: U-Net, DeepLabV3+, and SegFormer. CSNR-JMIM consistently outperformed RGB with an average improvements of 1.44% in Intersection over Union (IoU) and 2.18% in F1-score for pedestrian segmentation. Rider segmentation showed similar gains with 1.43% IoU and 2.25% F1-score improvements. These improved performance results from enhanced spectral discrimination of optimally selected HSI bands effectively reducing false positives. This study demonstrates robust pedestrian segmentation through optimal HSI band selection, showing significant potential for safety-critical automotive applications. I Introduction Ensuring the safety of pedestrians remains a critical objective in the development of Advanced Driver Assistance Systems (ADAS) and Autonomous Driving (AD) [ 1 ] , with precise pedestrian segmentation increasingly recognized as a crucial factor for predicting movement patterns and safe navigation decisions in complex urban scenarios. However, conventional RGB-based segmentation systems face significant challenges due to metamerism phenomena [ 2 ] , where pedestrians appear spectrally similar to the background (such as dark clothing against asphalt) under certain lighting conditions. This spectral ambiguity leads to segmentation failures at critical object boundaries, compromising the system’s ability to accurately delineate pedestrians. Unlike detection, which requires identifying pedestrian presence, segmentation tasks demand precise object and material discrimination at the pixel level. Fig. 1: Comparison of RGB and HSI: Standard RGB captures three broad spectral bands, whereas HSI can acquire hundreds of narrow and contiguous spectral channels, providing a three-dimensional data, known as a hypercube. Sample RGB and hypercube are illustrated from the H-City dataset [ 3 ] with 128 channels, covering 450 \sim 950nm of the Electromagnetic Spectrum. Note: Color mapping is for representation purposes only. Conventional perception systems typically rely on a single RGB sensor modality or a sensor suite of RGB, LiDAR, and radar. Although each sensor modality offers unique benefits, they struggle in scenarios with diverse lighting and weather conditions, sensors with their inherent limitations [ 4 ] , such as similar color-based ambiguity in RGB imaging. For instance, RGB can fail to distinguish a person with green clothing against vegetation or white clothing against a white wall in a certain lighting condition, even though their material properties are fundamentally different. LiDAR may show limitations in adverse weather conditions, and radar offers low-resolution data, making it difficult to differentiate a pedestrian from other objects [ 5 ] . This gap in perception capability requires the exploration of alternative sensing technologies that can provide a more robust representation of the scene. Hyperspectral Imaging (HSI) is one of the most promising candidates to overcome these challenges, such as metamerism and detection in adverse weather conditions [ 6 ] . By capturing image data across hundreds of narrow, contiguous spectral bands, HSI provides a high-resolution spectral signature for each spatial pixel [ 6 ] , as shown in Fig. 1 . HSI addresses these limitations by providing detailed information on the material composition of an object that is invisible to the human eye or RGB sensors [ 7 ] . Since pedestrian detection often depends on distinguishing clothing materials from background surfaces, their materials have unique spectral signatures that can be identified by an HSI sensor [ 8 ] . This allows for segmentation based on intrinsic material properties rather than color information. However, the primary barrier to the widespread adoption of HSI in real-time applications is effective processing of its large amount of data (such as H-City: 128 bands vs 3 in RGB), which imposes significant computational demands [ 9 ] . One promising approach is to apply dimensionality-reduction techniques that preserve the most discriminative spectral information, which remains a key research objective for effective segmentation. This paper presents a comparative study to determine whether dimensionality-reduced HSI data can offer a performance advantage over standard RGB images for pedestrian segmentation. The effectiveness of two dimensionality-reduction techniques is evaluated on H-City, one of the most spectrally diverse HSI datasets in ADAS/AD [ 10 ] . The first approach is Principal Component Analysis (PCA) [ 11 ] and the second is an optimal band selection method [ 12 ] that combines Contrast Signal-to-Noise Ratio (CSNR) [ 13 ] and Joint Mutual Information Maximization (JMIM) [ 14 ] . The resulting low-dimensional HSI datasets (pseudo-RGB), along with the corresponding RGB data, are then used to train and evaluate three well-established Semantic Segmentation Models (SSM): U-Net [ 15 ] , DeepLabv3+ [ 16 ] , and Segformer [ 17 ] . By comparing their performance, we provide a quantitative answer to whether the rich spectral information from HSI, even when compressed, provides a more robust foundation for pedestrian segmentation in comparison to standard RGB images. The main contributions of this work are: • Presenting the first comparative study that evaluates HSI versus standard RGB for pedestrian segmentation in ADAS/AD applications. • Providing quantitative and qualitative evidence across three segmentation models (U-Net, DeepLabV3+, and SegFormer) that HSI can effectively address the metamerism challenge in pedestrian detection systems. • Demonstrating that optimal band selection using CSNR-JMIM significantly outperforms both RGB and PCA-based dimensionality-reduction methods. This paper is structured as follows: Section II provides related work in the fields of HSI in ADAS/AD contexts and the application of pedestrian segmentation. Section III explains the methodology for the input modalities and SSM. Section IV presents the experimental setup and a thorough analysis of the results. Section V concludes with a discussion of our findings and their implications for the future of ADAS/AD perception systems. II Related Work RGB-based pedestrian segmentation has been well researched, using traditional approaches as well as deep learning methods. Early works relied on handcrafted features such as the Histogram of Oriented Gradients (HOG) [ 18 ] , while recent approaches leverage Convolutional Neural Networks (CNN) for enhanced feature extraction. Li et al. proposed a region-based CNN method that extends Faster R-CNN by incorporating SSM to reduce false positives in pedestrian detection [ 19 ] . More recently, transformer-based architectures have shown promising results, with Vision Transformers (ViTs) [ 20 ] demonstrating superior performance in complex scenarios. However, RGB-based methods still face significant challenges in varying and adverse lighting, shadows, and weather conditions, limiting their robustness in real-world applications [ 4 , 10 ] . To overcome the limitations of a single modality, researchers have explored

Read the complete text