Multi-Modal Camera-Based Detection of Vulnerable Road Users

Penelope Brown, Julie Stephany Berrio Perez, Mao Shan, Stewart Worrall · 2025

Read Multi-Modal Camera-Based Detection of Vulnerable Road Users by Penelope Brown, Julie Stephany Berrio Perez, Mao Shan, Stewart Worrall (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06333.txt.

Introduction

Pedestrians and cyclists die in traffic at high rates, and Penelope Brown and three co-authors build a camera system to spot them sooner in a 2025 paper. Vulnerable road users are hard to detect in poor light, bad weather, and skewed datasets. The framework fuses ordinary color images with thermal infrared and fine-tunes a YOLOv8 detector.

Training draws on the KITTI, BDD100K, and FLIR datasets, with class re-weighting and light augmentation to lift rare classes. Experiments find a sweet spot in image size and partial backbone freezing, and class-weighted losses raise recall for the rarest users. Thermal models reach the highest precision, while color-to-thermal augmentation boosts recall, pointing to safer intersections through fused sensing.

Catalog record

CREATORPenelope Brown, Julie Stephany Berrio Perez, Mao Shan, Stewart Worrall
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06333

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Modal Camera-Based Detection of Vulnerable Road Users Thanks: This research was supported by the 2025 Engineering Vacation Research Internship (VRI) Winter Program. Penelope Brown, Julie Stephany Berrio Perez, Mao Shan, Stewart Worrall Abstract Vulnerable road users (VRUs) such as pedestrians, cyclists, and motorcyclists represent more than half of global traffic deaths, yet their detection remains challenging in poor lighting, adverse weather, and unbalanced data sets. This paper presents a multimodal detection framework that integrates RGB and thermal infrared imaging with a fine-tuned YOLOv8 model. Training leveraged KITTI, BDD100K, and Teledyne FLIR datasets, with class re-weighting and light augmentations to improve minority-class performance and robustness, experiments show that 640-pixel resolution and partial backbone freezing optimise accuracy and efficiency, while class-weighted losses enhance recall for rare VRUs. Results highlight that thermal models achieve the highest precision, and RGB-to-thermal augmentation boosts recall, demonstrating the potential of multimodal detection to improve VRU safety at intersections. 1 Introduction Figure 1: Overview of the multimodal training pipeline integrating RGB and thermal data for vulnerable road user detection using the YOLOv8 model Road safety is a major concern throughout the world, particularly in relation to protecting vulnerable road users (VRUs). Globally, road traffic accidents claim about 1.2 million lives per year, and more than half of these fatalities are VRUs [ 1 ] . Many regions are seeing increasing trends in VRU casualties despite overall safety efforts. For example, Australia recorded 1,272 road deaths in 2023 (a 7.3% increase from 2022) [ 2 ] , with deaths among VRUs showing the highest surge of all user groups since 2020. VRUs are broadly defined to include pedestrians, cyclists, motorcyclists, riders of micromobility devices (such as e-scooters), users of mobility aids (e.g. wheelchairs) [ 3 ] , and road crossing animals. Improving the safety of these diverse participants is a complex problem, as VRUs are typically small, move at inconsistent speeds, and exhibit unpredictable behaviour in traffic [ 4 ] . Their lack of physical protection makes them disproportionately vulnerable in collisions and accounts for an acute and growing share of road fatalities, underscoring the need for effective detection and warning systems. Advances in computer vision and sensor technology offer new opportunities to mitigate VRU risks through early detection and alerting. In advanced driver assistance systems (ADAS) and intelligent transportation infrastructure, the ability to automatically detect VRUs in real time is needed to prevent accidents or trigger timely warnings. However, reliable VRU detection poses several key challenges. First, many VRUs (especially pedestrians and cyclists) appear as relatively small objects in images or sensor data, making them harder to detect than larger vehicles. Second, VRU detection is inherently a multiclass problem, the system must recognise a variety of classes (people, bikes, motorcycles, animals, etc.), which complicates training and often leads to unbalanced performance. A further challenge lies in environmental and sensor limitations. Most VRU detection implementations to date rely on a single sensor modality, often a visible-light (RGB) camera, due to its rich image detail and low cost. However, relying solely on an optical camera can be problematic in conditions such as night, glare, or fog, when visible-spectrum imaging may fail to clearly capture pedestrians or cyclists. Thermal infrared cameras, on the other hand, passively sense heat emissions and can readily highlight humans or animals in darkness or high-contrast lighting, but thermal imagery lacks colour/texture details and may have lower resolution. No single sensor is universally reliable in all scenarios. This limitation has prompted a trend toward multi-modal approaches in recent VRU detection research. Currently, the rapid progress of the past decade in object detection algorithms has equipped us with powerful models capable of accurately recognising various road users in real time. The introduction of the original YOLO model in 2016 demonstrated that a single CNN could directly predict bounding boxes and class probabilities at 45 frames per second. YOLOv8 (released in 2023) incorporates advanced network designs and training strategies that offer enhanced performance and robustness compared to previous versions. These state-of-the-art detectors provide a strong foundation for tackling the VRU detection problem, as they can be fine-tuned on specialised datasets to recognise the critical VRU classes. However, achieving an equally reliable detection for VRUs classes in challenging low-light conditions still requires careful consideration of training methodologies, data enhancement, and sensor input. In this paper, we address the above challenges by proposing a multimodal VRU detection approach that uses RGB and thermal camera data with a cutting-edge object detector. The system is designed for deployment at road intersections to actively identify nearby VRUs (e.g. pedestrians, cyclists, and scooter riders) and alert drivers in real time. We adopt a fine-tuned YOLOv8 model as the core detector, taking advantage of its balance of speed and accuracy for real-time performance. To boost detection of under-represented VRU classes, we employ targeted training enhancements and extensive data augmentation to improve diversity. These techniques are vital for developing a robust high-performance model that is suited to the complexities of VRU detection. In the following sections, we detail the related work in this domain and then present our methodology and results. 2 Related Work Early object detection approaches combined innovative features with boosting and cascades to achieve real-time performance. Viola and Jones [ 5 ] pioneered a face detector using an Integral Image for a rapid Haar-like feature computation, enabling AdaBoost-based detection at 15 frames per second. Shortly after, Dalal and Triggs [ 6 ] introduced Histograms of Orientated Gradients (HOG) within a sliding window framework, attaining near-perfect pedestrian detection in the MIT dataset and establishing the efficacy of robust feature descriptors for object detection. The rise of deep learning further revolutionised the field: He et al. [ 7 ] proposed the deep Residual Network (ResNet) that won the 2015 ImageNet competition by enabling substantially increased network depth without vanishing gradients, marking a new state of the art in image feature learning. Modern detectors build on these foundations with region proposal mechanisms and end-to-end training. Girshick et al. [ 8 ] introduced the Region-Based Convolutional Neural Network (R-CNN), which improved the mean average precision (mAP) by more than 50% relative to the previous PASCAL VOC champion, although at the cost of multiple-stage processing. Ren et al. [ 9 ] later developed Faster R-CNN, integrating a learnable region proposal network for near real-time performance while maintaining high detection accuracy. In parallel, Redmon et al. [ 10 ] pioneered one-stage detectors with the You Only Look Once (YOLO) model, which forwent explicit proposal generation in favour of a single unified network, dramatically increasing the inference speed to 45 FPS with a slight trade-off in localisation precision. Subsequent YOLO variants (v2 through v8) progressively improved the accuracy-speed trade-off, incorporating advances like multiscale predictions and deeper backbones. The latest YOLOv8, introduced in 2023 [ 11 , 12 ] , achieves state-of-the-art real-time performance on general object benchmarks. Although even more recent iterations (e.g., a YOLOv11 prototype) report further accuracy gains, they demand greater computational power and lack the ecosystem maturity of YOLOv8 [ 13 ] . Consequently, YOLOv8 remains a popular choice for real-time detection tasks, offering a robust baseline upon which our work builds on for vulnerable road user recognition. Despite progress in generic object detection, detecting vulnerable road users

Read the complete text