This paper has been accepted for publication at the IEEE International Joint Conference on Neural Networks (IJCNN), Rome, Italy 2025 RGB-Event Fusion with Self-Attention for Collision Prediction

Pietro Bonazzi 1 , Christian Vogt 1 , Michael Jost 1 , Haotong Qin 1 , Lyes Khacef 2 , Federico Paredes-Valles 2 , Michele Magno 1 · 2025

Read This paper has been accepted for publication at the IEEE International Joint Conference on Neural Networks (IJCNN), Rome, Italy 2025 RGB-Event Fusion with Self-Attention for Collision Prediction by Pietro Bonazzi 1 , Christian Vogt 1 , Michael Jost 1 , Haotong Qin 1 , Lyes Khacef 2 , Federico Paredes-Valles 2 , Michele Magno 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04258.txt.

Introduction

Pietro Bonazzi and co-authors fuse two camera kinds to warn drones of coming crashes. Their 2025 paper joins standard color video with event cameras, which report only pixel changes and react very fast. Two encoder arms read each stream, and a self-attention step blends the reads to call crash time and place.

On a shared test set at fifty calls per second, the fused model beat single-stream models by a small mean margin, and by more at longer range. Event-only reads beat color-only reads by wide margins on time error, at like cost. Shrunk event models trade a bit of accuracy for big savings. The gains cost more memory and math, a fair price for safer flight, the authors suggest.

Catalog record

CREATORPietro Bonazzi 1 , Christian Vogt 1 , Michael Jost 1 , Haotong Qin 1 , Lyes Khacef 2 , Federico Paredes-Valles 2 , Michele Magno 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-04258

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

This paper has been accepted for publication at the IEEE International Joint Conference on Neural Networks (IJCNN), Rome, Italy 2025 RGB-Event Fusion with Self-Attention for Collision Prediction Thanks: This work was funded by the Swiss National Science Foundation (Grant 219943) and by the SONY ”Research Award Program (RAP)”. Pietro Bonazzi 1 , Christian Vogt 1 , Michael Jost 1 , Haotong Qin 1 , Lyes Khacef 2 , Federico Paredes-Valles 2 , Michele Magno 1 Abstract Ensuring robust and real-time obstacle avoidance is critical for the safe operation of autonomous robots in dynamic, real-world environments. This paper proposes a neural network framework for predicting the time and collision position of an unmanned aerial vehicle with a dynamic object, using RGB and event-based vision sensors. The proposed architecture consists of two separate encoder branches, one for each modality, followed by fusion by self-attention to improve prediction accuracy. To facilitate benchmarking, we leverage the ABCD [ 8 ] dataset collected that enables detailed comparisons of single-modality and fusion-based approaches. At the same prediction throughput of 50Hz, the experimental results show that the fusion-based model offers an improvement in prediction accuracy over single-modality approaches of 1% on average and 10% for distances beyond 0.5m, but comes at the cost of +71% in memory and + 105% in FLOPs. Notably, the event-based model outperforms the RGB model by 4% for position and 26% for time error at a similar computational cost, making it a competitive alternative. Additionally, we evaluate quantized versions of the event-based models, applying 1- to 8-bit quantization to assess the trade-offs between predictive performance and computational efficiency. These findings highlight the trade-offs of multi-modal perception using RGB and event-based cameras in robotic applications. Reproducibility The code base and dataset can be installed using the link : https://github.com/pbonazzi/eva . I Introduction Unmanned aerial vehicles (UAVs) have revolutionized various fields, including logistics, surveillance, agriculture, and disaster management. As UAVs are deployed in dynamic environments [ 33 ] , efficient and accurate obstacle avoidance is critical for ensuring safety and operational effectiveness. Traditional frame-based cameras have been widely used for obstacle avoidance due to their high spatial resolution and compatibility with conventional computer vision algorithms [ 34 ] . However, their limitations such as motion blur, high latency, and low dynamic range hinder their performance in high-speed or low-light scenarios [ 35 ] . In contrast, event cameras, also referred to as event-based vision sensors (EVS), offer a promising alternative by capturing changes in pixel intensity asynchronously, resulting in significantly lower latency, higher temporal resolution, and a wider dynamic range [ 15 , 18 , 7 , 13 , 24 ] . These advantages make event cameras particularly well-suited for applications that require fast and precise motion analysis, including obstacle avoidance [ 29 , 12 , 6 , 25 ] . The maturity of the field has led to real-world demonstrators [ 17 , 5 ] , underscoring the demand for robust, real-time collision avoidance mechanisms in drone navigation. However, in-field demonstrations introduce significant challenges [ 22 ] , such as environment noise, and introduce the need for low-latency processing in dynamic environments [ 30 ] . These challenges necessitate multi-sensor fusion strategies. Event-based cameras capture pixel-level brightness changes with microsecond latency, enabling detection of fast-moving objects even in changing or low-light environments. In contrast, RGB cameras provide rich spatial and color information. By fusing these complementary data streams, researchers have shown it is possible to enhance the robustness of deep learning vision algorithms [ 38 , 10 , 14 , 26 , 16 ] . This work specifically tackles the problem of accurately predicting both the position and time of potential collisions of an object by integrating event-based and RGB data. A visual overview of the proposed framework is provided in Fig. 1 . Figure 1: A visual overview of the proposed architecture : (1) The FPGA-based drone setup; (2) Input data streams, including event-based and frame-based data; (3) Preprocessing and data transformation pipeline; (4) Neural network architectures; (5) Outputs predicting collision time and position. II Related Work Event-based obstacle avoidance has gained significant attention in recent years due to the advantages offered by event cameras, such as low latency, high temporal resolution, and high dynamic range. These characteristics make event cameras particularly suitable for dynamic obstacle avoidance scenarios. II-A Event-Based Obstacle Avoidance II-A 1 Optical-Flow Approaches Early work on event-based obstacle avoidance focused on optical flow. Clady et al. [ 9 ] proposed an asynchronous time-to-contact algorithm for collision detection based on event-based motion flows. Milde et al. [ 23 ] developed an event-driven collision avoidance algorithm based on optical flow. Yasin et al. [ 35 ] explored the detection and avoidance of obstacles in low light conditions by triangulating 2D features extracted with LC-Harris. Falanga et al. [ 12 ] demonstrated a dynamic obstacle avoidance system for quadrotors using event cameras. More recently, Rodriguez-Gomez et al. [ 28 ] demonstrated a dynamic sense-and-avoid system for ornithopter robot flight using optical flow and clustering. II-A 2 Data-Driven Approaches Deep learning approaches have also been applied to event-based obstacle avoidance. Sanket et al. [ 29 ] and Bhattacharya et al. [ 4 ] developed a static-obstacle avoidance method for quadrotors using a monocular event camera.. Bisulco et al. [ 6 ] proposed a method to understand fast motion using spatio-temporal neural networks of a moving toy dart. Paredes-Valles et al. [ 25 ] trained a spiking neural network that accepts raw event-based camera data and outputs low-level control actions to perform autonomous vision-based flight. Differently from these approaches, Ours is multi-modal, lightweight and evaluated on an open-source dataset, namely the Ball-Collision Dataset, which can serve as a benchmark for further data-driven algorithmic exploration. II-B Event-Based Object Catching Related to obstacle avoidance, Forrai et al. [ 13 ] showcased agile object catching based on events with a quadrupedal robot. Monforte et al. [ 24 ] proposed the use of event cameras for trajectory prediction with a robotic arm, leveraging a stateful LSTM network trained in simulation. Notably, the event-based method achieved higher accuracy compared to the respective traditional frame-based version. Our work builds upon this evaluation framework while also integrating event-based perception with a more sophisticated multi-modal fusion approach and leveraging lightweight stateless networks. II-C Multi-Modal Fusion Approaches with Event Cameras Walters et al. [ 32 ] and He et al. [ 19 ] developed a system for the detection and tracking of dynamic objects using both event and depth sensing. Devulapally et al. [ 11 ] proposed a unified transformer-based architecture that fuses event and RGB data to improve monocular depth estimation accuracy. Zhou et al. [ 38 ] introduced RENet, an RGB-event fusion network that enhances moving object detection in autonomous driving using temporal and bi-directional fusion modules. Creß et al. [ 10 ] developed the TUMTraf Event dataset with synchronized event and RGB data, offering various fusion strategies to improve detection in transportation systems. Fradi and Papadakis [ 14 ] presented an event-RGB fusion framework that probabilistically combines detections from both modalities to enhance object detection in autonomous vehicles. These studies highlight the potential of event-RGB fusion in enhancing perception tasks. Our methodology adopts similar strategies, fusing event and RGB data

Read the complete text