DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance

Oskar Natan, Andi Dharmawan, Aufaclav Zatu Kusuma Frisky, Jazi Eko Istiyanto, Jun Miura · 2026

Read DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance by Oskar Natan, Andi Dharmawan, Aufaclav Zatu Kusuma Frisky, Jazi Eko Istiyanto, Jun Miura (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01277.txt.

Introduction

Natan and co-authors fuse two sensor types to dodge sudden crossers. Frame cameras suffer lag and blur in fast scenes. A person who darts out is just such a scene. DeepIPCv3 pairs LiDAR point clouds with event streams. Event sensors report change at microsecond speed. A cross-modal attention block links the two streams. It lets fast updates take priority at once.

Fused features map to waypoints and control commands. A hybrid policy blends rule-based tracking with neural calls. Live tests would put people at real risk. So tests run offline on a custom dataset. Data spans bright noon and dim evening scenes. The fusion gives the lowest path and control errors.

Catalog record

CREATOROskar Natan, Andi Dharmawan, Aufaclav Zatu Kusuma Frisky, Jazi Eko Istiyanto, Jun Miura
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01277

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance Oskar Natan, Andi Dharmawan, Aufaclav Zatu Kusuma Frisky, Jazi Eko Istiyanto, Jun Miura Abstract Current end-to-end autonomous driving systems predominantly rely on frame-based sensors, which suffer from inherent perception latency and motion blur during highly dynamic encounters, specifically sudden pedestrian crossings. To address this critical safety vulnerability, we propose DeepIPCv3, a novel multi-modal autonomous navigation framework that synergizes the dense 3D spatial geometry of LiDAR point clouds with the microsecond-level asynchronous event streams of a Dynamic Vision Sensor (DVS). We introduce a Transformer-inspired cross-modal attention mechanism to dynamically correlate these distinct modalities, allowing the network to instantaneously prioritize high-speed dynamic updates without sacrificing structural scene awareness. The fused latent representations are then mapped to safe local waypoints and executable control commands via a hybrid policy network that blends heuristic trajectory tracking with direct neural predictions. Due to the severe physical risks associated with live testing of these sudden-crossing scenarios, the framework is rigorously evaluated offline using a custom multi-modal dataset collected across both well-illuminated noon and challenging evening conditions. Extensive comparative and ablation studies demonstrate that DeepIPCv3 achieves state-of-the-art predictive performance. By effectively eliminating exposure failures and motion blur, the proposed LiDAR and DVS fusion yields the lowest trajectory and control command errors, enabling highly reactive, mathematically bounded evasive maneuvers regardless of ambient illumination. To support future research, we will release the codes to our GitHub repo at https://github.com/oskarnatan/DeepIPCv3 . I Introduction End-to-end autonomous driving has emerged as a compelling paradigm for outdoor point-to-point navigation, offering a streamlined alternative to traditional, heavily modularized pipelines [ 42 ] [ 31 ] . By directly mapping raw multi-modal sensor inputs to navigational waypoints and control actions, these architectures reduce compounding errors and simplify system deployment [ 15 ] [ 25 ] . Recent advancements leveraging high-fidelity sensors, such as RGB-D cameras and 3D LiDAR, have significantly improved the robustness of environmental perception, semantic scene understanding, and trajectory prediction under standard operating conditions [ 41 ] [ 43 ] . However, ensuring consistent and safe navigation in unstructured, dynamic outdoor environments remains a formidable challenge, particularly when the vehicle must balance goal-directed routing with instantaneous obstacle avoidance [ 3 ] [ 11 ] . Fig. 1 : DeepIPCv3 observes the environment with LiDAR and DVS. Then, predict a set of navigational outputs. While current approaches successfully fuse spatial geometry with temporal modeling to navigate static and predictable environments, they frequently struggle under highly dynamic conditions [ 34 ] [ 7 ] . A critical safety scenario that remains inadequately addressed is the sudden, unpredictable appearance of dynamic obstacles [ 45 ] [ 13 ] . In this study, we specifically focus on sudden dynamic obstacle encounters, specifically a pedestrian rapidly crossing the vehicle’s path. Conventional frame-based sensors suffer from inherent latency and motion blur during fast movements, which bottlenecks the system’s reaction time [ 33 ] [ 32 ] [ 18 ] . Although event-based vision like the Dynamic Vision Sensor (DVS) offers microsecond-level temporal resolution by asynchronously capturing localized pixel-intensity changes [ 35 ] [ 2 ] , effectively integrating these sparse, asynchronous event streams with dense 3D point clouds remains an open research problem. To bridge this gap, we propose DeepIPCv3 as shown in Fig. 1 , an upgrade of our prior works [ 26 ] [ 27 ] , a novel end-to-end autonomous driving framework that integrates LiDAR-based spatial perception with DVS event streams to achieve robust point-to-point navigation and ultra-low-latency responses to execute safe avoidance maneuvers. Recognizing the distinct data modalities of 3D point clouds and event-based intensity changes, we introduce a sophisticated data fusion mechanism driven by Transformer-inspired attention blocks to dynamically learn cross-modal relationships. Rather than deploying the system in a high-risk online environment for these sudden-crossing scenarios, the framework is rigorously evaluated offline. DeepIPCv3 focuses strictly on learning the accurate policy mapping from the raw, fused sensor data to precise control commands. The main contributions of this work are summarized as follows: • We propose a multi-modal architecture that synergizes LiDAR and DVS inputs, enabling the system to maintain stable point-to-point navigation while improving responsiveness to sudden dynamic obstacles. • We introduce a Transformer-inspired cross-modal fusion module, which can extracts and correlates features between dense 3D spatial geometry and asynchronous event streams to generate optimal driving policies. • We conduct extensive offline evaluations using a custom multi-modal dataset featuring sudden pedestrian crossings under both well-illuminated and illumination-degraded conditions. The results demonstrate that DeepIPCv3 achieves state-of-the-art predictive accuracy, effectively decoupling dynamic responsiveness from ambient lighting to execute contextually safe avoidance behaviors, such as evasive steering or full-stop braking. II Related Works In this section, we do literature review on some related works that focus on end-to-end autonomous driving in sudden dynamic encounters and event-based autonomous navigation. Then, we point out some remain issues and also highlight key ideas for comparative study. II-A Autonomous Driving in Sudden Dynamic Encounters The shift toward end-to-end autonomous driving has streamlined the navigation pipeline, yet maintaining robustness under highly dynamic conditions remains a challenge [ 20 ] [ 16 ] . For instance, research into dynamic deception demonstrates that end-to-end models heavily reliant on frame-based perception can be easily compromised by pedestrians suddenly crossing the vehicle’s path. Recent studies leveraging high-fidelity simulations, such as CARLA [ 14 ] , have increasingly focused on the vulnerability of these systems to sudden dynamic obstacles [ 22 ] [ 29 ] [ 6 ] . However, standard vision sensors operate at fixed frame rates suffer from inherent latency and motion blur during rapid, unpredicted environmental changes [ 40 ] . To mitigate this, several recent frameworks have attempted to append physics-informed safety controllers [ 19 ] or model predictive control (MPC) modules [ 10 ] to the end-to-end pipeline. While these hybrid approaches can enforce hard safety constraints to prevent collisions, they often do so at the cost of the system’s fluidity, reacting aggressively rather than proactively due to the delayed perception of the fast-moving obstacle. Despite these algorithmic advancements, the fundamental bottleneck remains the sensory latency during rapid dynamic events, which algorithmic safety patches cannot entirely eliminate. Rather than relying solely on post-perception control overrides, DeepIPCv3 addresses this vulnerability at the sensory root by integrating an asynchronous event camera, so that it can fundamentally reduces perception latency, natively capturing the high-speed motion of sudden pedestrian crossings before frame-based sensors can register the blur. II-B Event-Based Autonomous Navigation To overcome the limitations of traditional frame-based cameras, event-based vision has gained significant traction in the autonomous driving domain [ 5 ] [ 23 ] . Dynamic Vision Sensors (DVS) operate differently from standard cameras by asynchronously recording only localized changes in pixel intensity, thereby providing microsecond-level temporal resolution and an exceptionally high dynamic range [ 1 ] [ 36 ] . Pioneering datasets such as DDD20 [ 17 ] and EvTTC [ 37 ] have catalyzed the development of deep learning models, including brain-inspired liquid neural networks and specialized convolutional recurrent networks, that directly map event streams to various driving tasks. Recent architectures have also demonstrated the efficacy of event cameras in overcoming motion blur and varying illumination during continuous driving tasks [ 12 ] [

Read the complete text