Toward Deep Representation Learning for Event-Enhanced Visual Autonomous Perception: the eAP Dataset
Jinghang Li, Shichao Li, Qing Lian, Peiliang Li, Xiaozhi Chen, Yi Zhou · 2026
Read Toward Deep Representation Learning for Event-Enhanced Visual Autonomous Perception: the eAP Dataset by Jinghang Li, Shichao Li, Qing Lian, Peiliang Li, Xiaozhi Chen, Yi Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16303.txt.
Introduction
Li and co-authors release eAP, a large dataset for event-based driving perception. Normal cameras fail in harsh light, but event cameras cope better. No large event dataset existed for self-driving scenes. eAP is the largest such set so far. With it, the team improved a known 3D vehicle detector under hard lighting. They also built the best event-based time-to-contact estimator, running at 200 frames per second. Data, code, and trained models will be public.
Catalog record
| CREATOR | Jinghang Li, Shichao Li, Qing Lian, Peiliang Li, Xiaozhi Chen, Yi Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80376 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16303
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Toward Deep Representation Learning for Event-Enhanced Visual Autonomous Perception: the eAP Dataset Jinghang Li, Shichao Li, Qing Lian, Peiliang Li, Xiaozhi Chen, Yi Zhou Abstract Recent visual autonomous perception systems achieve remarkable performances with deep representation learning. However, they fail in scenarios with challenging illumination. While event cameras can mitigate this problem, there is a lack of a large-scale dataset to develop event-enhanced deep visual perception models in autonomous driving scenes. To address the gap, we present the eAP ( e vent-enhanced A utonomous P erception) dataset, the largest dataset with event cameras for autonomous perception. We demonstrate how eAP can facilitate the study of different autonomous perception tasks, including 3D vehicle detection and object time-to-contact (TTC) estimation, through deep representation learning. Based on eAP , we demonstrate the first successful use of events to improve a popular 3D vehicle detection network in challenging illumination scenarios. eAP also enables a devoted study of the representation learning problem of object TTC estimation. We show how a geometry-aware representation learning framework leads to the best event-based object TTC estimation network that operates at 200 FPS. The dataset, code, and pre-trained models will be made publicly available for future research. Multimedia Material Video: https://youtu.be/6nuFrPViD3U Project Page: https://nail-hnu.github.io/eAP_dataset/ I Introduction Detection and motion estimation of vehicles in 3D space has indispensable significance for autonomous driving systems to navigate in complex environments. Numerous studies address this problem using LiDAR sensors [ 64 ] , yet they incur high costs and suffer from the sparsity of the point cloud. Visual 3D object perception [ 43 ] using RGB cameras, on the other hand, can fully unleash the potential of low-cost autonomous systems. The performance of frame-based Standard-Dynamic-Range (SDR) cameras, a cornerstone of conventional visual perception systems, is fundamentally limited in challenging conditions. Their reliance on a global exposure time causes motion blur and over/under-exposure in scenes with fast-moving objects or extreme lighting. Neuromorphic event cameras overcome these limitations by asynchronously measuring per-pixel brightness changes [ 25 , 28 , 8 ] , providing high temporal resolution and High-Dynamic-Range (HDR). This work explores how these distinctive properties can be leveraged to enhance the robustness and safety of autonomous perception in diverse outdoor driving scenarios. Specifically, we study two important sub-tasks for event-enhanced autonomous perception: 3D vehicle detection and object time-to-contact (TTC) estimation. The former task aims to discover and locate vehicles in 3D space, and the latter performs object motion estimation to prevent collisions. This selection of tasks has three reasons: • Importance of tasks: These two tasks span from object detection [ 71 ] to motion estimation [ 2 ] , which are both fundamental problems in robotic perception. • Scope of representations: Recent autonomous perception approaches are dominated by two types of representations: the bird’s eye view (BEV) representation [ 44 ] and the classical front-view (FV) representation [ 69 , 41 , 49 ] . The former builds dense voxels in the 3D space for high accuracy, while the latter utilizes image-level 2D representations for fast inference. Our choice of problems covers these two mainstream approaches. • Value of applications: Studying these two tasks enables an autonomous agent to identify vehicles and mitigate potential collisions. FV TTC estimation systems are widely adopted in modern Automated Emergency Braking (AEB) systems, which include the Forward Collision Warning (FCW) functionality [ 23 , 12 ] for fast response to potential collisions in life-critical situations. Unfortunately, we encounter a data dilemma when pursuing performance beyond the state of the art. Datasets Year Duration (in hours) #3D Bbox TTC Anno #Class #Resolution (RGB) #Resolution (Event) Aim Data source Scenarios MVSEC [ 70 ] 2018 0.5 - - - 752\times 480 346\times 260 Eval Collected Urban DDD20 [ 33 ] 2020 51 - - - 346\times 260 346\times 260 Train&Eval Collected Expressways, Highways, Urban DSEC-Det [ 28 ] 2021 1 - - 6 1440\times 1080 640\times 480 Train&Eval Collected Urban, Suburban, Rural areas VECtor [ 27 ] 2022 0.36 - - - 1224\times 1024 640\times 480 Eval Collected Indoor, School M3ED [ 9 ] 2023 3.42 (car 2) 1 - - 11 1280\times 800 1280\times 720 Train&Eval Collected Urban, Forest DSEC-3DOD [ 11 ] 2 2025 0.15 ✓ - 2 1440\times 1080 640\times 480 Eval Collected Urban, Suburban, Highways Crash to Not Crash [ 35 ] 2019 - - - 1 710\times 400 - Eval Internet - DADA-2000 [ 22 ] 2019 - - - 7 1584\times 660 - Train&Eval Internet - TSTTC [ 57 ] 2023 27.8 - ✓ 1 1024\times 576 - Train&Eval Collected Urban, Highway EvTTC [ 61 ] 2024 0.2 - ✓ 2 1920\times 1200 1280\times 720 Eval Collected Test track, Clear Road FCWD [ 37 ] 2024 0.05 - ✓ 1 1920\times 1200 1280\times 720 Eval Collected Clear Road EV-TTC [ 4 ] 3 2025 3.42 (car 2) 1 - ✓ 11 1280\times 800 1280\times 720 Train&Eval Collected Urban, Forest eAP (Ours) 2025 4.8 532K+ ✓ 8 1920\times 1200 1280\times 720 Train&Eval Collected Highways, Urban, Low-light 1 The M3ED dataset [ 9 ] includes 1.9 hours of driving data (approximated to 2 hours) out of a 3.42-hour total. 2 The DSEC-3DOD [ 11 ] is built upon the DSEC [ 28 ] , providing novel 3D bounding box annotations for a portion of its “zurich city” sequence. 3 The EV-TTC dataset [ 4 ] is built upon the M3ED dataset [ 9 ] , which is the first event dataset offering dense TTC ground truth. TABLE I: A comprehensive overview of recent collision prediction and TTC estimation datasets. eAP features a large dataset scale, diverse scenes, high resolution, a narrow-baseline setup for cross-modal tasks, along with object-level 3D bounding box and TTC annotations. The potential of event cameras for visual 3D vehicle detection remains largely unexplored. Recent approaches adopt an end-to-end BEV paradigm and have achieved remarkable performance [ 44 ] . However, they require a large amount of training data to generalize in complex environments. To facilitate studying the problem of event-enhanced visual 3D vehicle detection, a large-scale dataset with event cameras and 3D box labels is highly desirable. Unfortunately, to our best knowledge, no such dataset exists. A similar dilemma exists for the event-enhanced visual TTC estimation problem. TTC is defined as the time remaining until the object’s depth reaches zero, assuming the current relative motion between the camera and the object remains unchanged [ 32 ] . Previous event-based TTC estimation methods employ hand-crafted features to establish data association across events. These methods [ 26 , 15 , 17 , 58 , 52 , 37 ] typically formulate the TTC estimation problem as a geometric-model-fitting task. However, these methods often employ overly simplified parametric models and are evaluated in environments with limited variation in object types, lighting conditions, and TTC ground truths. As a result, they may not adequately capture the complexity and diversity of real-world autonomous driving scenarios. More importantly, these model-fitting-based approaches cannot learn data-driven representations, which prevents them from leveraging the growing volume of event data to improve performance and generalization capabilities. Large-scale event-camera datasets for autonomous perception that offer sufficient scene diversity and object-level TTC labels are currently scarce, hindering the development of deep learning-based approaches that typically