ROAD-Waymo: Action Awareness at Scale for Autonomous Driving
Salman Khan, Izzeddin Teeti, Reza Javanmard Alitappeh, Mihaela C. Stoian, Eleonora Giunchiglia, Gurkirt Singh, Andrew Bradley, Fabio Cuzzolin · 2024
Read ROAD-Waymo: Action Awareness at Scale for Autonomous Driving by Salman Khan, Izzeddin Teeti, Reza Javanmard Alitappeh, Mihaela C. Stoian, Eleonora Giunchiglia, Gurkirt Singh, Andrew Bradley, Fabio Cuzzolin (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01683.txt.
Description
Abstract Autonomous Vehicle (AV) perception systems require more than simply seeing, via e.g., object detection or scene segmentation. They need a holistic understanding of what is happening within the scene for safe interaction with other road users. Few datasets exist for the purpose of developing and training algorithms to comprehend the actions of other road users.
Catalog record
| CREATOR | Salman Khan, Izzeddin Teeti, Reza Javanmard Alitappeh, Mihaela C. Stoian, Eleonora Giunchiglia, Gurkirt Singh, Andrew Bradley, Fabio Cuzzolin |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01683
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ROAD-Waymo: Action Awareness at Scale for Autonomous Driving Salman Khan, Izzeddin Teeti, Reza Javanmard Alitappeh, Mihaela C. Stoian, Eleonora Giunchiglia, Gurkirt Singh, Andrew Bradley, Fabio Cuzzolin Abstract Autonomous Vehicle (AV) perception systems require more than simply seeing, via e.g., object detection or scene segmentation. They need a holistic understanding of what is happening within the scene for safe interaction with other road users. Few datasets exist for the purpose of developing and training algorithms to comprehend the actions of other road users. This paper presents ROAD-Waymo, an extensive dataset for the development and benchmarking of techniques for agent, action, location and event detection in road scenes, provided as a layer upon the (US) Waymo Open dataset. Considerably larger and more challenging than any existing dataset (and encompassing multiple cities), it comes with 198k annotated video frames, 54k agent tubes, 3.9M bounding boxes and a total of 12.4M labels. The integrity of the dataset has been confirmed and enhanced via a novel annotation pipeline designed for automatically identifying violations of requirements specifically designed for this dataset. As ROAD-Waymo is compatible with the original (UK) ROAD [ 57 ] dataset, it provides the opportunity to tackle domain adaptation between real-world road scenarios in different countries within a novel benchmark: ROAD++. Dataset and code are available at: Dataset 1 1 1 https://github.com/salmank255/Road-waymo-dataset , Code 2 2 2 https://github.com/salmank255/ROAD_Waymo_Baseline . 1 Introduction Several large-scale, comprehensive AV training datasets have recently been released [ 9 , 20 , 69 , 56 , 65 , 45 ] , leveraging multiple sensors ( e.g . cameras, LiDAR, GPS, etc . ) for the purposes of object detection, semantic segmentation, object tracking and trajectory forecasting. However, a safe interaction with other road users requires much more than simply seeing things: rather, it requires understanding actions and events being conducted or happening in the scene. To this end, Singh et al . [ 57 ] introduced the ROAD dataset, which provides multi-label annotations of agents and actions for event awareness in autonomous driving. ROAD represents a starting point along the path to enable AVs to understand events. It features some degree of diversity in terms of weather conditions (including bright sunshine, wet roads, rain, snow and fog) and can be used as a benchmark for object/agent detection, action detection and event detection in a natural multi-label setting. This was indeed the case for the ICCV 2021 ROAD workshop and challenge 3 3 3 https://sites.google.com/view/roadchallangeiccv2021/challenge , which saw a combined 200+ entries for the aforementioned tasks. Since then, the problem of situation awareness for AVs has gained even more traction [ 22 , 70 ] . Figure 1 : Exemplary annotations of road users in ROAD-Waymo style, explained from the (ego) viewpoint of an AV. Each road user is annotated with three distinct labels: agent, action, and location. Only agents with a distinct status are labeled to avoid confusion from overlapped labels. Nevertheless, ROAD remains a small-scale dataset, lacking in both scale and true diversity to allow the further demonstration and testing of the benefits of event awareness. It is limited to scenes from just one, relatively small city - Oxford, UK (population 160,000) - with a high density of narrow streets, relatively low-speed driving on few routes. This makes it suitable for early stage development, but is significantly less challenging than real-world driving in complex environments. This paper introduces ROAD-Waymo, which tackles the diversity and scale issues of ROAD [ 57 ] , building on the Waymo-Open dataset [ 61 ] . Spatiotemporal [ 57 ] annotations, in the form of agent tracks plus action and location labels, are densely provided in the scene to study event awareness, action and event detection. As such annotations are expensive - we inherited 52K tracks from the Waymo dataset, adding extensive labelling based upon the road event = \{ agent, action, location \} protocol (illustrated in Figure 1 ), resulting in 4.1M bounding boxes with 4.1M agent, 4.3M action, and 4.3M location label instances. In combination with the 7K tracks present in ROAD, ROAD-Waymo provides an extensive playground for the development of advanced perception systems for autonomous vehicles and the tackling of novel problems, with a focus on domain adaption (in particular, between cities in the UK and the US) for action/agent/event detection tasks. Contributions: The contributions of this work are: 1. ROAD-Waymo A large-scale multi-label dataset for action and event detection in autonomous vehicles. Provided as an extension to the Waymo AV dataset [ 61 ] , it is considerably more comprehensive and challenging than any existing such benchmark, including ROAD. 2. Scale: ROAD-Waymo contains 4.3M action labels, making it 7 times larger than the original ROAD dataset, and the largest known dataset for this purpose. At the same time, it extends the Waymo dataset by providing a total of 12.7M additional labels. 3. Multi-City: The dataset encompasses 4 cities from several US states, providing a plethora of different driving scenarios in a range of operating domains. 4. Complexity: The labelling provided is highly comprehensive - actors are labelled at great distances from the ego vehicle, with many active participants in each scene. Several complex scenes contain the presence of emergency vehicles (11 times more than the ROAD), six weather conditions (two times of ROAD), many lanes, busy roads, pedestrians and so on. Thus, ROAD-Waymo provides a considerably more challenging perception testbed than any other dataset of this kind. 5. Verified Annotations: Importantly, ROAD-Waymo is the first AV dataset guaranteed to be compliant with a set of commonsense constraints. Each annotation was automatically verified to be compliant with 251 domain-dependent logical requirements elicited for this task. 6. ROAD++ domain adaptation framework: The new ROAD-Waymo dataset is fully compatible with the original ROAD data. Together, they form the basis for a benchmarking framework for domain adaptation in autonomous driving (between two countries UK and the US and their 5 different cities), termed ROAD++ . Tasks and Baselines: Experiments were conducted to provide a benchmark for the following perception tasks: 1. Object/Agent, Action, and Event detection tasks using various types of models. 2. Cross-country (datasets) and cross-city real-to-real unsupervised domain adaptation for same tasks. 3. Neuro-symbolic prediction enhancement ( i.e . the use of domain knowledge to guide the training of the neural models to ensure compliance with commonsense rules). 2 Related Work Autonomous Driving Datasets . Autonomous vehicle perception has recently made significant progress, thanks to modern benchmarks that offer more diverse environments and agent categories. For instance, previous datasets like NGSIM-180 [ 8 ] and highD [ 37 ] captured cars on highways using drones and surveillance cameras, while PIE [ 50 ] and JAAD [ 36 ] focused solely on pedestrian annotations using a single ego-vehicle camera. KITTI [ 20 ] was among the first to provide multimodal inputs, including camera frames and LiDAR point clouds, and annotations for both cars and pedestrians. Further down the line, Lyft [ 26 ] , Waymo [ 13 ] , nuScenes [ 4 ] , Argoverse [ 65 ] , Argoverse 2 [ 66 ] , and Zenseact [ 1 ] provide large-scale, comprehensive annotation, HD-maps, odometer information and additional labels for other agents and environmental factors like weather and lighting conditions. A