WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
Seungjun Yu, Seonho Lee, Namho Kim, Jaeyo Shin, Junsung Park, Wonjeong Ryu, Raehyuk Jung, Hyunjung Shim · 2025
Read WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving by Seungjun Yu, Seonho Lee, Namho Kim, Jaeyo Shin, Junsung Park, Wonjeong Ryu, Raehyuk Jung, Hyunjung Shim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20022.txt.
Introduction
Seungjun Yu and seven co-authors introduce WaymoQA, a dataset for safety-critical reasoning in self-driving. Multimodal models read driving scenes well, but hard cases remain: avoiding one traffic risk can create another. A single front view often cannot support such judgment, so the dataset uses several views at once.
The authors split the target skill into two stages: resolve the immediate risk first, then reduce the new risks that the decision itself creates. WaymoQA holds tens of thousands of human-annotated question-answer pairs on complex, high-risk scenes, in multiple-choice and open forms, for images and video. Current models lag on these scenes compared with normal ones, but fine-tuning on WaymoQA lifts their reasoning clearly.
Catalog record
| CREATOR | Seungjun Yu, Seonho Lee, Namho Kim, Jaeyo Shin, Junsung Park, Wonjeong Ryu, Raehyuk Jung, Hyunjung Shim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20022
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving Seungjun Yu, Seonho Lee, Namho Kim, Jaeyo Shin, Junsung Park, Wonjeong Ryu, Raehyuk Jung, Hyunjung Shim Abstract Recent advancements in multimodal large language models (MLLMs) have shown strong understanding of driving scenes, drawing interest in their application to autonomous driving. However, high-level reasoning in safety-critical scenarios, where avoiding one traffic risk can create another, remains a major challenge. Such reasoning is often infeasible with only a single front view and requires a comprehensive view of the environment, which we achieve through multi-view inputs. We define Safety-Critical Reasoning as a new task that leverages multi-view inputs to address this challenge. Then, we distill Safety-Critical Reasoning into two stages: first resolve the immediate risk, then mitigate the decision-induced downstream risks. To support this, we introduce WaymoQA, a dataset of 35,000 human-annotated question–answer pairs covering complex, high-risk driving scenarios. The dataset includes multiple-choice and open-ended formats across both image and video modalities. Experiments reveal that existing MLLMs underperform in safety-critical scenarios compared to normal scenes, but fine-tuning with WaymoQA significantly improves their reasoning ability, highlighting the effectiveness of our dataset in developing safer and more reasoning-capable driving agents. Figure 1 : WaymoQA Overview. Multi-view scene and two-stage reasoning (Perception \rightarrow Prediction \rightarrow Planning) under compounding risks. The first plan detours around a parked motorcycle; the second plan returns to avoid an oncoming car. Counterfactual questions ask about alternative actions, and Safety-Critical Relationship questions capture spatial relations among agents in safety-critical scenes. 1 Introduction Recent progress in Multimodal Large Language Models (MLLMs) have demonstrated impressive performance across a wide spectrum of vision-language tasks, including image captioning, visual reasoning, and visual question answering (VQA) 1 ; 38 ; 62 ; 71 . These capabilities have naturally extended into autonomous driving, where MLLMs are being explored for end-to-end driving pipelines, scenario generation, and vision-language action modeling 13 ; 15 ; 17 ; 21 ; 38 ; 46 . Among these efforts, Driving Question Answering (Driving QA) has emerged as a promising paradigm for equipping autonomous agents with high-level scene understanding and decision-making capabilities. In visual question answering, the design of the dataset is the primary factor that determines the measurable capacity of models. 3 ; 24 ; 41 . Early efforts in Driving QA datasets, such as DRAMA 39 and DriveLM 50 , adopted a single-view setting. However, the increasing complexity of real-world driving and the advent of multi-sensor perception systems have highlighted the importance of multi-view scene understanding. This led to the development of datasets like NuScenes-QA 44 and NuPlan-QA 43 , which integrate multi-camera inputs to better capture spatial relationships and inter-agent dynamics, reflecting the perceptual richness of modern autonomous systems. Despite this progress, existing datasets 14 ; 31 ; 35 still focus primarily on normal scenarios and overlook safety-critical events 63 . These events are rare but highly consequential. Examples include pedestrians suddenly emerging from occlusions, abrupt braking triggered by unforeseen obstacles, or conflicts between multiple agents at intersections. The true reliability of an autonomous driving system is not determined by its performance in normal scenarios but by its behavior in these unpredictable, dangerous moments, specifically safety-critical scenarios. Recent works such as NAVSAFE 49 have recently recognized this and emphasized the importance of evaluating models on safety-critical scenarios. Accordingly, recent benchmarks such as DVBench 63 and VRU-Accident 27 developed safety-critical driving QA benchmarks. However, these works 63 ; 27 have three key limitations. ( L1 ) They are designed exclusively for evaluation and do not include training data, preventing MLLMs from learning safety-critical reasoning directly from examples. A dedicated training dataset is essential for MLLMs to learn safety-critical reasoning. ( L2 ) Like the early Driving QA works, they rely on single-view inputs, which limits the understanding of complex safety-critical scenarios. A single-view system can miss blind spots or occluded agents. ( L3 ) Consequently, and most critically , they lack support for high-level reasoning across the full autonomy stack, from perception to behavior prediction and risk-aware motion planning. As shown in Fig. 2 , the same scene yields opposite decisions under different view coverage. With a single front view the model concludes that a lane change is both feasible and safer. With multi view the rear right vehicle becomes visible, the lane change becomes infeasible, and slowing down becomes safer. This failure arises because limited view coverage produces an incomplete feasibility set and an incorrect risk ranking, which manifests as missing action and consequence reasoning. Such high-level reasoning, which is essential in safety-critical scenes, is largely missing from existing approaches. Figure 2 : Comparison across Difference Views of Same Scene. To address these limitations, we introduce WaymoQA, the first training-enabled, safety-critical multi-view driving QA dataset. Built from long-tail scenarios in the Waymo End-to-End dataset 59 and filtered using U.S. NHTSA (National Highway Traffic Safety Administration) 59 safety criteria, WaymoQA consists of 35,000 human-written question-answer pairs, covering safety-critical videos and key frames that are manually selected. To address ( L1 ) the absence of a training set, we release WaymoQA with 28,585 training set and 6,415 test set. The dataset supports two formats, multiple-choice questions for test and open-ended questions for train. Both are included in the training set to enable in-distribution validation. To tackle ( L2 ) the information loss from single front frame view, WaymoQA resolves scenes using multi view inputs at each time step. We aggregate eight synchronized views (front left, front center, front right, side left, side right, rear left, rear center, rear right) so that occlusions are reduced and the spatial field of view is expanded. This multi-view configuration expands the field of view, admits more evidence, and yields more accurate scene understanding, thereby providing the evidential basis required for reliable high-level reasoning. In short, the multi-view design provides the essential foundation for a principled formulation of safety-critical reasoning in autonomous driving. To address ( L3 ) the lack of high-level reasoning, building on this foundation, we adopt two complementary strategies. First , we explicitly define a new task, Safety-Critical Reasoning , structured into two progressive stages, and design 70% of WaymoQA to require such high-level reasoning. Stage 1 focuses on resolving the immediate risk as the model detects the safety-critical object, predicts its trajectory, and selects the safest feasible action. Stage 2 reasons for secondary risks that may be triggered by Stage 1 action, reflecting real-world driving where mitigating one hazard can introduce another. As shown in Fig. 1 , the ego first passes a parked motorcycle on the front right by briefly crossing the center line and then promptly returns inside the line to avoid an oncoming vehicle. These compounded decisions exemplify the two-stage Safety-Critical Reasoning elicited by WaymoQA. Second , we leverage the complementary strengths of video and image to enrich reasoning signals as shown in Fig. 5 . Video QA captures how events unfold and verifies what actually occurred, assessing both the motion of the safety-critical object and the ego response. Image QA presents a single key frame from the same situation and invites decision-oriented and counterfactual queries that are not tied to the realized timeline. While departing from the single realized path in the video, Image QA broaden the space