RefAV: Towards Planning-Centric Scenario Mining
Cainan Davidson, Deva Ramanan, Neehar Peri · 2025
Read RefAV: Towards Planning-Centric Scenario Mining by Cainan Davidson, Deva Ramanan, Neehar Peri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20981.txt.
Introduction
Cainan Davidson, Deva Ramanan, and Neehar Peri tackle a data problem for self-driving cars: finding the rare, instructive moments buried in fleet recordings, in this 2025 paper. Test vehicles log terabytes of video and sensor data. Engineers need the safety-critical scenes, a child darting out, a truck cutting in, but hunting them with hand-written search rules is slow and error-prone.
The team recast the hunt as a language task for vision-language models: given a written description of a scene, find whether and where it occurs in a log. They built RefAV, a dataset of 10,000 plain-language queries describing multi-agent interactions, drawn from one thousand real driving logs in a public dataset. Testing several tracking systems, they found that general-purpose vision-language models, used naively, perform poorly at this job. Results from a community competition round out the paper's lessons for the field.
Catalog record
| CREATOR | Cainan Davidson, Deva Ramanan, Neehar Peri |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 103 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20981
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RefAV: Towards Planning-Centric Scenario Mining Cainan Davidson, Deva Ramanan, Neehar Peri Abstract Autonomous Vehicles (AVs) collect and pseudo-label terabytes of multi-modal data localized to HD maps during normal fleet testing. However, identifying interesting and safety-critical scenarios from uncurated driving logs remains a significant challenge. Traditional scenario mining techniques are error-prone and prohibitively time-consuming, often relying on hand-crafted structured queries. In this work, we revisit spatio-temporal scenario mining through the lens of recent vision-language models (VLMs) to detect whether a described scenario occurs in a driving log and, if so, precisely localize it in both time and space. To address this problem, we introduce RefAV, a large-scale dataset of 10,000 10,000 diverse natural language queries that describe complex multi-agent interactions relevant to motion planning derived from 1000 1000 driving logs in the Argoverse 2 Sensor dataset. We evaluate several referential multi-object trackers and present an empirical analysis of our baselines. Notably, we find that naively repurposing off-the-shelf VLMs yields poor performance, suggesting that scenario mining presents unique challenges. Lastly, we discuss our recently held competition and share insights from the community. Our code and dataset are available on Github and Argoverse . Figure 1 : Scenario Mining Problem Setup. Given a natural language prompt such as vehicle making left turn through ego-vehicle’s path while it is raining , our problem setup requires models to determine whether the described scenario occurs within a 20-second driving log, and if so, precisely localize the referred object in 3D space and time from raw sensor data (LiDAR, 360 ∘ ring cameras, and HD maps). Based on the example above, a VLM should localize the start and end timestamps and 3D location of the red Mini Cooper executing a “Pittsburgh left” through the ego-vehicle’s path with a 3D track. Notably, the “Pittsburgh left” is a regional driving practice where a driver quickly makes a left turn before oncoming traffic proceeds. Although common in Pittsburgh, this maneuver is technically illegal. Therefore, we argue that scenario mining is critical for validating end-to-end autonomy in order to build a comprehensive safety case. Note that referred objects are shown in green, related objects in blue, and other objects in red. 1 Introduction Autonomous Vehicle (AV) deployment on public roads has increased significantly in recent years, with Waymo completing more than a million rides per week as of February 2025 [ 26 ] . Despite the maturity of such autonomous ride-hailing services, AVs can still sometimes require manual interventions [ 25 , 44 ] and can be prone to accidents [ 62 ] . Although data-driven simulators are an integral component in establishing a safety case [ 21 , 18 , 66 , 75 ] , validating end-to-end autonomy with real-world operational data remains a critical part of the testing stack. However, identifying interesting and safety-critical scenarios from uncurated real-world driving logs is akin to finding a "needle in a haystack" due to the scale of data collected during normal fleet operations. In this paper, we revisit the task of spatio-temporal scenario mining with recent vision-language models (VLMs) to identify interesting multi-agent interactions using natural language (cf. Fig. 1 ). Status Quo. Although language-based 3D scene understanding has been extensively studied in the context of referential multi-object tracking (RMOT) [ 71 , 72 , 29 ] , multi-modal visual question answering (VQA) [ 51 , 20 , 63 ] , and VLM-based motion planning [ 58 , 41 , 56 , 68 ] , we argue that spatiotemporal scenario mining presents unique challenges. RMOT extends referential grounding by associating referred objects over time. However, unlike RMOT, scenario mining does not guarantee that referred objects exist in a given driving log. Next, multi-modal VQA extends VQA with additional visual modalities like LiDAR. Although spatio-temporal scenario mining also leverages multi-modal input (e.g. RGB, LiDAR and HD maps), we require that methods output 3D tracks instead of text-based answers. Lastly, VLM-based motion planners directly estimate future ego-vehicle waypoints based on high-level language instructions and past sensor measurements. In contrast, scenario mining methods can reason over the full driving log to identify interactions between non-ego vehicles. Concretely, spatio-temporal scenario mining requires identifying if a described scenario occurs in a driving log from raw sensor measurements (e.g. RGB and LiDAR), and if so, precisely localizing all referred objects in both time and space with 3D tracks. To support this task, we propose RefAV, a large-scale dataset of 10,000 diverse natural language queries designed to evaluate a model’s ability to find a visual “needle in a haystack”. RefAV Scenario Mining Dataset. We repurpose the Argoverse 2 (AV2) Sensor dataset [ 70 ] , which contains 1,000 driving logs with synchronized LiDAR, 360 ∘ ring cameras, HD maps, and 3D track annotations for 30 categories. We curate a set of 10,000 natural language prompts that describe interesting (and often rare) scenarios (cf. Fig. 2 and Fig. 3 ) using a combination of manual annotations and procedural generation using large language models (LLMs). Notably, AV2 annotates ground truth tracks at 10 Hz, which allows for fine-grained motion understanding. In contrast, prior referential multi-object tracking benchmarks (the established task most similar to spatio-temporal scenario mining) use nuScenes [ 4 ] , which annotates ground truth tracks at 2 Hz (cf. Table 1 ). Argoverse’s higher temporal resolution uniquely allows RefAV to evaluate dynamic multi-agent interactions (unlike prior work that primarily evaluates referential expressions based on static attributes like vehicle color or relative heading). Interestingly, we find that simply repurposing VLMs for scenario mining yields poor performance. Referential Tracking by Program Synthesis. Although prior referential trackers and VLMs achieve reasonable performance with simple referential prompts (e.g. find all cars ), we find that such methods struggle with compositional reasoning and motion understanding (e.g. find all cars accelerating while changing lanes ). To address this problem, we propose Referential Tracking by Program Synthesis (RefProg), a modular approach that combines off-the-shelf 3D tracks with LLMs. Inspired by recent work in program synthesis [ 60 , 19 , 43 ] , our method uses an LLM to break down complex referential expressions into simpler compositional actions. Specifically, we define an API to describe hand-crafted atomic functions (e.g., turning , accelerating , changing lanes ) and use an LLM to synthesize a program that composes these atomic actions corresponding to the natural language prompt. We then execute the generated program to filter off-the-shelf tracks to identify the subset of tracks that best match the described scenario (cf. Fig. 4 ). Contributions. We present three major contributions. First, we introduce RefAV, a large-scale dataset designed to evaluate VLMs on 3D scene understanding and spatio-temporal localization. Our extensive experiments highlight the limitations of current methods, and demonstrate the effectiveness of our proposed program synthesis-based approach. Lastly, we highlight the results of our recent CVPR 2025 challenge hosted in conjunction with the Workshop on Autonomous Driving . 2 Related Works Referential Grounding and Tracking are long-standing challenges in vision-language understanding. Early datasets like RefCOCO, RefCOCO+, and RefCOCOg [ 22 , 77 , 42 ] helped popularize the task, which was later adapted for autonomous driving [ 12 , 65 ] . Although prior work focused on single-frame visual grounding, ReferKITTI [ 71 ] formalizes the problem of 2D