Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models
Haohan Chi, Huan-ang Gao, Ziming Liu, Jianing Liu, Chenyu Liu, Jinwei Li, Kaisen Yang, Yangcheng Yu, Zeda Wang, Wenyi Li, Leichen Wang, Xingtao Hu, Hao Sun, Hang Zhao, Hao Zhao · 2025
Read Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models by Haohan Chi, Huan-ang Gao, Ziming Liu, Jianing Liu, Chenyu Liu, Jinwei Li, Kaisen Yang, Yangcheng Yu, Zeda Wang, Wenyi Li, Leichen Wang, Xingtao Hu, Hao Sun, Hang Zhao, Hao Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23757.txt.
Introduction
Haohan Chi and co-authors build an open dataset to teach driving models the odd cases. Their 2025 paper tackles vision-language-action models that falter in messy, unstructured scenes. Corner cases are rare in normal driving logs, so the team curated them on purpose.
Over 80,000 clips were distilled from two million across eight public sources. A four-way scheme sorts the hard scenes, and each clip carries planning questions and action paths. Models trained on the set scored better on closed-loop safety tests and cut collision rates. Open-loop path prediction neared the best known scores. The question suite also exposed where models still misread scenes. Weights and data are public.
Catalog record
| CREATOR | Haohan Chi, Huan-ang Gao, Ziming Liu, Jianing Liu, Chenyu Liu, Jinwei Li, Kaisen Yang, Yangcheng Yu, Zeda Wang, Wenyi Li, Leichen Wang, Xingtao Hu, Hao Sun, Hang Zhao, Hao Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23757
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models Haohan Chi, Huan-ang Gao, Ziming Liu, Jianing Liu, Chenyu Liu, Jinwei Li, Kaisen Yang, Yangcheng Yu, Zeda Wang, Wenyi Li, Leichen Wang, Xingtao Hu, Hao Sun, Hang Zhao, Hao Zhao Abstract Vision-Language-Action (VLA) models for autonomous driving show promise but falter in unstructured corner case scenarios, largely due to a scarcity of targeted benchmarks. To address this, we introduce Impromptu VLA. Our core contribution is the Impromptu VLA Dataset : over 80,000 meticulously curated video clips, distilled from over 2M source clips sourced from 8 open-source large-scale datasets. This dataset is built upon our novel taxonomy of four challenging unstructured categories and features rich, planning-oriented question-answering annotations and action trajectories. Crucially, experiments demonstrate that VLAs trained with our dataset achieve substantial performance gains on established benchmarks—improving closed-loop NeuroNCAP scores and collision rates, and reaching near state-of-the-art L2 accuracy in open-loop nuScenes trajectory prediction. Furthermore, our Q&A suite serves as an effective diagnostic, revealing clear VLM improvements in perception, prediction, and planning. Our code, data and models are available at https://github.com/ahydchh/Impromptu-VLA . Figure 1: Visual Abstract of Impromptu VLA. We construct Impromptu VLA Dataset, which contains over 80K clips curated from 8 open-sourced datasets, focusing on four critical types of unstructured "corner case" scenarios that challenge current autonomous driving vehicles. It supports interconnected VLA tasks including scene understanding, prediction, meta planning and trajectory planning. Key experimental results demonstrates that VLA models trained with Impromptu VLA Dataset achieve significant performance improvements in both closed-loop and open-loop metrics. 1 Introduction Autonomous driving has achieved remarkable progress, demonstrating increasing proficiency in navigating the well-structured environments of urban centers and highways where clear lane markings and predictable traffic flows are the norm [ 23 , 29 , 58 ] . However, the ultimate ambition of ubiquitous self-driving compels us to look beyond these well-trodden paths towards the intricate and often unpredictable domain of unstructured roads. These unstructured scenarios—encompassing everything from rural tracks and dynamic construction zones to areas with ambiguous signage or those recovering from natural events—represent the next significant frontier. It is here that current autonomous systems often face their sternest tests, and where breakthroughs are essential for realizing the full potential of go-anywhere autonomous capabilities [ 74 ] . Successfully navigating this frontier is profoundly hindered by a critical scarcity of specialized data. While numerous driving datasets have been foundational to current progress, they predominantly capture common, structured traffic situations [ 7 , 8 , 21 , 42 , 43 , 55 , 59 , 68 ] . This leaves a significant blind spot concerning the sheer diversity and unique challenges posed by unstructured settings, such as ill-defined road boundaries, the appearance of unconventional dynamic obstacles, adherence to makeshift traffic rules, or dealing with treacherous road surfaces. Without large-scale, meticulously annotated datasets that specifically reflect these complex conditions [ 70 , 47 ] , the ability to train robust AI drivers and rigorously evaluate their adaptability in such scenarios remains severely constrained. To address this data void, we introduce the Impromptu VLA Dataset , a new large-scale benchmark specifically curated to propel research in autonomous driving on unstructured roads, as introduced in Figure 1 . Distilled from an initial pool of over two million clips from eight diverse public sources [ 7 , 8 , 21 , 42 , 43 , 55 , 59 , 68 ] , Impromptu VLA comprises approximately \sim 80,000 meticulously selected and verified clips. These are categorized into four distinct types of challenging unstructured scenarios—roads with unclear boundaries, temporary traffic rule changes, unconventional dynamic obstacles, and challenging road conditions—and are enriched with extensive multi-task annotations and planning trajectories. The dataset was constructed using an advanced pipeline that leverages Vision-Language Models (VLMs) with Chain-of-Thought reasoning [ 39 , 2 , 12 ] for nuanced understanding, followed by comprehensive human verification to ensure high-quality, reliable labels. Our comprehensive experimental evaluations rigorously validate the efficacy of the Impromptu VLA Dataset. We demonstrate that VLMs fine-tuned on our dataset exhibit substantially improved performance on established autonomous driving benchmarks. For instance, in challenging closed-loop NeuroNCAP [ 41 ] simulations (Table 2 ), our 3B model enhanced with Impromptu VLA saw its average NeuroNCAP score increase significantly to 2.15 /5.00 from 1.77/5.00 achieved by the baseline, while its average collision rate was critically reduced from 72.5% down to 65.5% . In open-loop nuScenes [ 7 ] evaluations for trajectory prediction, pre-training with our dataset also markedly reduced L2 errors; our 3B model fine-tuned with Impromptu VLA achieved an average L2 error of 0.30m, bringing its performance nearly on par with leading specialized methods like EMMA+ [ 25 ] (0.29m), despite the latter often benefiting from substantially larger proprietary training datasets [ 88 , 26 ] . Furthermore, evaluations on our dataset’s own diverse Q&A validation suite reveal significant and quantifiable gains in specific VLM capabilities related to perception, prediction, and planning within these demanding unstructured contexts. Our primary contributions are summarized as follows: • The Impromptu VLA Dataset: A publicly available, large-scale, and richly annotated resource meticulously focused on diverse and challenging unstructured driving scenarios, designed to fill a critical gap in existing data resources. • A systematic taxonomy for unstructured road conditions and a scalable, VLM-centric data curation pipeline for their identification, categorization, and comprehensive annotation with multi-task Q&A suitable for training advanced VLMs. • Extensive experimental evidence demonstrating that training with the Impromptu VLA Dataset significantly boosts results on standard driving benchmarks, and serves as an effective diagnostic tool for assessing and improving VLM capabilities in unstructured environments. 2 Impromptu VLA Dataset: Learning to Drive on Unstructured Roads Figure 2: Characteristics comparison of different driving scene datasets. Figure (a) illustrates the distribution of scene categories across various datasets and the number of video clips contained in each dataset, providing a direct view of the emphasis on different scene types and the data scale of each dataset. Figure (b) compares the trajectory distribution in the original with the trajectory distribution in our constructed dataset, explaining the trajectory diversity of our dataset. Figure (c) shows examples of different scene categories from 8 source datasets. Notably, the IDD dataset lacks data for the "Challenging Road Conditions" category. Attribute Mapillary ONCE NAVSIM nuScenes Waymo Argoverse-V2 KITTI IDD Sum. Camera Views 1 7 8 6 5 7 1 1 - Resolutions Variable 1920 x 1020 1920 x 1080 1600 x 900 640 x 360 1550 x 2048 1242 x 375 1920 x 1080 - FPS 2Hz* 2Hz 2Hz 2Hz 10Hz 20Hz 10Hz 15Hz Raw Clip Count 1000k 800k 90k 40k 40k 320k 20k 7k 2000k Labeled Clip Count 22062 18093 18600 11370 7530 2490 930 930 80k Raw Data Size 60GB 2TB 450GB 1.5TB 5TB 1TB 180GB 18GB 10TB Labeled Data Size 1GB 5GB 12GB 10GB 7GB 7GB 1GB 0.5GB 43.5GB Table 1: Dataset Information. Summary of key attributes for the datasets used in this study. Note that the Mapillary dataset exhibits variable resolutions. For the Mapillary dataset, the frequency is assumed to be 2Hz as specific FPS information is not explicitly provided. 2.1 Overview The research community currently lacks sufficient large-scale, diverse, and meticulously annotated datasets specifically focused on unstructured scenarios. To address this critical