VLM-C4L: Continual Core Dataset Learning with Corner Case Optimization via Vision-Language Models for Autonomous Driving

Haibo Hu, JiaCheng Zuo, Lou Yang, Yufei Cui, Jianping Wang, Nan Guan, Jin Wang, Yung-Hui Li, Chun Jason Xue · 2025

Read VLM-C4L: Continual Core Dataset Learning with Corner Case Optimization via Vision-Language Models for Autonomous Driving by Haibo Hu, JiaCheng Zuo, Lou Yang, Yufei Cui, Jianping Wang, Nan Guan, Jin Wang, Yung-Hui Li, Chun Jason Xue (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23046.txt.

Introduction

Self-driving cars stumble on rare, odd scenes such as glare and fog. Haibo Hu and co-authors build a learning frame that keeps teaching the car those hard cases, in work from twenty twenty-five. Rare case data is scarce and mixed, which is why models miss them. The frame uses vision-language models to mine and polish hard case sets.

A core replay trick then lets the car learn new cases without forgetting old ones. Vision-language checks lift the quality of the hard case pool as it grows. The team claims the first such steady-learning frame aimed squarely at driving corner cases. Their aim is a car stack that stays robust as fresh odd scenes appear on real roads.

Catalog record

CREATORHaibo Hu, JiaCheng Zuo, Lou Yang, Yufei Cui, Jianping Wang, Nan Guan, Jin Wang, Yung-Hui Li, Chun Jason Xue
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23046

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VLM-C4L: Continual Core Dataset Learning with Corner Case Optimization via Vision-Language Models for Autonomous Driving Haibo Hu, JiaCheng Zuo, Lou Yang, Yufei Cui, Jianping Wang, Nan Guan, Jin Wang, Yung-Hui Li, Chun Jason Xue Abstract With the widespread adoption and deployment of autonomous driving, handling complex environments has become an unavoidable challenge. Due to the scarcity and diversity of extreme scenario datasets, current autonomous driving models struggle to effectively manage corner cases. This limitation poses a significant safety risk—according to the National Highway Traffic Safety Administration (NHTSA), autonomous vehicle systems have been involved in hundreds of reported crashes annually in the United States, occurred in corner cases like sun glare and fog, which caused a few fatal accident. Press (2024) . Furthermore, in order to consistently maintain a robust and reliable autonomous driving system, it is essential for models not only to perform well on routine scenarios but also to adapt to newly emerging scenarios—especially those corner cases that deviate from the norm. This requires a learning mechanism that incrementally integrates new knowledge without degrading previously acquired capabilities. However, to the best of our knowledge, no existing continual learning methods have been proposed to ensure consistent and scalable corner case learning in autonomous driving. To address these limitations, we propose VLM-C4L, a continual learning framework that introduces Vision-Language Models (VLMs) to dynamically optimize and enhance corner case datasets, and VLM-C4L combines VLM-guided high-quality data extraction with a core data replay strategy, enabling the model to incrementally learn from diverse corner cases while preserving performance on previously routine scenarios, thus ensuring long-term stability and adaptability in real-world autonomous driving. We evaluate VLM-C4L on large-scale real-world autonomous driving datasets, including Waymo and the corner case dataset CODA. To assess the effectiveness of our approach, we employ Sparse R-CNN, the strongest model in the CODA benchmark, and Cascade-DETR, a widely recognized model. Experimental results show that VLM-C4L greatly improves object detection in challenging scenarios like light pollution and fog, with AP and AR scores close to those in normal conditions. our code and datasets are here: www.vlmc4l.site/ 1 Introduction Deep neural network-based autonomous driving technologies have advanced rapidly, achieving significant milestones in real-world deployment, especially in perception tasks such as object detection Cai & Vasconcelos (2018a) ; Jiang et al. (2020) ; Zhu et al. (2020a) ; Sun et al. (2021b) . These detection models excel at detecting common traffic participants (e.g. cars, pedestrians, and bicycles), primarily due to the availability of extensive and well-curated regular datasets, such as Waymo Sun et al. (2020) , nuScenes Caesar et al. (2020) , and KITTI Geiger et al. (2013) . However, handling corner cases—rare, unusual, and challenging driving conditions—remains a critical bottleneck for achieving robust performance in complex and dynamic environments Zhao et al. (2024b) ; Chen et al. (2024b) .According to a report released by the U.S. National Highway Traffic Safety Administration (NHTSA), nearly 400 collisions involving Level 2 Advanced Driver Assistance Systems (ADAS) were reported by automakers such as Tesla between July 2023 and May 2024. A significant portion of these incidents resulted from the failure of autonomous driving systems to handle rare or unexpected driving conditions, with Tesla reporting multiple crashes caused by factors such as sun glare, fog, and airborne dust Press (2024) . These real-world failure cases have motivated increasing research attention toward evaluating model robustness under challenging scenarios. Table 1 presents the object detection metrics(AP/AR) for the CODA dataset Li et al. (2022) , where the models were trained on the Waymo dataset. ”WAYMO” represents their object detection performance on the Waymo dataset, while ”COMMON” refers to their detection results for common objects (vehicles, pedestrians, and bicycles) on the CODA dataset. ”NOVAL” indicates the model’s detection performance on corner cases within the CODA dataset, and ”CORNER” combines results from both COMMON and NOVAL. The results clearly demonstrate that models trained on regular datasets struggle to handle corner cases effectively, with most scores falling below 10%. Table 1 : The Average Precision(AP) and Average Recall(AR) of Models for Autonomous Driving in Waymo and CODA Datasets Method WAYMO CORNER COMMON NOVEL AP(%) AR(%) mAR(%) mAR(%) mAR(%) RetinaNet Ross & Dollár (2017) 39.7 47.7 8.4 24.5 6.7 Faster R-CNN Ren et al. (2016) 40.9 47.0 6.8 20.9 5.5 Cascade R-CNN Cai & Vasconcelos (2018b) 42.6 48.1 6.6 18.9 5.3 D-DETR Zhu et al. (2020b) 40.4 49.8 7.3 28.5 5.2 Sparse R-CNN Sun et al. (2021b) 38.8 49.8 10.1 29.5 7.6 Cascade Swin Liu et al. (2021) 44.2 49.0 5.4 21.8 4.3 To address corner cases, existing autonomous driving models often rely on specialized datasets like CODA and fine-tune on unclassified corner case data ce.oda_track1 ; Zhao et al. (2024a) . While this improves recognition to some extent, it fails to explicitly identify which cases are resolved, which is inadequate for safety-critical applications. Simulators (e.g., CornerSim Daoud et al. (2024) ) offer another option, but the Sim-to-Real gap remains a major challenge Udupa et al. (2024) ; Zuo et al. (2025) . Ultimately, real-world data is essential. However, corner case data is both scarce and highly diverse—for instance, CODA contains fewer than 10,000 scenes, but covers over 40 different situations, compared to 230,000 in Waymo Sun et al. (2020) or millions in ONCE Mao et al. (2021) . Enabling models to handle specific, diverse corner cases with limited data is therefore a key challenge. Therefore, enabling autonomous driving models to effectively handle specific corner case scenarios while ensuring adaptability to the diversity of corner cases, all within the constraints of limited data, remains an open challenge. Continual learning offers a promising direction to address this issue by allowing models to incrementally adapt to newly emerging corner cases without forgetting previously learned knowledge, thus enhancing long-term generalization across diverse scenarios Wang et al. (2024) ; Bang et al. (2021) . Moreover, the effectiveness of continual learning relies on the availability of well-defined and representative corner case data, which remains scarce and often ambiguous. Recent breakthroughs in vision-language models (VLMs) have demonstrated their exceptional capabilities in reasoning Chen et al. (2024a) , classification Zhong et al. (2024) ; Addepalli et al. (2024) , and feature extraction Jin et al. (2025) across various domains. While VLMs have been extensively used in these areas, their potential to address challenges in autonomous driving—particularly in handling corner cases—remains largely unexplored. The logical reasoning and robust memory capabilities of VLMs offer promising opportunities to identify, interpret, and organize corner case scenarios, mitigating the limitations of manually curated datasets and providing a richer foundation for continual learning in autonomous systems. Based on our findings, we propose VLM-C4L, a continual learning framework that optimizes corner case datasets using VLMs to update core datasets dynamically. VLM-C4L first leverages VLMs to efficiently extract, distribute, and enhance corner case data, forming targeted corner case datasets. It then introduces a tailored continual learning strategy that preserves the original model’s capabilities using core datasets while fine-tuning the model with corner case data to improve its ability to handle corner case tasks. Finally, previously learned corner case scenarios are processed through uncertainty-based core data extraction, updating the core dataset to ensure that knowledge from prior corner cases is retained when adapting to new ones. In summary, our work

Read the complete text