Unlock the Power of Unlabeled Data in Language Driving Model

Chaoqun Wang, Jie Yang, Xiaobin Hong, Ruimao Zhang · 2025

Read Unlock the Power of Unlabeled Data in Language Driving Model by Chaoqun Wang, Jie Yang, Xiaobin Hong, Ruimao Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10586.txt.

Introduction

Chaoqun Wang and co-authors tap unlabeled video to teach driving models in 2025. Vision language models for driving need vast labeled data, which is costly to build. Roads offer endless unlabeled clips instead. The team first trains a model on the small labeled set it has.

Template prompts then draw scene facts from unlabeled clips as draft answers. A self consistency pass cleans those drafts before they rejoin training. Built on a strong open vision model, their system tops prior methods on driving scene questions. The recipe stretches scarce labels across the flood of raw road video.

Catalog record

CREATORChaoqun Wang, Jie Yang, Xiaobin Hong, Ruimao Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unlock the Power of Unlabeled Data in Language Driving Model Chaoqun Wang, Jie Yang, Xiaobin Hong, Ruimao Zhang Abstract Recent Vision-based Large Language Models (VisionLLMs) for autonomous driving have seen rapid advancements. However, such promotion is extremely dependent on large-scale high-quality annotated data, which is costly and labor-intensive. To address this issue, we propose unlocking the value of abundant yet unlabeled data to improve the language-driving model in a semi-supervised learning manner. Specifically, we first introduce a series of template-based prompts to extract scene information, generating questions that create pseudo-answers for the unlabeled data based on a model trained with limited labeled data. Next, we propose a Self-Consistency Refinement method to improve the quality of these pseudo-annotations, which are later used for further training. By utilizing a pre-trained VisionLLM (e.g., InternVL), we build a strong Language Driving Model (LDM) for driving scene question-answering, outperforming previous state-of-the-art methods. Extensive experiments on the DriveLM benchmark show that our approach performs well with just 5% labeled data, achieving competitive performance against models trained with full datasets. In particular, our LDM achieves 44.85% performance with limited labeled data, increasing to 54.27% when using unlabeled data, while models trained with full datasets reach 60.68% on the DriveLM benchmark. I Introduction Recent advancements in Vision Large Language Models for autonomous driving have made significant progress [ 1 , 2 , 3 ] . These models follow a common process, which perceives the driving scene and makes driving decisions by asking the corresponding questions, making driving scene VQA a critical task. However, as application scenarios continue to diversify, with increasingly stringent precision requirements, the need for large volumes of annotated data to improve model performance becomes more pressing, meanwhile, large-scale annotating is both labor-intensive and time-consuming. This naturally raises the following question: How to train the Language Driving Model with readily collectible unlabeled data? A straightforward and intuitive approach is to follow a semi-supervised learning paradigm, which aims to generate pseudo labels for unlabeled data using a model trained on a few labeled seed data. Nevertheless, for autonomous driving with complex VQA tasks, this strategy presents several challenges: 1) Diverse questions : How can we involve diverse questions that allow the model to infer meaningful VQA annotations? 2) High-quality answer : The quality of the predicted pseudo label is heavily coupled to the performance of the trained LDM and is crucial for sequential training. Fig. 1: (a): The proposed semi-supervised Language Driving Mode (LDM) traning pipeline. We first fine-tune an LDM in a supervised manner with a few labeled data. Then, given a fine-tuned LDM and unlabeled data, we generate VQAs via templated-based prompts and refine them by the proposed Self-Consistency Refinement (SCR). The refined VQAs are used for subsequent model training. (b): Preliminary experiment on DriveLM benchmark, inferring the fine-tuned LDM with hints could obtain better performance. To obtain high-quality and diverse pseudo annotations, we attempt to utilize the inherent rules in the driving scenes of the physical world. Empirical practice illustrates that in language-driving annotation, an image of a scene is often paired with multiple QAs, which display multiple objects related to the ego vehicle and their interactions. Therefore, another intuition arises as to whether these connections and interactions in scenes could help LDMs better understand the objects. To verify the positive effect of the “hints” for the object understanding, we conduct preliminary experiments as shown in Fig. 1 -(b). Instead of just predicting object states using question prompts, we extracted cues from the scene and added them to the prompts for better predictions. Finally, we found that by using the prompts with hints, LDM could obtain more accurate predictions. Based on the above observations, we propose a novel iterative training pipeline to unlock the potential of readily available unlabeled data in a semi-supervised learning manner as shown in Fig. 1 -(a). Firstly , we introduce a series of template-based prompts to extract the scene information by excavating the status of the objects related to safe driving, which could provide diverse questions to infer the LDM given multi-view images for a driving scene, which is trained with seed labeled data. Secondly , we construct the scene graph to adequately represent important objects in the scene and the relationships between them based on the generated multiple QA pairs of a driving scene, which provides important clues to predict their future state. Thirdly , we retrieve many hints from the scene graph for each QA pair and re-ask the question with the hints for a better response. Furthermore, we propose Self-Consistency Refinement (SCR) that considers various hints to achieve the refinement of pseudo-labels with arbitrary precision. In SCR, two different prompts for the same question constitute an important accordance for the trustworthiness computation of the sample. As shown in Fig. 1 -(a), given a fine-tuned LDM and large-scale unlabeled data, our proposed SCR could produce high-quality VQA pairs for the subsequent model training. In such a manner, we could constantly promote the LDM ability by interactively increasing the unlabeled data beginning with seed annotated data. To demonstrate the effectiveness of our proposal, we conduct experiments on the DriveLM benchmark and deliver an iterative training setting. First, we meticulously design the prompts to unlock the potential of large models in driving scenes and gain state-of-the-art performances based on the off-the-shelf pre-trained VisionLLM (e.g. InternVL2). Then, to show the effectiveness of the proposed interactive training pipeline based on the semi-supervised learning paradigm, we train the model with seed (5%) annotations and gain huge promotion with iterative incremental large-scale (20% and 75% for 2 iterations) unlabeled data. Specifically, our model trained on the full annotation achieves a 60.68 % final score, which is comparable with the past competition 1 1 1 https://opendrivelab.com/challenge2024/#driving_with_language winner (60.02% final score). Meanwhile, for the semi-supervised setting, our model gains 44.85% with 5% labels. By employing our proposed interactive training pipeline with 95% unlabeled images, our proposal earns a 54.27% final score and + 9.42 % promotion. The main contributions are summarized as follows: • As far as we know, this is the first work to unlock the potential of unconstrained unlabeled visual data to improve the Language Drive Model in a semi-supervised training paradigm. • We propose a novel self-consistency refinement method to improve the quality of generated pseudo-labels via the graph-based hints, paving a new path to scale up training data. • We conduct extensive experiments to demonstrate the effectiveness of our proposal and gain remarkable promotions compared to the baseline model. II Related Work II-A Large Driving Model Vision-based Large Language Models (VisionLLMs) seamlessly integrate Natural Language Processing (NLP) with Computer Vision (CV), which is instrumental in advancing applications like scene perception, planning, question answering, and content generation in autonomous driving. For example, leveraging the visual-text alignment capabilities of vision-language pre-trained models like CLIP, several prior works [ 4 , 5 , 6 ] have demonstrated success in detecting open-vocabulary objects in autonomous driving scenes. Additionally, some approaches [ 7 , 8 ] integrate cross-modal features into a prompt reasoning framework to predict 3D objects, while recent studies [ 9 , 10 , 11 , 12 ] directly use VisionLLMs to perceive complex driving scenarios and behaviors, such as identifying long-tail risk objects

Read the complete text