Scenario Understanding of Traffic Scenes Through Large Visual Language Models

Esteban Rivera, Jannik Lübberstedt, Nico Uhlemann, Markus Lienkamp · 2025

Read Scenario Understanding of Traffic Scenes Through Large Visual Language Models by Esteban Rivera, Jannik Lübberstedt, Nico Uhlemann, Markus Lienkamp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17131.txt.

Introduction

Self-driving systems learn from huge piles of street scenes, and sorting those scenes by type would help, but labeling them by hand is slow and costly. Esteban Rivera, Jannik Lübberstedt, Nico Uhlemann, and Markus Lienkamp test whether large visual language models can do the sorting. They try models such as GPT-4 and LLaVA on their own dataset and on a public driving dataset.

The models read each traffic scene and answer context questions, sorting scenes without retraining for new types. Numbers and close reading both showed the models grasp urban traffic scenes well. The team packages the approach as a captioning pipeline that can be pointed at fresh datasets. They present it as a cheap way to organize the data that driverless research runs on.

Catalog record

CREATOREsteban Rivera, Jannik Lübberstedt, Nico Uhlemann, Markus Lienkamp
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-17131

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scenario Understanding of Traffic Scenes Through Large Visual Language Models Esteban Rivera, Jannik Lübberstedt, Nico Uhlemann, Markus Lienkamp Abstract Deep learning models for autonomous driving, encompassing perception, planning, and control, depend on vast datasets to achieve their high performance. However, their generalization often suffers due to domain-specific data distributions, making an effective scene-based categorization of samples necessary to improve their reliability across diverse domains. Manual captioning, though valuable, is both labor-intensive and time-consuming, creating a bottleneck in the data annotation process. Large Visual Language Models (LVLMs) present a compelling solution by automating image analysis and categorization through contextual queries, often without requiring retraining for new categories. In this study, we evaluate the capabilities of LVLMs, including GPT-4 and LLaVA, to understand and classify urban traffic scenes on both an in-house dataset and the BDD100K. We propose a scalable captioning pipeline that integrates state-of-the-art models, enabling a flexible deployment on new datasets. Our analysis, combining quantitative metrics with qualitative insights, demonstrates the effectiveness of LVLMs to understand urban traffic scenarios and highlights their potential as an efficient tool for data-driven advancements in autonomous driving. 1 Introduction Models developed for autonomous driving require large amounts of training data to achieve acceptable functionality. However, even with large datasets, their performance can only be optimal within specific operational domains given the underlying data distribution [ 5 ] . Hence, to improve their generalization capabilities and, thus, model performance, it is essential to categorize the data to identify which situations are contained within the data distribution. Through this characterization, models can be optimized for a broader range of real-world scenarios, improving their overall reliability and robustness. In the case of autonomous driving, the categorization is performed through tags highlighting the properties of a scene [ 23 ] . As a prominent example, the daytime of a scene is important as camera object detectors trained solely on samples with sunlight suffer from decreased scores for nighttime scenes [ 11 ] . Similarly, crowded scenes provide challenges for different driving modules if only scenarios containing a few agents were present during training [ 27 ] . These two cases illustrate the need to choose suitable tags based on which an adequate distribution can be formed. Not only does this allow for a more balanced selection and subsequent training, but it also introduces a more effective workflow by replacing manual sample selection with a simple definition of the desired categories. In addition, it saves time and money as the manual labeling process can be both, costly and cumbersome. Figure 1 : Example for a common traffic scenario contained in our dataset, depicting the interaction of the ego vehicle with a group of pedestrians crossing over a crosswalk on a sunny day and a silver car parked on the sidewalk towards the left. Large Vision Language Models (LVLMs) have emerged as a promising tool to automate the task of categorization. By providing an image of a scene alongside a query reflecting the desired category, LVLMs can effectively estimate the similarity of the scene with the provided tag, yielding the desired categories after a few post-processing steps. Since these models are trained on vast amounts of data, retraining or fine-tuning is usually not required even when introducing new categories. This is in contrast to classical approaches where often convolutional neural networks are employed to classify a given image. Here, re-labeling of the data followed by re-training of the network is necessary anytime a new category is added, which both involves a significant effort and associated cost. Both, proprietary models like GPT-4 [ 17 ] and open-source models like LLaVA [ 13 ] , have achieved impressive results on visual questioning tasks recently [ 24 ] . Therefore, their capability to understand and reason about scenarios relevant for autonomous driving will be investigated in this work, paving the way for their potential application as automated categorizers for large datasets. In summary, the contributions of this study are threefold: • First, we provide a quantitative analysis of the scenario understanding of several LVLMs by categorizing traffic scenes for different tags on an in-house and the BDD100K dataset. • Second, we conduct a qualitative analysis of their performance for representative categories, highlighting strengths and shortcomings. • Lastly, we implement an automated categorization pipeline based on the investigated models which is applicable to arbitrary datasets focusing on traffic scenes. 1 1 1 https://github.com/TUMFTM/CatPipe 2 Related work Datasets for autonomous driving Datasets are the foundation for every learning-based approach and subsequently their performance evaluation. Since our goal is to use VLMs to classify images representing scenes in the context of autonomous driving, datasets found within the research community can offer a diverse spectrum of categories. One problem with this approach is that each has different tags containing information about the environment or detected objects themselves. In addition, some categories we want to detect are not captured at all. While information about pedestrians, bicyclists, traffic lights, and traffic signs can be obtained from object detection datasets like BDD100k [ 23 ] , Waymo [ 21 ] , Apolloscape [ 7 ] , Zenseact [ 25 ] and Cityscapes [ 2 ] , weather tags are only available in the metadata of the first two. Moreover, since BDD100k provides additional tags for the urban environment, it is therefore chosen for the verification of our results obtained for the different image classification approaches. Closer to the LVLM application for autonomous driving is the MAPLM dataset and benchmark [ 1 ] , which collects multimodal information relevant for question-answering. We do not look for comparison in the natural-language level from MAPLM, because we want to compare directly the tagging task, which does not require Natural Language capabilities to be solved. Visual Question Answering The term Visual Question Answering (VQA) describes the general approach of using LVLMs to extract visual information from images constrained by text. This is closely related to LVLM-based image classification. Therefore, we selected the models for the evaluation based on their performance on selected VQA datasets. VQA 2.0 [ 4 ] focuses on general vocabulary without requiring background information about the topics not conveyed in the image. In contrast, Pope [ 12 ] is a method to evaluate how susceptive a model is to hallucinations. Its focus is object detection, as it uses the ground truth of the COCO dataset to construct questions. For each object present in an image, positive samples are created. Negative samples are constructed by randomly sampling objects from an object list that are absent from a scene’s ground truth. Additionally, we use the MMBench [ 14 ] dataset as it evaluates a model’s susceptiveness to circular shift. This is important to reduce the influence of the input order in which the possible categories are provided in. Vision Language Models The models used for Visual Question Answering are typically LLMs with a vision encoder component. LLaVA uses a model from the CLIP [ 19 ] model family to obtain a feature vector of the input image which is projected and fed into the LLM. This approach can be generalized to larger input image resolutions by splitting the image into patches and feeding each patch into the vision encoder as done with ComposerHD

Read the complete text