Vision-language Models for Driver Monitoring Systems: A Driver Activity Description Dataset

David J. Lerch, Sarath Mulugurthi, Manuel Martin, Frederik Diederichs, Rainer Stiefelhagen · 2026

Read Vision-language Models for Driver Monitoring Systems: A Driver Activity Description Dataset by David J. Lerch, Sarath Mulugurthi, Manuel Martin, Frederik Diederichs, Rainer Stiefelhagen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02273.txt.

Introduction

Lerch and co-authors build a data set of fine-grained driver activity descriptions. Driver monitoring systems need to read subtle driver actions. Existing vision-language models train on general data sets. They struggle with fine distinctions in driver behavior. A detailed language version of the Drive and Act data set is created.

Three vision-language models are evaluated on the new benchmark. LLM-based scoring methods grade their output descriptions. The models cannot reliably produce accurate fine-grained descriptions. A new description data set is built from the labeled source. It is meant to train models on driver activity understanding. Cross data set tests use the Driver Monitoring Dataset. A fine-tuned model improves on that external test. Safer driver monitoring is the end goal.

Catalog record

CREATORDavid J. Lerch, Sarath Mulugurthi, Manuel Martin, Frederik Diederichs, Rainer Stiefelhagen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02273

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vision-language Models for Driver Monitoring Systems: A Driver Activity Description Dataset David J. Lerch, Sarath Mulugurthi, Manuel Martin, Frederik Diederichs, Rainer Stiefelhagen Abstract Understanding subtle driver actions is essential for building reliable driver monitoring systems. Existing vision-language models (VLMs) are trained on general datasets and struggle to recognize fine distinctions in driver behaviors. This paper addresses this limitation by creating a detailed natural language version of the Drive&Act dataset. We evaluate three VLMs on our new benchmark using LLM-based scoring methods. Their performance on the new benchmark shows that they cannot reliably generate accurate fine-grained driver activity descriptions. Based on the labeled Drive&Act dataset we create a new Drive&Act description dataset containing fine-grained descriptions to train VLMs on driver activity understanding. Cross dataset evaluation on the Driver Monitoring Dataset (DMD) shows that the VLM fine-tuned on our new Drive&Act description dataset generalizes well to actions in the DMD dataset. The VLM fine-tuned on our Drive&Act description dataset achieves an ACCR score of 76 outperforming the zero-shot VLM baseline with an ACCR score of 66. These findings demonstrate that adapting VLMs with richly described driver actions can significantly improve their ability to interpret driver behavior while also highlighting the need for more diverse datasets to support broader generalization in future applications. Our Drive&Act description dataset and code will be publicly available on GitHub. I Introduction Driver monitoring systems (DMS) have emerged as a critical technology for modern intelligent vehicles because human error, particularly inattention, distraction, and drowsiness, remains a dominant cause of road crashes and fatalities [ 9 ] . According to the EuroNCAP Vision 2030 [ 8 ] the requirements for DMS continue to increase. In addition to safety systems driver activity recognition can also contribute to comfort e.g. to prevent motion sickness [ 7 ] . Within this context, driver activity recognition focuses on identifying what the driver is doing, such as using a phone, reaching for objects, adjusting controls, or taking hands off the wheel, in order to enable timely warnings, adapt automation behavior, and support shared control between human and vehicle. Video-based and multi-modal activity recognition methods are therefore increasingly used to detect distracted, impaired, or unsafe driving behaviors [ 4 , 12 ] . Fig. 1: Overview over the pipeline of automatic description generation for driver actions. In the first step we use a labeled dataset and a LLM to generate ground truth descriptions. Thereafter we train a video-language model to predict the description based on a prompt and video input. While conventional deep models can classify predefined activities [ 26 ] , they struggle with recognizing subtle distinctions between semantically similar fine-grained actions e.g. opening and closing a bottle [ 17 ] . This limitation is increasingly problematic as vehicles become more automated and in-cabin activities diversify, creating long-tail behaviors and domain shifts that are difficult to cover with fixed taxonomies and closed-set classifiers. Furthermore, interpretability is imperative in safety-critical automotive systems. For instance, controllers and human operators must comprehend not only the class label but also the rationale underlying the system’s belief that a driver is distracted or engaged in a specific secondary task. Vision-language models (VLMs) are promising for driver activity recognition because they can align visual signals with expressive natural language, enabling open-vocabulary descriptions and queries about driver behavior rather than restricting inference to a small set of predefined classes. By grounding video segments in textual descriptions, VLMs can represent the internal structure of activities as sequences of fine-grained events, which is well aligned with hierarchical driver behavior annotations such as atomic action triplets in datasets like Drive&Act [ 17 ] . This expressiveness allows a single model to recognize both seen and novel driver behaviors via language prompts, improving generalization, which is particularly important as new in-cabin interfaces, devices, and interaction patterns appear over a vehicle’s lifetime. Parameter-efficient fine-tuning strategies (e.g. LoRA) further make it feasible to specialize large pre-trained VLMs to the driver monitoring domain. Consequently, integrating VLMs with fine-grained, temporally aligned driver activity annotations offers a compelling path toward more accurate, robust, and semantically meaningful driver monitoring systems that better support the safety objectives of future intelligent vehicles. We show that current VLMs fail in describing driver activities. In order to deploy VLMs in DMS, driver activity datasets are required that enable effective training of such models [ 3 ] . This paper proposes a pipeline (see Figure 1 ) to train VLMs on driver activity recognition for DMS. Our main contributions are: 1. Benchmarking VLMs on Driver Activity Recognition: We benchmark three SotA VLMs on the downstream task of driver activity recognition. We show that SotA VLMs fail in the downstream task of driver activity recognition underlining the necessity for finetuning and consequently training datasets. 2. New Driver Action Description Dataset: We create a new dataset based on the existing Drive&Act dataset containing fine-grained activity descriptions. Fine-grained descriptions are necessary for VLM training datasets. 3. Cross-dataset Generalization Capabilities: We demonstrate that a VLM trained on our Drive&Act-Description dataset can generalize to the DMD dataset [ 20 ] . This finding underscores the necessity of fine-grained description datasets for in-cabin monitoring. II Related Work II-A Vision-language Models VLMs were developed to bridge the gap between visual perception and linguistic understanding. The earliest efforts toward combining vision and language centered primarily on image captioning systems. These models relied on convolutional neural networks (CNNs) to extract visual features, which were then fed into recurrent neural networks such as LSTM-based decoders to generate textual descriptions. Examples include Show and Tell [ 23 ] , which coupled a CNN encoder with an LSTM decoder, and Show, Attend and Tell [ 28 ] , which introduced attention mechanisms allowing the system to focus on specific image regions. Although these early practices established the foundation for cross-modal learning, they were limited by their dependence on supervised datasets and their inability to generalize across tasks without retraining. A major step in vision-language learning were contrastive learning–based models like CLIP (Contrastive Language–Image Pretraining) [ 21 ] , which jointly trained image and text encoders using large-scale image-text pairs. Building on earlier models like CLIP, VideoCLIP [ 27 ] applied the idea of contrastive learning to video. It used video and text pairs to train a model that could link motion in videos with language, using features that captured movement and appearance. ViCLIP [ 24 ] extended this line of work by introducing a contrastive loss tailored for video data. This allowed the model to better track how visual content changes over time, improving its ability to align video clips with text in a temporally consistent way. LLaVA [ 16 ] was one of the first models to link a vision encoder with an instruction-following language model. It connected the visual encoder from CLIP to the language model Vicuna using a small projection layer. This allowed the model to follow multi-modal instructions with only a few changes to the architecture. Video-LLaVA [ 15 ] used image-video pre-alignment which allows the LLM to learn a single unified visual representation. These earlier VLMs were limited in their ability to process long videos and capture fine-grained actions due to constraints in temporal modeling and input length. To overcome these limitations, specialized

Read the complete text