VISTA: Vision-Language Imitation of Situational Thinking and Attention for Human-Like Driver Focus in Dynamic Environments

Kaiser Hamid, Khandakar Ashrafi Akbar, Nade Liang · 2025

Read VISTA: Vision-Language Imitation of Situational Thinking and Attention for Human-Like Driver Focus in Dynamic Environments by Kaiser Hamid, Khandakar Ashrafi Akbar, Nade Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05852.txt.

Introduction

Where a driver looks is oft shown as a heat map, which tells a spot but not the why of a glance. Kaiser Hamid, Khandakar Ashrafi Akbar, Peihang Li, and Nade Liang ask in a 2025 study if a vision and word model can say that why in plain words. They draw on a large set of road gaze data.

The team picks the frames where gaze jumps most and grades model takes in three modes: no help, one shot of help, and tuned on eighty expert fixed takes. The tuned mode beat the raw mode on word match and on how well named road things lined up. Expert grades rose too. Word takes, the team says, can sit next to heat maps in car watch work.

Catalog record

CREATORKaiser Hamid, Khandakar Ashrafi Akbar, Nade Liang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05852

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VISTA: Vision-Language Imitation of Situational Thinking and Attention for Human-Like Driver Focus in Dynamic Environments Kaiser Hamid, Khandakar Ashrafi Akbar, Nade Liang Abstract Driver visual attention prediction is a critical task in autonomous driving and human-computer interaction (HCI) research. Most prior studies focus on estimating attention allocation at a single moment in time, typically using static RGB images such as driving scene pictures. In this work, we propose a vision-language framework that models the changing landscape of drivers’ gaze through natural language, using few-shot and zero-shot learning on single RGB images. We curate and refine high-quality captions from the BDD-A dataset using human-in-the-loop feedback, then fine-tune LLaVA to align visual perception with attention-centric scene understanding. Our approach integrates both low-level cues and top-down context (e.g., route semantics, risk anticipation), enabling language-based descriptions of gaze behavior. We evaluate performance across training regimes (few shot, and one-shot) and introduce domain-specific metrics for semantic alignment and response diversity. Results show that our fine-tuned model outperforms general-purpose VLMs in attention shift detection and interpretability. To our knowledge, this is among the first attempts to generate driver visual attention allocation and shifting predictions in natural language, offering a new direction for explainable AI in autonomous driving. Our approach provides a foundation for downstream tasks such as behavior forecasting, human-AI teaming, and multi-agent coordination. 1 Introduction Driving is a highly visual task. Drivers’ visual attention allocation is critical for sufficient situation awareness and appropriate decision-making, particularly in safety-critical situations. Less optimal time and spatial allocation of visual attention (due to distraction, drowsiness, drunk driving etc.) is among the leading causes of road accidents [ 1 ] . For example, a disproportionate number of crashes occur at intersections because drivers fail to notice crossing traffic (often due to neglect or occlusion), and many lane-change collisions stem from not checking mirrors or blind spots [ 2 ] . Advanced Driver Assistance Systems (ADAS) and autonomous vehicles (AVs) need sufficient understanding of the divers’ visual attention allocation in diverse scenarios to facilitate human-AI teaming in both driver-vehicle and vehicle-vehicle interactions in mixed-traffic environments. Furthermore, an automated vehicle that understands or mimic human-like attention allocation could better anticipate hazards and improve safety. Indeed, a key open question is whether incorporating human selective attention mechanisms can improve autonomous driving (AD) performance, a topic that remains under-explored in current research [ 3 ] . Effectively modeling driver attention is challenging with conventional supervised learning models despite its importance. Training data typically consist of driver eye-tracking or gaze fixation annotations on driving scene recordings (e.g., [ 4 ] ), which are labor-intensive to collect and often limited in scenario diversity. Fully supervised models tend to be data hungry and struggle with domain-specific or long-tail scenarios that are rare in the training distributions [ 1 ] . For instance, dangerous but infrequent events (e.g., road debris, work zones, accidents) are underrepresented in datasets, leading to biased models that perform poorly in these corner cases [ 5 ] . As a result, even state-of-the-art end-to-end driving systems can be limited when faced with novel conditions or rare events not seen during training [ 4 ] . This data inefficiency motivates new approaches that can be learned from limited examples and still handle the diverse and evolving conditions of real roads. Moreover, accurately modeling and interpreting driver attention—including the span and focus of attention—can significantly enhance cross-verification of the vehicle’s decisions and responses across different modules, such as sensor inputs, attention prediction systems, and planning components. By understanding where and what the system is focusing on, it becomes possible to validate whether a particular maneuver (e.g., a turn or stop) aligns with the perceived scene context. This attention modeling can also support real-time interventions, such as providing haptic feedback to passengers or initiating fallback maneuvers, including remote override through teleoperation in edge-case scenarios. Integrating attention-based insights into the decision-making pipeline also supports the formation of interpretable decision chains, ultimately strengthening system transparency, and strengthen user trust in human-AI teams. To address the current issues and to lay foundation for robust autonomous driving systems, we introduce a novel few-shot driver attention modeling framework using a pre-trained vision-language model (LLaVA). By prompting the model with a small number of annotated examples, we enable attention reasoning without extensive supervised training. Our method predicts not only where a driver would look but also why, using free-form natural language. This in-context approach introduces a cognitively inspired, interpretable, and scalable paradigm for driving attention prediction. 2 Related Works 2.1 Driver Attention Modeling Driver attention modeling has long been studied as a visual saliency prediction task, where the objective is to estimate spatial gaze heatmaps based on dashcam images or video sequences [ 6 ] . Early works applied convolutional neural networks (CNNs) to this task, achieving reasonable results by learning correlations between scene features and human eye fixations. Subsequent models incorporated temporal dynamics and multi-modal inputs to improve prediction accuracy, using architectures such as CNN-LSTMs and feature fusion from RGB frames, optical flow, and semantic segmentations [ 3 , 7 ] . Existing driving attention models typically fall in one of the two categories: 1) bottom-up models that emphasize low-level visual features (e.g., motion, brightness), and 2) top-down models that integrate task-level signals such as GPS coordinates, navigation intent, or traffic semantics [ 8 , 2 ] . An established cognitive engineering framework, the Salience Effort Expectancy Value (SEEV) model [ 9 ] , outlines four integrated factors influencing overt visual attention allocation. These factors collectively fall into the bottom-up (salience, effort) and top-down (expectancy, value) categories. However, this “black box" approach provides limited insights into the underlying cause or attribution of theses four components, making it challenging to develop persuasive mitigation strategies. 2.2 Vision-Language Approaches for Interpretable Attention To enhance explainability of existing driver attention models, emerging work has applied vision-language models (VLMs) to model driver attention in a more interpretable and semantically rich way. Rather than outputting a heatmap, these models aim to describe the driver’s focus using natural language. For instance, “Attention Neural Baby Talk” [ 10 ] generates image captions that highlight hazardous elements, such as a pedestrian or cyclist, by aligning them with attention masks. Similarly, the DRAMA dataset [ 11 ] introduces driving videos annotated with natural-language question-answer pairs that explain risk, offering a human-interpretable rationale for attentional focus (e.g.,“The bicycle ahead is swerving into traffic”). These vision-language approaches suggest that semantic understanding and risk reasoning can complement or even surpass saliency maps in capturing true driver intent and awareness. 2.3 Large Vision-Language Models and In-Context Learning Modern VLMs such as BLIP [ 12 ] , Flamingo [ 13 ] , and LLaVA [ 14 ] have achieved impressive performance on image captioning, VQA, and multi-modal reasoning tasks with minimal supervision. BLIP used filtered web captions for pretraining, while Flamingo demonstrated few-shot capabilities via frozen LLMs and interleaved visual-text prompts. LLaVA linked a vision encoder with an LLM (Vicuna) to enable image-grounded dialogue and in-context learning [ 15 ] . These models generalize across tasks using only a few examples, making them attractive for domains where large labeled datasets are impractical. Recognizing these advantages, early research has begun to explore the application of powerful vision-language models

Read the complete text