Pedestrian Intention Prediction via Vision-Language Foundation Models

Mohsen Azarmi, Mahdi Rezaei, He Wang · 2025

Read Pedestrian Intention Prediction via Vision-Language Foundation Models by Mohsen Azarmi, Mahdi Rezaei, He Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04141.txt.

Introduction

Mohsen Azarmi, Mahdi Rezaei, and He Wang test vision-language foundation models for pedestrian crossing prediction in this two thousand twenty-five study. Knowing whether a pedestrian intends to cross is critical for autonomous vehicles. Conventional vision methods often generalize poorly and reason weakly about context and causes. The authors instead feed visual frames, physical cue observations, and ego-vehicle dynamics into hierarchical prompt templates for the models.

Experiments ran on three standard datasets, JAAD, PIE, and FU-PIP. Adding vehicle speed, its change over time, and time-aware wording raised accuracy by up to about twenty percent. An automatic prompt engineering framework added further gains of about twelve percent. The results suggest foundation models beat conventional vision methods on generalization and contextual understanding for this safety task.

Catalog record

CREATORMohsen Azarmi, Mahdi Rezaei, He Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04141

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pedestrian Intention Prediction via Vision-Language Foundation Models Mohsen Azarmi, Mahdi Rezaei, He Wang Abstract Prediction of pedestrian crossing intention is a critical function in autonomous vehicles. Conventional vision-based methods of crossing intention prediction often struggle with generalizability, context understanding, and causal reasoning. This study explores the potential of vision-language foundation models (VLFMs) for predicting pedestrian crossing intentions by integrating multimodal data through hierarchical prompt templates. The methodology incorporates contextual information, including visual frames, physical cues observations, and ego-vehicle dynamics, into systematically refined prompts to guide VLFMs effectively in intention prediction. Experiments were conducted on three common datasets—JAAD, PIE, and FU-PIP. Results demonstrate that incorporating vehicle speed, its variations over time, and time-conscious prompts significantly enhances the prediction accuracy up to 19.8%. Additionally, optimised prompts generated via an automatic prompt engineering framework yielded 12.5% further accuracy gains. These findings highlight the superior performance of VLFMs compared to conventional vision-based models, offering enhanced generalisation and contextual understanding for autonomous driving applications. I Introduction The safe and efficient operation of autonomous vehicles (AVs) relies on understanding pedestrian crossing intentions. Accurate intention prediction enhances safety by enabling AVs to anticipate actions and adjust their speed and trajectory accordingly. However, this task remains challenging due to the complexity of pedestrian behaviour, influenced by individual attributes, social interactions, and environmental factors [ 1 ] . Conventional computer vision-based models for crossing intention prediction often use deep learning techniques, such as CNNs [ 2 , 3 , 4 ] , RNNs [ 5 , 6 , 7 ] , GCNs [ 8 , 9 ] , and Transformers [ 10 , 11 , 12 ] , focusing on visual features like body pose [ 13 , 14 ] , spatio-temporal relationships [ 15 , 16 ] , and pedestrian-vehicle dynamics [ 17 ] . While effective, these models struggle with generalizability, context understanding, and causal reasoning in dynamic traffic environments [ 18 ] . Fig. 1: Overview of vision-language crossing intention prediction model. Automatic Prompt Engineer (APE) generates hierarchical prompts based on task role, pedestrian behaviour, and dynamic motion cues (e.g., vehicle speed). Vision-language foundation models (VLFMs) are large-scale, pre-trained machine learning models that integrate visual and textual modalities to achieve a unified understanding of multimodal data. By combining multimodal data, VLFMs address some limitations of conventional vision-based models, improving cross-modal comprehension tasks [ 19 ] . While some recent works have explored the use of VLFMs for pedestrian intention prediction tasks [ 20 , 21 , 22 ] , they exhibit key limitations. Firstly, these approaches often fail to effectively capture temporal dependencies critical for dynamic pedestrian behaviour, relying instead on static or limited contextual cues. Secondly, their performance is sensitive to prompt variations, as prompts are often designed in an arbitrary or ad hoc manner, lacking a systematic structure. Finally, these studies frequently overlook the significant role of vehicle dynamics—such as speed and acceleration—on pedestrian intent, which are crucial for understanding traffic scenes [ 10 , 23 , 24 ] . This study explores the use of four state-of-the-art VLFMs for pedestrian crossing intention prediction, with a particular focus on prompt engineering and the integration of vehicle speed variations and time-conscious prompts into the intent-predictive framework. This research introduces structured hierarchical templates for prompt development. The hierarchy progresses from simpler to more complex prompts, each designed to capture different levels of contextual information. This progressive approach reflects human reasoning, which often builds on foundational knowledge before addressing more intricate details [ 25 ] . These prompts are subsequently optimised using the Automatic Prompt Engineer (APE) framework [ 26 ] , which iteratively refines them to identify the high-score prompts. The study also investigates how varying levels of contextual information influence crossing intention prediction performance against three task-specific datasets. Then our contributions can be summarised as follows: • VLFMs Adaptation: We integrate vehicle speed variations and refine prompts to improve the understanding of temporal and physical cues in pedestrian intention prediction. • Hierarchical Prompt Design: We propose a structured series of progressively complex prompts to evaluate VLFMs’ predictive capabilities. • Prompt Optimisation: We employ the APE framework to systematically optimise prompt templates, balancing accuracy and confidence, and improving the overall performance of the VLFMs in pedestrian crossing intention prediction. II Methodology This section describes our approach to pedestrian crossing intention prediction using vision-language foundation models (VLFMs). We integrate vehicle dynamics into the prompt structure through hierarchical templates and optimise prompts to enhance VLFMs’ effectiveness for the task. II-A Input Data Preparation The data we utilised to input to the VLFMs comprises both visual and textual components as detailed below. II-A 1 Visual Data The visual data consists of a series of frames, \mathbf{V}_{i}={frame_{1},frame_{2},\ldots,frame_{N}} , with N=16 frames captured at a rate of 30 fps. Following the same framework employed by other conventional vision-based benchmark models for performance evaluation, the model is configured to analyse information from the previous 16 frames and generate predictions for the next 16 frames [ 27 ] . To guide the model in understanding temporal sequences, each frame includes annotations with timestamps and a red bounding box highlighting the target ( i -th) pedestrian (as illustrated in Figure 1 ). II-A 2 Textual Data The textual data consists of a set of prompt templates \mathbf{P}=\{\rho_{1},\rho_{2},\ldots,\rho_{n}\} . These prompts direct the VLFMs to predict the pedestrian’s crossing intention based on visual frames and prompts, which are detailed below. II-B Hierarchical Prompt Templates To systematically evaluate the impact of contextual richness on the performance of VLFMs in pedestrian crossing intention prediction, we propose a series of progressively complex prompt templates as suggested by [ 25 ] . Each level of these templates builds upon the previous one, incrementally incorporating additional information to guide the model more effectively. This hierarchical structure ensures a comprehensive analysis of how varying degrees of context affect the model’s understanding of the scene. II-B 1 Role Templates ( \mathbf{P}_{R} ) The system role initially is based on definitions suggested in the study [ 21 ] , where the model acts as an autonomous vehicle equipped with a front-view dashboard camera capturing traffic scenes, including pedestrians and vehicles. The task requires predicting a pedestrian’s behaviour 16 frames into the future based on visual inputs and ego-vehicle speed information. The defined role helps the VLFM focus on the task effectively, as supported by [ 28 ] . The baseline prompt poses a simple question to establish a control for performance evaluation: ”Does the pedestrian in the red bounding box intend to cross the street?” . This minimal template serves as a benchmark for comparison with more contextually rich prompts. Moreover, referring to the bounding box in the prompt reduces ambiguity by ensuring the model targets the correct individual. Providing this localised visual reference enables the model to focus on the pedestrian more accurately, avoiding inaccuracies that may arise from textual descriptions of bounding box coordinates [ 29 ] . II-B 2 Physical Cues Observation ( \mathbf{P}_{B} ) This adds a layer of behavioural insight by specifying ”Observe the pedestrian’s posture, limb positions, and body orientation” . By explicitly focusing on these micro-level features, this template provides critical insights into the pedestrian’s readiness and intent to cross [ 30 ] . II-B 3

Read the complete text