Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues

1 st Pallavi Zambare, 2 nd Venkata Nikhil Thanikella, 3 rd Ying Liu · 2025

Read Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues by 1 st Pallavi Zambare, 2 nd Venkata Nikhil Thanikella, 3 rd Ying Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21161.txt.

Introduction

Pallavi Zambare, Venkata Nikhil Thanikella, and Ying Liu published a 2025 paper on predicting whether a pedestrian will cross, from raw video rather than frames. Self-driving cars have to read such intentions in city scenes. Standard systems learn from labeled frame sequences and need retraining for each new setting. The authors' method asks a multimodal model to judge continuous clips directly.

The system, built on Gemini 2.5 Pro, receives the clip plus extras: bounding boxes around the pedestrian and the car's speed, through specialized prompts. On a benchmark it reached seventy-three percent accuracy in its zero-shot form. Using uninterrupted video matters because intention lives in motion: a glance, a hesitation, a shift of weight. The work tests whether AI models can step into a safety-critical perception job off the shelf.

Catalog record

CREATOR1 st Pallavi Zambare, 2 nd Venkata Nikhil Thanikella, 3 rd Ying Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-21161

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues * Thanks: 1 st Pallavi Zambare, 2 nd Venkata Nikhil Thanikella, 3 rd Ying Liu Abstract Pedestrian intention prediction is essential for autonomous driving in complex urban environments. Conventional approaches depend on supervised learning over frame sequences and require extensive retraining to adapt to new scenarios. Here, we introduce BF PIP (Beyond Frames Pedestrian Intention Prediction), a zero-shot approach built upon Gemini 2.5 Pro. It infers crossing intentions directly from short, continuous video clips enriched with structured JAAD metadata. In contrast to GPT-4V–based methods that operate on discrete frames, BF-PIP processes uninterrupted temporal clips. It also incorporates bounding-box annotations and ego-vehicle speed via specialized multimodal prompts. Without any additional training, BF-PIP achieves 73% prediction accuracy, outperforming a GPT-4V baseline by 18 %. These findings illustrate that combining temporal video inputs with contextual cues enhances spatiotemporal perception and improves intent inference under ambiguous conditions. This approach paves the way for agile, retraining-free perception module in intelligent transportation system. I Introduction As autonomous vehicles (AVs) increasingly become a reality in modern transportation. Consequently, the necessity for an accurate and timely understanding of pedestrian behavior has emerged as a critical safety requirement. Among the various behavioral cues, pedestrian crossing intention prediction is mainly essential for anticipating potential road conflicts and ensuring safe navigation in urban environments [ 1 , 2 ] . Early research relied on handcrafted features such as trajectories, body orientation, and gaze, processed with RNNs or LSTMs to capture temporal dynamics [ 3 , 4 ] Recently, approaches like Pedestrian Graph+ and STCrossingPose employed Graph Convolutional Networks (GCNs) to represent spatial relationships and enhance posebased inference [ 5 , 6 ] . Transformer based methods like PIT-Block and IntFormer introduced enhanced spatiotemporal attention and outperformed earlier sequence models [ 7 , 8 ] . Despite these advances, supervised methods depend on static image sequences, require extensive labeled data, and generalize poorly to novel environments [ 2 , 9 , 10 ] . To overcome these limitations, Multimodal Large Language Models (MLLMs) such as GPT-4V, LLAVA, and GPT-4o have emerged. They offer robust zero-shot reasoning across vision and language inputs [ 11 , 12 , 13 ] . OmniPredict [ 14 ] , a new benchmark using GPT-4o, established the probability of instruction-prompted multimodal inference. It predicted pedestrian behavior using scene images, bounding boxes, and vehicle speed. Although effective, it still processes discrete frame sequences, restraining its ability to capture motion cues like hesitation, body shifts, or gaze changes [ 15 , 16 ] . Even at its best, OmniPredict employs static frame sequences and treats the scene as a series of discrete observations. This limits the temporal continuity that is naturally captured in video. In contrast, video-based inputs can capture motion dynamics, hesitation, gaze shifts, and interaction cues that are not easily inferred from still frames. Fig. 1: BF-PIP Framework In this work, we introduce BF-PIP, a zero-shot pedestrian intention prediction framework that leverages short video clips and along with structured metadata (bounding boxes and ego-vehicle speed) in a multimodal prompt to Gemini 2.5 Pro. In contrast to previous methods that relied on image sequences, our method allows for continuous motion capture, enabling spatiotemporal reasoning without the need for retraining. Evaluation was conducted using the JAAD beh [ 19 ] dataset, a widely recognized benchmark in autonomous driving research. BF-PIP reached an impressive 73% accuracy in a zero-shot setting, representing an 18% improvement over the performance of GPT4V-PBP [ 4 ] and outperforming OmniPredict [ 14 ] by 6%, which is currently the leading MLLM-based approach. Without requiring additional training, BF-PIP outperforms domain-specific models in predicting pedestrian intent. Qualitative analysis confirms Gemini 2.5 Pro’s ability to interpret complex scenes using spatial and behavioral cues. An ablation study further quantifies the contribution of each input modality. In conclusion, BF-PIP marks a significant advance beyond traditional vision-based models by directly analyzing continuous video streams with structured metadata. It operates within a prompt-driven, zero-shot framework. Results indicate that BF-PIP’s ability to process raw video input reduces the need for extensive preprocessing. This capability enables zero-shot intent prediction in unfamiliar environments, contributing to more efficient and safer autonomous driving operations. II Methodology Gemini 2.5 Pro represents a significant advancement in multimodal AI, offering the ability to process and reason over rich combinations of video, image, and text data through a single prompt interface. Unlike previous models, Gemini 2.5 Pro natively supports raw video input, enabling temporally grounded understanding of pedestrian motion and scene context. In our experiments, we used the Google Gemini API to perform zero-shot pedestrian intention prediction tasks on the JAAD dataset [ 2 ] through the Gemini 2.5 Pro model. Specifically, we focused on the JAADbeh subset [ 2 ] , which includes detailed annotations of pedestrian behavior and environmental factors. II-A Task Identification Pedestrian crossing intention prediction is formulated as a binary classification task: the model must determine whether a pedestrian will cross or not cross the road within a fixed future time horizon. Given a temporal sequence of observations captured from the ego-vehicle’s front-facing camera, the goal is to anticipate the pedestrian’s crossing action 30 frames (1 second) into the future, referred to as the Time-To-Event (TTE). For each pedestrian instance, an observation window of 16 frames (approximately 0.5 seconds) immediately preceding the TTE point is defined. These frames capture the pedestrian’s motion history and environmental context leading up to the moment of prediction. This temporal segment is extracted following the JAAD benchmark protocol [ 17 , 2 ] , where t 0 represents the last observed frame and prediction occurs at t 0 + TTE. The task is defined under two input configurations: annotated and unannotated. In the annotated setting, short video clips are provided with per-frame pedestrian bounding boxes from the JAAD dataset, preserving temporal continuity and enabling precise spatial localization. In contrast, the unannotated setting involves raw video clips without bounding boxes, allowing assessment of Gemini 2.5 Pro’s ability to infer pedestrian intent solely from visual motion and contextual cues. II-B Input Modalities The BF-PIP pipeline accepts three main types of input data to support pedestrian intention prediction. These modalities are incorporated either as visual embeddings within the video stream or as structured metadata attached to each instance. • Short Video Clip : A temporally continuous short clip is constructed from the JAAD video sequences, capturing the target pedestrian within a 16-frame observation window. These clips maintain motion continuity and more closely resemble real-world perception compared to static image sequences. • Bounding Box Coordinates : Each frame includes bounding box annotations (x, y, width, height) extracted from JAAD metadata. These are either rendered into the clip (for the annotated case) or provided as separate input (for the structured mode). They help localize the pedestrian in the scene and focus the model’s attention. • Vehicle Speed : At each frame, vehicle speed is included as part of the ego-motion metadata to support contextual reasoning. For JAAD, categorical ego-speed values are used to indicate the trend of the vehicle’s motion (e.g., decelerating, maintaining constant speed, or accelerating). This multimodal input (video, spatial annotations, and motion data) is embedded into a prompt that’s given to

Read the complete text