Context-Aware Human Behavior Prediction Using Multimodal Large Language Models: Challenges and Insights

Yuchen Liu, Lino Lerch, Luigi Palmieri, Andrey Rudenko, Sebastian Koch, Timo Ropinski, Marco Aiello · 2025

Read Context-Aware Human Behavior Prediction Using Multimodal Large Language Models: Challenges and Insights by Yuchen Liu, Lino Lerch, Luigi Palmieri, Andrey Rudenko, Sebastian Koch, Timo Ropinski, Marco Aiello (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00839.txt.

Introduction

Yuchen Liu and co-authors test whether multimodal language models can forecast what a person in a scene will do next. Robots sharing human spaces need such foresight, and classic predictors must be retrained for each setting. Big models promise one flexible brain for all contexts, but they choke on long inputs, fuss over prompts, and cost much to tune.

The team builds a modular test bench varying the model, the inputs, in-context examples, and step-by-step generation. The best setup reached 92.8 percent semantic similarity to the true next action and 66.1 percent exact label matches. The paper maps which choices matter, offering a field guide for behavior prediction without bespoke training.

Catalog record

CREATORYuchen Liu, Lino Lerch, Luigi Palmieri, Andrey Rudenko, Sebastian Koch, Timo Ropinski, Marco Aiello
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00839

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Context-Aware Human Behavior Prediction Using Multimodal Large Language Models: Challenges and Insights Yuchen Liu, Lino Lerch, Luigi Palmieri, Andrey Rudenko, Sebastian Koch, Timo Ropinski, Marco Aiello Abstract Predicting human behavior in shared environments is crucial for safe and efficient human-robot interaction. Traditional data-driven methods to that end are pre-trained on domain-specific datasets, activity types, and prediction horizons. In contrast, the recent breakthroughs in Large Language Models (LLMs) promise open-ended cross-domain generalization to describe various human activities and make predictions in any context. In particular, Multimodal LLMs (MLLMs) are able to integrate information from various sources, achieving more contextual awareness and improved scene understanding. The difficulty in applying general-purpose MLLMs directly for prediction stems from their limited capacity for processing large input sequences, sensitivity to prompt design, and expensive fine-tuning. In this paper, we present a systematic analysis of applying pre-trained MLLMs for context-aware human behavior prediction. To this end, we introduce a modular multimodal human activity prediction framework that allows us to benchmark various MLLMs, input variations, In-Context Learning (ICL), and autoregressive techniques. Our evaluation indicates that the best-performing framework configuration is able to reach 92.8% semantic similarity and 66.1% exact label accuracy in predicting human behaviors in the target frame. I Introduction The ability to accurately predict human behavior is essential for robotic systems operating in environments shared with humans [ 1 ] . This skill is critical for ensuring safety and efficiency across a wide range of applications, including autonomous driving [ 2 ] , intralogistics [ 3 ] , and home automation [ 4 ] . Predictive collision avoidance [ 5 ] , anticipatory human-robot interaction (HRI) [ 6 ] and safe navigation [ 7 ] all rely on the system’s capacity to anticipate human actions and movements effectively. Traditional learning-based methods struggle with transferring to unseen environments and activities that were not part of the training set [ 8 ] . In contrast, Large Language Models (LLMs) exhibit promising generalization capabilities. Leveraging rich contextual embeddings and vast commonsense knowledge, LLMs can handle a large variety of environments and domains, considerably improving the performance and efficiency of the downstream tasks [ 9 ] . However, directly applying LLMs for human behavior prediction is challenging since they can only process unimodal information, i.e., textual tokens, and lack visual understanding capabilities. As a result, they often hallucinate the contents of the scene, which is particularly problematic for long-term, real-world human behavior forecasting tasks [ 10 ] . Extended with visual encoders and adapters, Multimodal Large Language Models (MLLMs) can process several types of inputs and ground them in the real world, as illustrated in Fig. 1 . This grounding makes MLLMs suitable for a wide range of vision-enabled robotics applications and offers promising avenues for predicting human behavior by combining visual scene understanding with natural language reasoning [ 11 ] . Fig. 1 : Context-aware human behavior prediction using Multimodal Large Language Model (MLLM). The MLLM (shaded zone) is the main component of the prediction system. It consists of an LLM, a visual encoder, and an adapter (e.g., MLP, Q-Transformer, or cross-attention layers for generating tokens passed to the LLM). The MLLM takes the user instruction and the historical visual observation as input, and forecasts the human behavior. With the goal of analyzing the applicability of pre-trained MLLMs for context-aware human behavior prediction, we introduce a modular MLLM-based prediction framework and conduct rigorous experiments analyzing the impact of different model components on prediction performance. In particular, we make use of several representative proprietary and open-source general-purpose MLLM backbones, In-Context-Learning (ICL) prompt techniques, and encodings of historical observations described with past activity labels, images, and scene captioning. We also review the effect of varying the numbers of ICL examples (including zero-shot) and applying autoregressive vs. direct output. Prior art human behavior prediction systems typically address the problem of predicting the end-point of the video sequence as early as possible, using the few initial frames as input. These methods are pre-trained on a given domain and produce a single action label as the prediction. In contrast, we propose to use general-purpose MLLMs to predict potentially several future activities at an arbitrary future time instance. To the best of our knowledge, this is the first attempt to systematically analyze how MLLMs can be applied to anticipate potentially multiple human activities in the same frame. To that end, we formulate and evaluate several hypotheses on building a performant prediction system: H1: MLLMs are capable of correctly predicting non-trivial sequences of human activities (i.e., such where the target label is different from the latest observation, or where multiple correct activity labels are given as the ground truth). H2: Adding visual context yields more accurate predictions, compared to using purely text-based LLMs. H3: Prediction accuracy generally improves with the number of ICL examples, compared to zero-shot prediction. H4: Predicting intermediate actions between the current and the target time frame in an autoregressive manner improves the accuracy. In our extensive evaluation on PROX and PROX-S datasets [ 12 , 13 ] , the best-performing configuration (in terms of the visual representation type, number of ICL examples, and autoregressive prediction method) is able to reach 92.8\% semantic similarity and 66.1\% exact label accuracy in predicting multiple human behaviors in the target frame. Our findings indicate that a small number of ICL examples benefits the performance of the state-of-the-art MLLMs. Compared to textual inputs, images provide significant additional value in improving visual scene understanding capability, and image captioning further improves prediction accuracy. Fig. 2 : System overview for human behavior prediction with scene context. The top part depicts the ground truth human interactions . The prediction system is illustrated in the lower part, which consists of an MLLM that is instructed with a prompt including the task description to predict future interaction labels and a set of In-Context Learning (ICL) examples. We ablate the visual input with four types of visual context representations : blind (no visual input), image, image sequence, and image caption, varying from one to three past time steps. An autoregressive prediction step is added to improve the predictions, which predicts the intermediate action labels. II Related Work Human behavior prediction has been a central focus in various fields, ranging from robotics to video analysis. Early approaches mainly relied on classical Deep Learning (DL)-based and data-driven techniques, leveraging models like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to process spatiotemporal patterns from visual input data, such as images, videos, or sensor signals, for activity recognition and trajectory prediction tasks [ 1 , 14 , 15 , 16 , 17 , 18 ] . Approaches like Social LSTM [ 15 ] and Social GAN [ 16 ] use RNNs and Generative Adversarial Networks (GANs), respectively, to predict pedestrian trajectories by learning interactions between individuals in shared spaces. However, these conventional methods often struggle with the growing prediction horizon and the complexity of human-object interactions [ 19 , 14 ] . Furthermore, these data-driven approaches have difficulties in generalizing to unseen environments and human behaviors [ 20 ] . The challenges can be tackled with the rapid development of more advanced methods, such as LLMs and MLLMs. The rich embedded common-sense knowledge enables LLMs to

Read the complete text