MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding

Renjie Li, Ruijie Ye, Mingyang Wu, Hao Frank Yang, Zhiwen Fan, Hezhen Hu, Zhengzhong Tu · 2025

Read MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding by Renjie Li, Ruijie Ye, Mingyang Wu, Hao Frank Yang, Zhiwen Fan, Hezhen Hu, Zhengzhong Tu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12463.txt.

Introduction

Renjie Li and co-authors wrote a 2025 paper introducing MMHU, a large benchmark for machine understanding of human behaviour in driving scenes. Safe self driving needs to read people: how they move, where they head, and what they intend. Past data sets each cover a slice.

MMHU gathers tens of thousands of motion clips and well over a million frames from driving sets, web video, and fresh collection. Human checked captions describe motion and intent, with safety critical labels. Tasks span forecasting motion, generating it, and answering questions about behaviour. The suite gives the field one shared yardstick.

Catalog record

CREATORRenjie Li, Ruijie Ye, Mingyang Wu, Hao Frank Yang, Zhiwen Fan, Hezhen Hu, Zhengzhong Tu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12463

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding Renjie Li, Ruijie Ye, Mingyang Wu, Hao Frank Yang, Zhiwen Fan, Hezhen Hu, Zhengzhong Tu Abstract Humans are integral components of the transportation ecosystem, and understanding their behaviors is crucial to facilitating the development of safe driving systems. Although recent progress has explored various aspects of human behavior—such as motion, trajectories, and intention—a comprehensive benchmark for evaluating human behavior understanding in autonomous driving remains unavailable. In this work, we propose MMHU , a large-scale benchmark for human behavior analysis featuring rich annotations, such as human motion and trajectories, text description for human motions, human intention, and critical behavior labels relevant to driving safety. Our dataset encompasses 57k human motion clips and 1.73M frames gathered from diverse sources, including established driving datasets such as Waymo, in-the-wild videos from YouTube, and self-collected data. A human-in-the-loop annotation pipeline is developed to generate rich behavior captions. We provide a thorough dataset analysis and benchmark multiple tasks—ranging from motion prediction to motion generation and human behavior question answering—thereby offering a broad evaluation suite. Our dataset will be released to promote further human-centric research in this vital area of autonomous driving. Project Page : https://MMHU-Benchmark.github.io Figure 1: We propose MMHU , a large-scale dataset for human behavior understanding. We collected 57k human instances with diverse behaviors such as playing mobile phone, holding object, or using mobility devices, from diverse scenes such as in the city, school, park, and alley. We provide rich annotations including motion and trajectory, text descriptions for human motions, and recognize the behaviors that are critical to driving safety. 1 Introduction Humans play an essential role in transportation systems, making the comprehensive understanding of human behaviors—such as motion xu2023auxiliary , intention osman2023tamformer ; xie2024gtranspdm ; yang2022predicting , and trajectory zhang2024incorporating ; fang2024behavioral —critical for developing safe autonomous driving systems. To effectively interact with humans, autonomous vehicles must answer human-centric questions, such as “What is the person doing?”, “Is the person going to cross the street?”, and “Where does the person intend to go?” Failing to accurately comprehend these behaviors could lead to misinterpretations of human intent, potentially resulting in fatal accidents. While significant efforts have been devoted to understanding individual aspects of human behaviors by investigating human motion, intention, and trajectory, the absence of a unified dataset limits the comprehensive evaluation of algorithms for human behavior understanding, especially in autonomous driving scenarios that mainly account for human safety. Existing driving datasets are typically designed for general driving tasks, such as depth estimation, 2D or 3D object detection, odometry, and semantic segmentation sun2020scalability ; caesar2020nuscenes ; geiger2013vision , or narrowly designed human-related tasks, such as intention prediction rasouli2019pie ; bhattacharyya2021euro ; rasouli2017they , motion and trajectory prediction von2018recovering , or motion reconstruction wangpedestrian ; kim2024text ; liu2024learning . Moreover, with the emergence of driving-oriented vision-language models (VLMs) touvron2023llama ; lin2023vila ; chen2024longvila ; liu2024nvila ; liu2023llava ; liu2024llavanext ; liu2023improvedllava ; zhang2024videoinstructiontuningsynthetic ; bai2023qwen , human behavior understanding tasks can now be approached in a more integrated and flexible manner through images and text queries. However, existing training data for these VLMs are not specifically tailored to human behavior, limiting their effectiveness in capturing critical human-centric details essential for safe driving. In this work, we aim to answer three core questions regarding human behavior understanding tasks in autonomous driving scenarios: ❶ What aspects of human behavior are critical to autonomous driving? ❷ How effectively do current approaches model human behaviors in autonomous driving contexts? ❸ How can a comprehensive benchmark advance the development of human behavior understanding algorithms? To this end, we propose MMHU , a large-scale unified benchmark explicitly designed for comprehensively understanding various human behaviors in driving scenarios. MMHU includes rich annotations generated by a human-in-the-loop annotation pipeline, enabling scalable and precise labeling from diverse data sources using only monocular video inputs. Specifically, we have collected 1.73M frames featuring 57K human instances from source videos obtained from Waymo sun2020scalability , YouTube, and self-collected data. The dataset provides detailed annotations covering: (1) human motion and trajectory; (2) text descriptions of human motions generated using templates and VLMs; (3) critical human behaviors extracted via VLMs, along with question-answer (QA) pairs designed to benchmark driving-oriented and generalist VLMs. Our contributions can be summarized as follows: • We introduce MMHU , a unified, human-centric dataset that provides a comprehensive understanding of humans’ behaviors in driving scenarios that can be used as a benchmark for a range of human-centric understanding tasks. • We develop a scalable, human-in-the-loop annotation pipeline employing multi-source fitting strategies to produce accurate labeling across diverse video sources, ranging from driving videos and general YouTube videos to self-collected streams. • We evaluated baseline methods of human behavior understanding and analyze their performance, we further demonstrated that our dataset helps these methods achieve better performance. 2 Related Works 2.1 Human Motion Human motion is essential to autonomous driving. We categorize human motion representations into 2D and 3D representations. 2D human motion jiang20242d ; belagiannis2017recurrent ; luo2021rethinking ; li2021tokenpose ; jin2020whole leverages keypoints or heatmaps to mark the local body motion on the image. For 3D representations, the SMPL series SMPL:2015 ; MANO:SIGGRAPHASIA:2017 ; SMPL-X:2019 provide compact and expressive representation via learned parameters. While most human motion datasets focus on general human motions ionescu2013human3 ; von2018recovering ; xu2024motionbank ; mahmood2019amass ; lin2023motion , there are several datasets are specially designed for driving scenarios wangpedestrian ; liu2024learning ; kim2024text . However, these datasets mainly focus on the human movements and their text description, the behaviors of humans remains unavailable. Based on the representation and datasets, several efforts have been put into human motion reconstructions from temporally aligned multi-view cameras huang2021dynamic , unaligned multi-view cameras dong2020motion , and monocular cameras luvizon2023scene ; li2022d ; ye2023decoupling ; yuan2022glamr . Other works have explored human motion generation from action labels cervantes2022implicit or text tevet2022human ; zhang2024motiondiffuse . However, due to the lack of high-quality data, there is little work that specifically generates human motion in driving scenarios. 2.2 Human Behavior Understanding Understanding human behavior in driving situations is essential for driving safety. Although there are some datasets and methods to understand human behavior and actions zhang2024incorporating ; rai2021home ; punnakkal2021babel ; wang2012mining ; li2010action ; soomro2012ucf101 ; niebles2010modeling ; kay2017kinetics ; marszalek2009actions , they mainly focus on recognizing human actions in general or sports scenes. While sharing some common behavior that concerns driving safety, they mainly focus on general actions like shaking hands, dancing, or running. The behaviors specifically concerning driving safety remain unexplored. In autonomous driving, besides motion reconstruction, there are some approaches and datasets for understanding several aspect of human behaviors such as (1) human trajectory prediction zhang2024incorporating ; goncalves2022auxformer ; medina2024context ; wang2024gcnext ; guo2023back ; mao2020history , where models are required to predict the future trajectory from previous ones, or (2) human intention prediction zhang2023trep ; osman2023tamformer ; sharma2023visual ; rasouli2019pie ; kotseruba2021benchmark ; Rasouli2017IV , where pedestrians are simply classified into two states - crossing the street and not crossing the street. Besides the binary classification of crossing the street, there are several datasets kwak2017pedestrian ; quintero2015pedestrian ; schneider2013pedestrian ; rasouli2017they that provide more detailed behavior labels such as stopping, glancing, or running. However, these works

Read the complete text