TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos
Korawat Charoenpitaks, Van-Quang Nguyen, Masanori Suganuma, Kentaro Arai, Seiji Totsuka, Hiroshi Ino, Takayuki Okatani · 2025
Read TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos by Korawat Charoenpitaks, Van-Quang Nguyen, Masanori Suganuma, Kentaro Arai, Seiji Totsuka, Hiroshi Ino, Takayuki Okatani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05733.txt.
Introduction
Self-driving systems need to read traffic behavior over time and space, and today's multi-modal AI models are barely trained for it. Korawat Charoenpitaks and six co-authors build TB-Bench, a benchmark of eight perception tasks from the driver's viewpoint using dashcam images and video. They also release two instruction-tuning datasets, with 100,000 and 250,000 examples, plus simple baseline models.
Off-the-shelf models flopped: even GPT-4o averaged under 35 percent accuracy. Fine-tuned on the new data, the baselines reached up to 85 percent. Mixing the data with another traffic set improved that set's scores too. The authors frame the release as groundwork for bringing multi-modal models into driving perception and planning. The paper appears in 2025.
Catalog record
| CREATOR | Korawat Charoenpitaks, Van-Quang Nguyen, Masanori Suganuma, Kentaro Arai, Seiji Totsuka, Hiroshi Ino, Takayuki Okatani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-05733
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos Korawat Charoenpitaks, Van-Quang Nguyen, Masanori Suganuma, Kentaro Arai, Seiji Totsuka, Hiroshi Ino, Takayuki Okatani Abstract The application of Multi-modal Large Language Models (MLLMs) in Autonomous Driving (AD) faces significant challenges due to their limited training on traffic-specific data and the absence of dedicated benchmarks for spatiotemporal understanding. This study addresses these issues by proposing TB-Bench, a comprehensive benchmark designed to evaluate MLLMs on understanding traffic behaviors across eight perception tasks from ego-centric views. We also introduce vision-language instruction tuning datasets, TB-100k and TB-250k, along with simple yet effective baselines for the tasks. Through extensive experiments, we show that existing MLLMs underperform in these tasks, with even a powerful model like GPT-4o achieving less than 35% accuracy on average. In contrast, when fine-tuned with TB-100k or TB-250k, our baseline models achieve average accuracy up to 85%, significantly enhancing performance on the tasks. Additionally, we demonstrate performance transfer by co-training TB-100k with another traffic dataset, leading to improved performance on the latter. Overall, this study represents a step forward by introducing a comprehensive benchmark, high-quality datasets, and baselines, thus supporting the gradual integration of MLLMs into the perception, prediction, and planning stages of AD. 1 Tohoku University 2 RIKEN Center for AIP 3 DENSO CORPORATION [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] * Corresponding authors Introduction Figure 1: Examples of four tasks from TB-Bench; additional task examples are provided in the supplementary material. The application of MLLMs to Autonomous Driving (AD) has gained increasing attention, particularly for predicting risks and planning actions based on images or videos from in-vehicle cameras. Notably, MLLMs have demonstrated their effectiveness in the international competitions like Autonomous Grand Challenge ( Renz et al. 2024 ) and in specific tasks such as traffic sign detection ( Zhang et al. 2024b ) . However, two major challenges remain. First, current MLLMs, ranging from proprietary models like GPT-4o ( Achiam et al. 2023 ) and Gemini ( Team et al. 2023 ) to open-source models like LLaVA ( Liu et al. 2024b ) , are not optimized for dashcam images or traffic scenes. These models are primarily trained on vast amounts of web-based text and image-text pairs, with minimal traffic-specific data, limiting their effectiveness in AD scenarios. To improve the generalizability of MLLMs, incorporating high-quality domain-specific datasets into the pre-training data is crucial, as shown in ( Li et al. 2024a ; Zhang et al. 2024a ) . Second, it lacks a dedicated benchmark for evaluating MLLMs’ abilities in spatiotemporal understanding tasks, given their capabilities in vision-centric tasks are still developing. While these models are designed to handle diverse vision-language tasks, they struggle with complex visual understanding, such as spatial reasoning and object relationships ( Tian et al. 2024 ) . Even in common domains unrelated to traffic scenes, there are insufficient benchmarks (e.g., Cambrian-1 ( Tong et al. 2024 ) , V-star benchmark ( Wu and Xie 2024 ) ). Given that AD requires sophisticated geometric and spatiotemporal understanding to capture the dynamic interactions between the vehicle and other entities, high-quality and dedicated benchmarks are much needed. While MLLMs are increasingly applied in AD, the aforementioned challenges remain insufficiently addressed. Recent research has primarily focused on using pretrained MLLMs derived from web data for specific AD tasks, without thoroughly investigating these challenges. Another issue is to determine which AD tasks across different stages and levels should be addressed by MLLMs. Within the “perception, prediction, and planning” framework, the question becomes: which stages should MLLMs handle? Focusing solely on the perception stage and its associated tasks, it may not necessarily appear optimal to use MLLMs. Established technologies like LIDAR and CV methods such as object detection and visual odometry can accurately capture the vehicle’s position and spatiotemporal relationships in Euclidean space. However, when considering the use of MLLMs (or LLMs) in later stages, relying on such “Euclidean geometrically accurate” information in the earlier stage might not be optimal. Firstly, it is unclear how to input this information into LLMs. Moreover, achieving advanced understanding in later stages may require information representations specifically suited for MLLMs to perform higher-level tasks. This suggests the need for MLLM involvement from the perception stage. This study adopts this perspective, aiming to involve MLLMs in the perception stage by expressing spatiotemporal traffic scene tasks in natural language text. The underlying conjecture, as mentioned above, is that this approach could be important for prediction and planning in later stages by MLLMs. To address the challenge of lacking dedicated benchmarks, we introduce TB-Bench, one of the first comprehensive benchmarks specifically designed to evaluate MLLM’s understanding of traffic behaviors. This benchmark assesses MLLM’s capabilities to perform perception tasks based on dashcam images or videos from the ego-centric views of vehicles, including determining the spatial position or orientation of other vehicles and interpreting the behaviors of both ego-vehicles and surrounding traffic. Compared to existing benchmarks, TB-Bench encompasses a wider array of eight distinct perception tasks, 1 1 1 Eight proposed tasks: Relative Distance, Spatial Reasoning, Orientation Reasoning, Other Lane to Ego, Other Lane Changing, Other Turning, Ego Turning, and Ego Traverse Distance. each corresponding to a typical driver maneuver. Figure 1 shows examples of several tasks. To ensure consistent evaluation across a diverse range of MLLMs, we employ a straightforward protocol. Specifically, we pair questions with images or video clips, requiring an MLLM to respond in plain text. Performance of the MLLM is then assessed by measuring response accuracy. To address the challenge of insufficient training data for AD perception tasks, we introduce a high-quality dataset focused on traffic behavior understanding from ego-centric views. This dataset aligns with the task design of TB-Bench and is used for vision-language instruction tuning (VLIT) of MLLMs. We generate high-quality question-and-answer pairs using samples from established datasets such as KITTI, ONCE, and Argoverse 2. In total, we create TB-Bench comprising 2,000 manually constructed samples, along with two versions of training datasets: TB-250k containing 250,000 samples, and TB-100k (a more balanced version). In addition to evaluating existing MLLMs, we introduce a generic framework that serves as a strong baseline for our tasks, consisting of three standard components: a pretrained vision encoder, a multi-modal connector, and a pretrained LLM. The vision encoder extracts visual representations from inputs with varying number of frames, while the connector projects these embeddings into the LLM’s embedding space, finally the LLM generates task-specific responses on our benchmark. This lightweight model is designed for efficient fine-tuning on our proposed dataset(s). Using TB-Bench to evaluate popular proprietary models (GPT-4o and Gemini) and various state-of-the-art open-source MLLMs (LLaVA, Bunny, and InternVL), we find that none of these models excels across all traffic behavior understanding tasks. On average, the open-source models underperform random guessing, while proprietary models achieve only slightly better results, with average accuracy below 35%. In contrast, when fine-tuned on TB-100k or TB-250k, our proposed baseline models demonstrate strong performance across all tasks, with average accuracy ranging from 77% to 85%. This highlights the effectiveness of our dataset in enhancing MLLM traffic behavior understanding. Overall, our contributions are fourfold : 1) we introduce TB-Bench, a benchmark for assessing MLLMs on eight perception tasks of traffic behavior understanding; 2) we