ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Model
Qingyao Xu, Ya Zhang, Yanfeng Wang, Siheng Chen · 2025
Read ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Model by Qingyao Xu, Ya Zhang, Yanfeng Wang, Siheng Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13938.txt.
Introduction
Xu and co-authors teach a sight-word model to read top-down road maps. Full road sense is a base skill for smart road nets. Such sense backs sim runs that test road plans in code. Sight-word models show strong thought, yet top-view work lags still. Scant marked data and thin task scopes hold that work back. Their quiz bank, ChatBEV-QA, holds past 137,000 quiz pairs. Pairs probe whole scenes, car-lane ties, and car-car ties.
A tuned model, ChatBEV, learns to read those top maps well. The team then plugs it in a word-led road sim frame. Its scene sense and route thought give the sim firm guide marks. Path miss marks fell by up to 20.9 percent in tests. Crash rates in scenes fell by up to 37.9 percent.
Catalog record
| CREATOR | Qingyao Xu, Ya Zhang, Yanfeng Wang, Siheng Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13938
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Model Qingyao Xu, Ya Zhang, Yanfeng Wang, Siheng Chen Abstract Comprehensive traffic scene understanding is a foundational capability for Intelligent Transportation Systems (ITS) underpinning applications such as traffic simulation. While VisionLanguage Models (VLMs) have demonstrated strong reasoning potential, their application to Bird’s-Eye View (BEV) maps in traffic contexts remains limited by narrow task definitions and scarce annotated data. We introduce ChatBEV-QA, a large-scale BEV VQA benchmark of 137K+ QA pairs, designed to evaluate global scene understanding, vehicle-lane interactions, and vehiclevehicle interactions within complex traffic environments. Building on this, we fine-tune ChatBEV, a specialized VLM that accurately interprets diverse scene understanding queries from BEV maps. To demonstrate downstream utility in ITS applications, we integrate ChatBEV into a language-guided traffic simulation framework. Its global understanding and navigation reasoning provide crucial context-aware guidance, reducing trajectory displacement error by up to 20.9% and scenario collision rates by up to 37.9% over text-only baselines. Code is available at https://github.com/xuqingyao/ChatBEV . I Introduction Traffic scene understanding [ 1 , 2 ] is a fundamental task aims perceiving and interpreting the surrounding traffic environment. It serves as the cornerstone for intelligent transportation systems and autonomous driving, enabling informed decision-making and safe vehicle operation. In intelligent transportation systems, it optimizes traffic flow and prevents accidents by analyzing lane structures, traffic conditions, and vehicle interactions. For autonomous driving, it facilitates safer navigation by enhancing real-time motion planning. It also provides context-aware guidance for scene simulation, leading to more precise and controllable generation results. Bird’s-Eye View (BEV) maps, with their accurate representation of road geometry, lane boundaries, and vehicle positions, serve as a natural bridge between frontend perception and back-end ITS tasks. Given BEV’s central role, enabling semantic scene understanding of these BEV maps is a critical and timely challenge. This work thus focuses on Bird’s-Eye View (BEV)-based traffic scene understanding and its downstream application in controllable traffic simulation. Traditional traffic scene understanding methods [ 3 , 4 , 5 ] often focus on isolated objectives, lacking the holistic reasoning needed for comprehensive interpretation. Vision-Language Models (VLMs) [ 6 , 7 ] offer exceptional visual reasoning and the ability to answer free-form queries, making them a natural fit for BEV-based traffic understanding. However, their application to BEV traffic understanding remains underdeveloped, as existing benchmarks provide neither the task coverage nor the scale required to adapt VLMs to the structured geometry of BEV maps. Specifically, benchmarks like Talk2BEV [ 8 ] and MAPLM [ 9 ] are limited in scale or focus solely on object attributes, omitting the critical vehicle-lane and vehicle-vehicle interactions essential for traffic safety and conflict detection. Fig. 1: ChatBEV-QA is a scalable BEV VQA benchmark for comprehensive traffic scene understanding. Fine-tuned on ChatBEV-QA, ChatBEV provides high-level scene interpretation that directly supports downstream applications such as language-guided traffic simulation. This semantic gap has direct consequences for downstream ITS tasks. Consider traffic simulation, which is widely used for safety testing and autonomous vehicle certification: existing language-driven simulators [ 10 , 11 , 12 , 13 ] condition trajectory generation on text intent and raw HD map pixels, but they have no semantic understanding of the scene. Generated scenarios therefore exhibit lane violations, ignored right-of-way rules, and unrealistic collisions, reducing their value for safety-critical validation. To address these limitations, We introduce ChatBEV-QA , a large-scale BEV VQA benchmark designed to cover the full range of traffic-relevant scene understanding tasks. An efficient automated construction pipeline generates 137K+ QA pairs from real-world driving datasets, spanning six question types across three dimensions: global understanding including area type and lane type, which determines the macro-level traffic context that governs vehicle behavior; vehicle-lane interactions such as lane centerline and navigation information, which underpin trajectory prediction and route planning in ITS; and vehicle-vehicle interactions (presence detection and relative orientation), which are essential for traffic conflict analysis and cooperative driving. All answers are generated by carefully designed rulebased functions to ensure label accuracy. Building on ChatBEV-QA, we fine-tune ChatBEV , a specialized VLM-based scene understanding model that accurately interprets structured BEV map content directly from BEV images paired with natural language queries. ChatBEV-LLaVA-1.5 achieves the optimal balance across all six question types and generalizes well to unseen traffic datasets without additional fine-tuning. Finally, we design a novel language-guided traffic simulation framework wherein ChatBEV serves as a specialized scene understanding module. By incorporating the structured scene understanding information extracted by ChatBEV including global scene context encoding road area types and navigation guidance providing lane centerlines, our framework synthesizes more realistic and map-compliant trajectories aligned with textual intent. Experiments on nuPlan demonstrate that leveraging this semantic knowledge consistently improves trajectory realism and substantially reduces scenario collision rates. II Related Work II-A Traffic Scene Understanding Traditional methods [ 3 , 4 , 5 ] are generally task-specific, concentrating on discrete objectives like object detection, object recognition, or trajectory prediction. These approaches cannot answer free-form queries and lack the holistic reasoning needed for general applications. Recent research [ 14 , 15 , 16 , 17 , 8 , 9 ] has introduced vision-language models to traffic scenes to enable more flexible scene understanding. However, Talk2BEV [ 8 ] adapts VLMs to BEV understanding but is limited in scale and lacks vehicle-lane reasoning. MAPLM [ 9 ] focuses on road condition recognition using point clouds and language but omits vehicle interaction understanding. Our work fills this gap with a comprehensive BEV VQA benchmark and a specialized model covering all key traffic scene understanding dimensions. II-B Language-guided Traffic Simulation Realistic traffic simulation is essential for ITS safety testing and autonomous vehicle validation. While rule-based simulators [ 18 ] offer controllability, they lack behavioral realism. Learning-based methods [ 19 , 20 ] improve fidelity, and recent language-driven approaches [ 11 , 13 ] use LLMs to translate natural language into generation objectives. However, existing models often ignore high-level map semantics, such as how lane structures and area contexts should constrain trajectories. We address this by integrating ChatBEV’s structured scene understanding into the simulation pipeline to ensure map-compliant and realistic vehicle behaviors. III ChatBEV-QA: A BEV-based Traffic Scene Understanding Benchmark III-A Task Definition: Multi-Level Traffic Scene Understanding Effective traffic scene understanding from BEV maps requires reasoning at multiple levels that directly correspond to ITS operational needs. We define three complementary dimensions, each covered by two question types, so that the benchmark spans the full range of semantic queries. • Global Understanding : Traffic scene interpretation requires determining the environment type, as right-of-way norms, speed limits, and permitted maneuvers differ between intersections, highways, and parking areas. We use area type to classify this environment and lane type to identify lane function (e.g., straight, left-turn), providing the macro-level context that governs all subsequent behavioral judgments. • Vehicle-Lane Interaction : It is critical to determine whether a vehicle occupies its designated lane and which lanes are consistent with its intended maneuver. We use location to identify the lane currently occupied and navigation to identify the lanes relevant to the next maneuver, enabling lane-level compliance reasoning that is central to ADAS monitoring functions. • Vehicle-Vehicle Interaction : Assessing conflict risk and coordinating evasive responses depends on understanding the relative positions and headings of nearby vehicles. We use existence to detect