DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Driving
Anqing Jiang, Yu Gao, Zhigang Sun, Yiru Wang, Jijun Wang, Jinhao Chai, Qian Cao, Yuweng Heng, Hao Jiang, YunDa Dong, Zongzheng Zhang, Xianda Guo, Hao Sun, Hao Zhao · 2025
Read DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Driving by Anqing Jiang, Yu Gao, Zhigang Sun, Yiru Wang, Jijun Wang, Jinhao Chai, Qian Cao, Yuweng Heng, Hao Jiang, YunDa Dong, Zongzheng Zhang, Xianda Guo, Hao Sun, Hao Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19381.txt.
Introduction
Anqing Jiang and co-authors guide diffusion planning for self-driving with vision-language models in a 2025 paper. End-to-end driving stacks promise joint optimization of perception and planning. Costly bird's-eye computation and weak decisions in complex scenes limit them. Action diversity also suffers in prior designs.
Their system, DiffVLA, uses a hybrid sparse-dense diffusion policy. Sparse diffusion represents multi-modal driving behavior efficiently. A vision-language model supplies high-level decision guidance. Deep interaction among agents, map elements, and model output steers trajectories. The design rethinks how language model decisions should guide motion. A 2025 grand challenge with reactive synthetic scenes tested it. The method posted a leading score in that competition. Real and synthetic scenarios were both included. The authors claim superior performance over prior end-to-end entries. Planning quality in messy scenes is the headline gain.
Catalog record
| CREATOR | Anqing Jiang, Yu Gao, Zhigang Sun, Yiru Wang, Jijun Wang, Jinhao Chai, Qian Cao, Yuweng Heng, Hao Jiang, YunDa Dong, Zongzheng Zhang, Xianda Guo, Hao Sun, Hao Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19381
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Driving Anqing Jiang, Yu Gao, Zhigang Sun, Yiru Wang, Jijun Wang, Jinhao Chai, Qian Cao, Yuweng Heng, Hao Jiang, YunDa Dong, Zongzheng Zhang, Xianda Guo, Hao Sun, Hao Zhao Abstract Research interest in end-to-end autonomous driving has surged owing to its fully differentiable design integrating modular tasks, i.e. perception, prediction and planing, which enables optimization in pursuit of the ultimate goal. Despite the great potential of the end-to-end paradigm, existing methods suffer from several aspects including expensive BEV (bird’s eye view) computation, action diversity, and sub-optimal decision in complex real-world scenarios. To address these challenges, we propose a novel hybrid sparse-dense diffusion policy, empowered by a Vision-Language Model (VLM), called Diff-VLA. We explore the sparse diffusion representation for efficient multi-modal driving behavior. Moreover, we rethink the effectiveness of VLM driving decision and improve the trajectory generation guidance through deep interaction across agent, map instances and VLM output. Our method shows superior performance in Autonomous Grand Challenge 2025 which contains challenging real and reactive synthetic scenarios. Our methods achieves 45.0 PDMS. 1 Introduction End-to-end autonomous driving has emerged as a significant and rapidly growing research area. With the abundance of available human driving demonstrations, there is considerable potential to learn human-like driving policies from large-scale datasets. Methods such as UniAD hu2023planning , VAD jiang2023vad take sensor data as input and regress a single-mode trajectory within one fully optimizable model. sun2024sparsedrive further explores the sparse presentation and propose a symmetric sparse perception module and a parallel motion planner.However, these approaches overlook the intrinsic uncertainty and the multi-modal characteristics of driving behavior. Leveraging powerful diffusion concepts in generation, methods chitta2022transfuser ; zheng2025diffusion are capable of modeling multi-mode action distributions. Through proper denoising process, liao2024diffusiondrive further speeds up diffusion process via an anchored Gaussian distribution design. jiang2024senna ; zhou2025opendrivevla connects VLM to an end-to-end model to improve the trajectory planning accuracy. Although existing methods are trained and evaluated on well-known benchmarks such as nuScenes, NAVSIMv1, nuPlan, achieving robust performance in a close-loop manner and going beyond recorded states remains a open challenge. In this report, we revisit the concepts of sparsity, diffusion, VLM and propose a more comprehensive approach with proven performance in closed-loop assessment. Our framework consists of three key components: a VLM guidance module, a sparse-dense hybrid perception module, and a diffusion-based planning module. 1. VLA Guidance Module : The module takes multi-view images as input and outputs both a trajectory and high-level driving commands. These commands are then combined with the external driving command (e.g. navigation instructions) to serve as input for the diffusion-based planning module. 2. Hybrid Perception Module : Our hybrid perception model have two branch for different perception tasks. The dense perception branch constructs a dense Bird’s Eye View (BEV) feature representation, which is fed into the planning module as a primary input. To improve the planner’s understanding of obstacles and road structure, the sparse perception branch extracts information at the instance level (e.g., detected obstacles, lane boundaries, centerlines, stop lines, etc.) and propagates it to the planning module. 3. Diffusion-Based Planning Module : We use truncated diffusion policy that utilizes multi-modal anchors as priors and employs a shortened diffusion schedule. To further improvement the diffusion model performance, we propose a hierarchical information encoding strategy to integrate heterogeneous inputs. We train and evaluate our approach on navsim-v2, which provides a comprehensive closed-loop assessment of robustness and generalization via introducing reactive background traffic participants and realistic synthetic multi-view camera images. Leveraging these techniques, our solution achieves 45.0 EPDMS on the private test set in navsim v2 competition. Figure 1 : The architecture of our proposed perception enhanced diffusion VLA framework. 2 Perception Agents and map information are critical for representing a traffic scene. Our design philosophy aims to maximize the representation of both explicit and implicit features. In our current solution, we introduce two parallel perception modules: a sparse perception module and a dense perception module. The sparse perception module employs the sampling strategy from Lin2023Sparse4D and jiang2025sparsemextunlockingpotentialsparse for 3D object detection and online map generation. In contrast, the dense module utilizes the BEV feature projection method from huang2021bevdet to produce a BEV feature space. The sparse module outputs 3D bounding boxes and map vectors, while the dense module generates BEV feature vectors. Both outputs are integrated into a subsequent trajectory head. The purpose of these dual modules is to simultaneously leverage implicit features of agents and the environment alongside explicit object and map information, overcoming the limitations of using only projection-based or sampling-based methodologies for constructing the BEV feature space. Explicit 3D bounding boxes are formatted as standard object boxes, containing pose, size, heading angle, and velocity. Map vectors are represented as 20 map points per element. Both explicit object and map-related information are encoded using a MLP to generate explicit object and map embeddings. For the implicit branch, the BEV grid size is set to 128\times 128 , covering a perception range of 64\times 64 meters along the x and y directions in the ego coordinate space. We aggregate features from 30 agents and one ego vehicle to provide implicit guidance for the subsequent trajectory diffusion process. Additionally, the explicit outputs of objects and maps enable the planner to perform collision detection and drivable area checks, enhancing the planning process beyond feature-based trajectory selection alone. The training process for the perception modules is divided into two stages. The sparse branch is trained using detection losses for 3D objects and map elements. Subsequently, the dense branch is trained alongside the trajectory head, following the completion of sparse branch training. All perception branches utilize the VoV-99 lee2020centermask backbone. 3 VLM To enable effective processing and fusion of multimodal information in autonomous driving scenarios, we propose the VLM Command Guidance module. This module is built upon the Senna-VLM framework jiang2024senna , which leverages a multi-image encoding strategy and multi-view prompting mechanisms to achieve efficient and comprehensive scene understanding. The Senna-VLM architecture comprises four primary components: Vision Encoder, Driving Vision Adaptor, Text Encoder, and Large Language Model (LLM). The Vision Encoder processes multi-view image sequences I\in\mathbb{R}^{N_{\mathrm{img}}\times H\times W\times 3} from Navsim dauner2024navsim as input and extracts image features. These features are further encoded and compressed by the Driving Vision Adapter, producing image tokens E_{\mathrm{img}}\in\mathbb{R}^{N_{\mathrm{img}}\times M_{\mathrm{img}}\times C} , where N_{\mathrm{img}} is the number of images, M_{\mathrm{img}} is the number of image tokens per image, C is the feature dimension of the LLM, H is the height of the image and W is the width of the image. The Text Encoder encodes user instructions and navigation commands into text tokens E_{\mathrm{txt}}\in\mathbb{R}^{M_{\mathrm{txt}}\times C} , with M_{\mathrm{txt}} signifying the number of text tokens. Both the image and text tokens are then fed into a Large Language Model (LLM), which generates high-level driving decisions. In our implementation, the Vision Encoder adopts ViT-L/14 from CLIP radford2021learning and the LLM is Vicuna-v1.5-7B zheng2023judging . We adhere to the standard Senna-VLM configuration by processing images from all onboard camera sensors. The VLM Command Guidance module produces high-level planning decisions, which are decomposed into lateral control (e.g., lane changes and turning) and longitudinal control (e.g., acceleration and braking). These decisions are encoded via a one-hot encoding mechanism and subsequently integrated with external driving