DiffVLA++: Bridging Cognitive Reasoning and End-to-End Driving through Metric-Guided Alignment

Yu Gao, Anqing Jiang, Yiru Wang, Wang Jijun, Hao Jiang, Zhigang Sun, Heng Yuwen, [1pt] Wang Shuo, Hao Zhao, Hao Sun, [2pt] RIX, Bosch · 2025

Read DiffVLA++: Bridging Cognitive Reasoning and End-to-End Driving through Metric-Guided Alignment by Yu Gao, Anqing Jiang, Yiru Wang, Wang Jijun, Hao Jiang, Zhigang Sun, Heng Yuwen, [1pt] Wang Shuo, Hao Zhao, Hao Sun, [2pt] RIX, Bosch (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17148.txt.

Introduction

Yu Gao and nine co-authors blend two styles of self-driving control in this 2025 paper. One style plans safe paths well but fails on rare road scenes. The other uses broad world sense but can plan paths cars cannot drive. The team wants the strengths of both in one system.

Their system builds sense-based paths and a dense set of safe path picks. A score tool then lines up the two sets by road measures. The blend keeps world sense while paths stay fit to drive. In a 2025 grand challenge test, the system scored well on the board. The work shows sense and safe planning can aid each other.

Catalog record

CREATORYu Gao, Anqing Jiang, Yiru Wang, Wang Jijun, Hao Jiang, Zhigang Sun, Heng Yuwen, [1pt] Wang Shuo, Hao Zhao, Hao Sun, [2pt] RIX, Bosch
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17148

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiffVLA++: Bridging Cognitive Reasoning and End-to-End Driving through Metric-Guided Alignment Yu Gao, Anqing Jiang, Yiru Wang, Wang Jijun, Hao Jiang, Zhigang Sun, Heng Yuwen, [1pt] Wang Shuo, Hao Zhao, Hao Sun, [2pt] RIX, Bosch Abstract Conventional end-to-end (E2E) driving models are effective at generating physically plausible trajectories, but often fail to generalize to long-tail scenarios due to the lack of essential world knowledge to understand and reason about surrounding environments. In contrast, Vision-Language-Action (VLA) models leverage world knowledge to handle challenging cases, but their limited 3D reasoning capability can lead to physically infeasible actions. In this work we introduce DiffVLA++ , an enhanced autonomous driving framework that explicitly bridges cognitive reasoning and E2E planning through metric-guided alignment. First, we build a VLA module directly generating semantically grounded driving trajectories. Second, we design an E2E module with a dense trajectory vocabulary that ensures physical feasibility. Third, and most critically, we introduce a metric-guided trajectory scorer that guides and aligns the outputs of the VLA and E2E modules, thereby integrating their complementary strengths. The experiment on the ICCV 2025 Autonomous Grand Challenge leaderboard shows that our model achieves EPDMS of 49.12. Figure 1 : Overview of the DiffVLA++ architecture. It consists of three main components: (i) a VLA module, (ii) a conventional E2E module, both capable of directly generating planning trajectories, and (iii) a trajectory scorer that serves as a metric-guided aligner to unify their outputs. 1 Introduction End-to-end (E2E) autonomous driving frameworks have achieved remarkable progress in recent years by directly mapping multi-modal sensory inputs to control signals or driving trajectories [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ] . These models benefit from powerful spatio-temporal representations and can generate physically plausible driving behaviors under normal conditions. However, they often struggle in long-tail or unseen scenarios due to their limited capability in high-level scene understanding and semantic reasoning [ 14 , 15 , 16 , 17 ] . DiffVLA [ 18 ] attempts to enhance the reliability of spatio-temporal reasoning for both static and dynamic objects by integrating dense and sparse Bird’s Eye View (BEV) perception streams. Nevertheless, its limitation arises from the heavy reliance on structured pattern-recognition modules rather than human-level cognitive modeling, which leaves it lacking the generalizable world knowledge that human drivers naturally possess. In DiffVLA++, we retain only the dense BEV branch and aim to address this issue by incorporating cognitive knowledge to enhance reasoning ability. To further tackle this limitation, recent studies have explored the integration of Large Language Models (LLMs) and Vision-Language-Action (VLA) architectures into autonomous driving systems. Some approaches [ 19 , 20 , 21 ] leverage the extensive world knowledge encoded in LLMs to generate high-level driving decisions, while others [ 22 , 23 , 24 , 25 , 26 ] directly produce driving trajectories, forming Vision-Language-Action (VLA) models. In DiffVLA, we adopt the former paradigm for its simplicity to integration. In DiffVLA++, we further exploit the VLA framework to generate semantically rich and directly executable driving trajectories. This raises a key challenge: while E2E models excel at grounded trajectory prediction and VLA models excel at cognitive reasoning with world knowledge, a principled way to bridge these two paradigms has not been fully explored. The difficulty lies in combining semantically rich yet occasionally infeasible VLA trajectories with physically plausible but semantically limited E2E trajectories. As shown in Fig. 1 , we propose DiffVLA++ , a framework that explicitly bridges reasoning and planning through a metric-guided alignment mechanism . We systematically compare VLA and E2E models on the NavsimV2 benchmark [ 27 ] and integrate their complementary strengths via the following components: • VLA Module : A fully integrated and differentiable VLA model that generates semantically grounded trajectories with explicit 3D reasoning. • E2E Module : A dense BEV-based E2E model with a transformer-based trajectory head, ensuring physical feasibility. • Metric-Guided Alignment : An MLP-based trajectory scorer that shares the BEV feature space with the E2E module, regressing rule-based metrics such as No At-Fault Collisions (NC), Drivable Area Compliance (DAC), and aligning the trajectories from both VLA and E2E to unify their strengths. Evaluations on the ICCV 2025 Autonomous Grand Challenge benchmark show DiffVLA++ achieves Extended Predictive Driver Model Score (EPDMS) [ 27 ] of 49.12. 2 Fully-Differentiable VLA Model for E2E Driving The Vision-Language-Action (VLA) framework is designed around four core modules that jointly enable multimodal reasoning and trajectory generation. The visual processing stream employs a CLIP-based Vision Transformer (ViT-L/14) that encodes multi-view images into a compact set of visual tokens. Each input image is resized and partitioned into patches, which are embedded into high-dimensional representations capturing spatial context. These tokens are then adapted through a Driving Vision Adapter that performs compression and projection, ensuring compatibility with the downstream language model. In parallel, the linguistic stream encodes navigation commands and high-level driving instructions. A pretrained tokenizer first converts the input into subword units, which are then embedded and processed by a transformer-based text encoder. This produces text tokens that capture both semantic intent and syntactic dependencies. The two modalities are integrated within a large language model (Vicuna-v1.5-7B), which performs multimodal fusion through a shared embedding space. Visual tokens are concatenated with text tokens, and the model employs causal attention to preserve autoregressive text generation while allowing cross-modal interactions. This design enables the model to jointly reason about visual observations and driving instructions in a unified space. Finally, the last layer of the LLM projects the fused hidden states into continuous future trajectories of the ego vehicle. Instead of discretizing actions, the model directly predicts waypoints over a four second horizon, where each waypoint includes lateral position, longitudinal position, and heading angle. This fully differentiable design avoids discretization errors, improves smoothness, and allows end-to-end optimization of the driving policy. To align with the E2E model (Sec. 3 ) and the metric-guided trajectory scorer (Sec. 4 ), features along the VLA-generated trajectory are sampled from the BEV feature map and fed into the trajectory scorer. 3 Conventional End-to-End Driving Model The conventional end-to-end (E2E) driving module in our framework operates on a dense BEV representation generated by BevFormer [ 28 ] . We adopt VoVNet-99 [ 29 ] as the image backbone to enhance the visual representational capacity. The BEV feature map is discretized into a 128\times 128 grid, covering a spatial extent of 64\times 64 meters along the ego-vehicle’s x and y axes. Following the multi-task architecture of Transfuser [ 2 ] , the E2E module comprises three prediction heads: (i) an agent detection head that regresses the states of dynamic agents, (ii) a semantic segmentation head that predicts scene semantics in BEV space, and (iii) a trajectory planning head that generates the ego vehicle’s future motion. The agent detection head employs a set of learnable agent queries \mathbf{Q}_{\text{agent}}\in\mathbb{R}^{N\times d} with N=32 . Each query interacts with the BEV features via deformable cross-attention to produce agent-centric embeddings \mathbf{F}_{a}\in\mathbb{R}^{N\times d} . These embeddings are decoded into bounding box parameters (x,y,w,h,\theta) , representing the center coordinates, width, length, and heading of each detected agent. Together with the semantic segmentation output, this ensures that the BEV representation encodes

Read the complete text