DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving

Muxi Diao, Lele Yang, Hongbo Yin, Zhexu Wang, Yejie Wang, Daxin Tian, Kongming Liang, Zhanyu Ma · 2025

Read DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving by Muxi Diao, Lele Yang, Hongbo Yin, Zhexu Wang, Yejie Wang, Daxin Tian, Kongming Liang, Zhanyu Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20665.txt.

Introduction

Muxi Diao and his seven co-authors, publishing in 2025, present DriveRX, a vision-language model trained to reason through self-driving step by step. Ordinary driving models jump straight from camera input to action and stumble in messy scenes, while earlier language-based ones handled each task in isolation. The team's training framework, AutoDriveRL, treats seeing, predicting, planning, and behaving as four linked question-and-answer tasks, each with its own scoring model giving feedback during learning.

DriveRX beats GPT-4o on behavior reasoning in a public benchmark and stays steady when scenes grow complex or inputs are corrupted. Its stage-by-stage reasoning chains agree with each other better than single-shot answers. The authors also note the chains double as high-quality labels for training smaller planning and control systems, and they release the framework and model for others to build on.

Catalog record

CREATORMuxi Diao, Lele Yang, Hongbo Yin, Zhexu Wang, Yejie Wang, Daxin Tian, Kongming Liang, Zhanyu Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20665

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving Muxi Diao, Lele Yang, Hongbo Yin, Zhexu Wang, Yejie Wang, Daxin Tian, Kongming Liang, Zhanyu Ma Abstract Autonomous driving requires real-time, robust reasoning across perception, prediction, planning, and behavior. However, conventional end-to-end models fail to generalize in complex scenarios due to the lack of structured reasoning. While recent vision-language models (VLMs) have been applied to driving tasks, they typically rely on isolated modules and static supervision, limiting their ability to support multi-stage decision-making. We present AutoDriveRL , a unified training framework that formulates autonomous driving as a structured reasoning process over four core tasks. Each task is independently modeled as a vision-language QA problem and optimized using task-specific reward models, enabling fine-grained reinforcement signals at different reasoning stages. Within this framework, we train DriveRX , a cross-task reasoning VLM designed for real-time decision-making. DriveRX achieves strong performance on the public benchmark, outperforming GPT-4o in behavior reasoning and demonstrating robustness under complex or corrupted driving conditions. Our analysis further highlights the impact of vision encoder design and reward-guided reasoning compression. We will release the AutoDriveRL framework and the DriveRX model to support future research. 1 Introduction Autonomous driving systems operate in dynamic and uncertain traffic environments [ 1 , 2 , 3 ] , requiring robust, real-time capabilities in perception, prediction, and decision-making [ 4 , 5 , 6 ] . However, in complex scenarios such as occlusion or abnormal behaviors, conventional end-to-end models exhibit limited generalization due to the lack of structured reasoning and semantic interpretability [ 7 , 8 , 9 ] . Recent progress in vision-language models (VLMs) has extended their applications to autonomous driving tasks, including object detection [ 10 , 11 , 12 ] , trajectory prediction [ 13 , 14 , 15 ] , and planning [ 16 , 15 ] . Prior work [ 17 , 15 , 18 ] demonstrates VLMs’ potential through language prompts and QA mechanisms, yet primarily focuses on isolated task modules without a unified reasoning framework. This fragmentation hinders systematic understanding and cross-task generalization [ 19 , 18 ] . However, applying VLMs to autonomous driving remains challenging due to: (1) absence of intermediate reasoning chains [ 20 , 16 , 17 ] ; (2) isolation across perception, prediction, planning, and behavior [ 21 , 22 , 23 ] ; (3) reliance on static prompts and annotations without adaptive feedback [ 24 , 25 , 26 ] . Meanwhile, large language models (LLMs) have shown promising solutions to similar challenges [ 27 , 28 , 29 , 30 ] . OpenAI’s o1 [ 30 ] leverages chain-of-thought (CoT) prompting to generate intermediate reasoning steps, while DeepSeek’s R1-Zero [ 29 ] uses reinforcement learning (RL) to improve model behavior through dynamic feedback—without relying on supervised fine-tuning. These approaches address key limitations in current VLMs by enabling structured reasoning, unifying task semantics, and supporting feedback-driven optimization, motivating their extension to autonomous driving. Figure 1: Overview of the AutoDriveRL framework. To tackle the challenge of cross-task reasoning in autonomous driving, AutoDriveRL decomposes complex scenarios into four core tasks— Perception , Prediction , Planning and Behavior —each formulated in a VQA style. These tasks form a structured reasoning chain and are jointly optimized via RL. During training, we design task-specific reward models for each task to provide fine-grained feedback. The resulting model, DriveRX , demonstrates strong generalization and robustness under challenging driving conditions. Several multimodal studies have explored integrating reasoning mechanisms to enhance model capabilities. For example, LlamaV-o1 [ 31 ] combines Monte Carlo Tree Search (MCTS) with CoT prompting to improve logical consistency in visual question answering (VQA); VLM-R1 [ 32 ] and MM-Eureka [ 33 ] apply RL and rule-based supervision to extend reasoning into math and science domains. While effective in their domains, these approaches are not designed for the real-time, decision-critical demands of autonomous driving. Long-form reasoning chains, common in CoT-based methods, are often too slow or verbose for driving scenarios that require concise, executable outputs. AlphaDrive [ 16 ] explores the application of RL in autonomous driving, but only within the planning task, and does not address multi-task coordination. This motivates the development of a unified vision-language framework that supports structured reasoning, reinforcement learning, and multi-task coordination across core autonomous driving tasks. To this end, we propose AutoDriveRL, a multi-task training framework for VLMs in autonomous driving. AutoDriveRL combines language-driven reasoning with task-specific RL strategies, aiming to systematically improve model performance and robustness across core autonomous driving tasks. AutoDriveRL builds on prior work that decomposes autonomous driving into four core sub-tasks: perception, prediction, planning, and behavior. These tasks are organized into a sequential reasoning chain, but each task is independently modeled as a VQA problem without relying on outputs or contextual information from previous stages. Crucially, AutoDriveRL introduces task-specific reward models that provide fine-grained, stage-level reinforcement signals during training. From a global perspective, this structured supervision enables process-level feedback across reasoning steps, supports interpretable intermediate reasoning, facilitates cross-task optimization, and significantly enhances the model’s stability and overall performance in autonomous driving decision-making. Within this framework, we train a unified cross-task vision-language model, DriveRX, where “RX” denotes its focus on reasoning and cross-task generalization in autonomous driving. DriveRX consistently outperforms existing models across perception, prediction, planning, and behavior tasks, in both in-distribution and out-of-distribution (OOD) benchmark settings. On the public benchmark DriveBench [ 34 ] , DriveRX achieves a score of 62.02 in behavior, surpassing GPT-4o’s [ 27 ] 55.04 by 6.98 points. It maintains stable outputs under challenging conditions such as rain, occlusion, and lighting variation, demonstrating strong robustness and generalization. On the DriveBench-Corruption subset, DriveRX achieves 55.24, again outperforming GPT-4o (53.97). Notably, driven by the reward design in AutoDriveRL, DriveRX produces more compact reasoning traces and faster responses than existing reasoning-based models—better aligning with the real-time requirements of autonomous driving. Our main contributions are as follows: • We propose AutoDriveRL, a unified training framework that leverages semantic reward modeling to enable structured reasoning across four core autonomous driving tasks: perception, prediction, planning, and behavior. • We develop a cross-task vision-language model, DriveRX, which achieves strong performance across multiple driving tasks and complex traffic scenarios, demonstrating strong generalization and robustness, especially in behavior tasks. • We conduct an in-depth analysis of the training dynamics, identifying cross-task interactions, vision encoder bottlenecks in perception, and the impact of reward-guided reasoning compression. 2 Related Work 2.1 Vision-Language Models for Autonomous Driving Recent works have explored using vision-language models (VLMs) to enhance perception and reasoning in autonomous driving. DriveLM [ 35 ] formulates multiple driving tasks as structured graph-based QA, constructing a multi-task dataset. Dolphins [ 36 ] focuses on dialogue-based reasoning with historical control and counterfactuals. DriveVLM [ 15 ] proposes a three-stage language-driven planning pipeline with templated prompts and task-specific instructions. These methods largely rely on supervised fine-tuning (SFT) over static human-annotated data for task-specific outputs. AlphaDrive [ 16 ] introduces reinforcement learning (RL) but is limited to the planning stage, lacking support for end-to-end reasoning and broader task-level optimization. In contrast, we adopt an RL framework that decomposes driving into four semantically distinct subtasks, each equipped with a task-specific reward model to evaluate response quality and guide policy learning. This

Read the complete text