DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving
Muxi Diao, Lele Yang, Hongbo Yin, Zhexu Wang, Yejie Wang, Daxin Tian, Kongming Liang, Zhanyu Ma · 2025
Read DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving by Muxi Diao, Lele Yang, Hongbo Yin, Zhexu Wang, Yejie Wang, Daxin Tian, Kongming Liang, Zhanyu Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20665.txt.
Introduction
Muxi Diao and his seven co-authors, publishing in 2025, present DriveRX, a vision-language model trained to reason through self-driving step by step. Ordinary driving models jump straight from camera input to action and stumble in messy scenes, while earlier language-based ones handled each task in isolation. The team's training framework, AutoDriveRL, treats seeing, predicting, planning, and behaving as four linked question-and-answer tasks, each with its own scoring model giving feedback during learning.
DriveRX beats GPT-4o on behavior reasoning in a public benchmark and stays steady when scenes grow complex or inputs are corrupted. Its stage-by-stage reasoning chains agree with each other better than single-shot answers. The authors also note the chains double as high-quality labels for training smaller planning and control systems, and they release the framework and model for others to build on.
Catalog record
| CREATOR | Muxi Diao, Lele Yang, Hongbo Yin, Zhexu Wang, Yejie Wang, Daxin Tian, Kongming Liang, Zhanyu Ma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20665
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving Muxi Diao, Lele Yang, Hongbo Yin, Zhexu Wang, Yejie Wang, Daxin Tian, Kongming Liang, Zhanyu Ma Abstract Autonomous driving requires real-time, robust reasoning across perception, prediction, planning, and behavior. However, conventional end-to-end models fail to generalize in complex scenarios due to the lack of structured reasoning. While recent vision-language models (VLMs) have been applied to driving tasks, they typically rely on isolated modules and static supervision, limiting their ability to support multi-stage decision-making. We present AutoDriveRL , a unified training framework that formulates autonomous driving as a structured reasoning process over four core tasks. Each task is independently modeled as a vision-language QA problem and optimized using task-specific reward models, enabling fine-grained reinforcement signals at different reasoning stages. Within this framework, we train DriveRX , a cross-task reasoning VLM designed for real-time decision-making. DriveRX achieves strong performance on the public benchmark, outperforming GPT-4o in behavior reasoning and demonstrating robustness under complex or corrupted driving conditions. Our analysis further highlights the impact of vision encoder design and reward-guided reasoning compression. We will release the AutoDriveRL framework and the DriveRX model to support future research. 1 Introduction Autonomous driving systems operate in dynamic and uncertain traffic environments [ 1 , 2 , 3 ] , requiring robust, real-time capabilities in perception, prediction, and decision-making [ 4 , 5 , 6 ] . However, in complex scenarios such as occlusion or abnormal behaviors, conventional end-to-end models exhibit limited generalization due to the lack of structured reasoning and semantic interpretability [ 7 , 8 , 9 ] . Recent progress in vision-language models (VLMs) has extended their applications to autonomous driving tasks, including object detection [ 10 , 11 , 12 ] , trajectory prediction [ 13 , 14 , 15 ] , and planning [ 16 , 15 ] . Prior work [ 17 , 15 , 18 ] demonstrates VLMs’ potential through language prompts and QA mechanisms, yet primarily focuses on isolated task modules without a unified reasoning framework. This fragmentation hinders systematic understanding and cross-task generalization [ 19 , 18 ] . However, applying VLMs to autonomous driving remains challenging due to: (1) absence of intermediate reasoning chains [ 20 , 16 , 17 ] ; (2) isolation across perception, prediction, planning, and behavior [ 21 , 22 , 23 ] ; (3) reliance on static prompts and annotations without adaptive feedback [ 24 , 25 , 26 ] . Meanwhile, large language models (LLMs) have shown promising solutions to similar challenges [ 27 , 28 , 29 , 30 ] . OpenAI’s o1 [ 30 ] leverages chain-of-thought (CoT) prompting to generate intermediate reasoning steps, while DeepSeek’s R1-Zero [ 29 ] uses reinforcement learning (RL) to improve model behavior through dynamic feedback—without relying on supervised fine-tuning. These approaches address key limitations in current VLMs by enabling structured reasoning, unifying task semantics, and supporting feedback-driven optimization, motivating their extension to autonomous driving. Figure 1: Overview of the AutoDriveRL framework. To tackle the challenge of cross-task reasoning in autonomous driving, AutoDriveRL decomposes complex scenarios into four core tasks— Perception , Prediction , Planning and Behavior —each formulated in a VQA style. These tasks form a structured reasoning chain and are jointly optimized via RL. During training, we design task-specific reward models for each task to provide fine-grained feedback. The resulting model, DriveRX , demonstrates strong generalization and robustness under challenging driving conditions. Several multimodal studies have explored integrating reasoning mechanisms to enhance model capabilities. For example, LlamaV-o1 [ 31 ] combines Monte Carlo Tree Search (MCTS) with CoT prompting to improve logical consistency in visual question answering (VQA); VLM-R1 [ 32 ] and MM-Eureka [ 33 ] apply RL and rule-based supervision to extend reasoning into math and science domains. While effective in their domains, these approaches are not designed for the real-time, decision-critical demands of autonomous driving. Long-form reasoning chains, common in CoT-based methods, are often too slow or verbose for driving scenarios that require concise, executable outputs. AlphaDrive [ 16 ] explores the application of RL in autonomous driving, but only within the planning task, and does not address multi-task coordination. This motivates the development of a unified vision-language framework that supports structured reasoning, reinforcement learning, and multi-task coordination across core autonomous driving tasks. To this end, we propose AutoDriveRL, a multi-task training framework for VLMs in autonomous driving. AutoDriveRL combines language-driven reasoning with task-specific RL strategies, aiming to systematically improve model performance and robustness across core autonomous driving tasks. AutoDriveRL builds on prior work that decomposes autonomous driving into four core sub-tasks: perception, prediction, planning, and behavior. These tasks are organized into a sequential reasoning chain, but each task is independently modeled as a VQA problem without relying on outputs or contextual information from previous stages. Crucially, AutoDriveRL introduces task-specific reward models that provide fine-grained, stage-level reinforcement signals during training. From a global perspective, this structured supervision enables process-level feedback across reasoning steps, supports interpretable intermediate reasoning, facilitates cross-task optimization, and significantly enhances the model’s stability and overall performance in autonomous driving decision-making. Within this framework, we train a unified cross-task vision-language model, DriveRX, where “RX” denotes its focus on reasoning and cross-task generalization in autonomous driving. DriveRX consistently outperforms existing models across perception, prediction, planning, and behavior tasks, in both in-distribution and out-of-distribution (OOD) benchmark settings. On the public benchmark DriveBench [ 34 ] , DriveRX achieves a score of 62.02 in behavior, surpassing GPT-4o’s [ 27 ] 55.04 by 6.98 points. It maintains stable outputs under challenging conditions such as rain, occlusion, and lighting variation, demonstrating strong robustness and generalization. On the DriveBench-Corruption subset, DriveRX achieves 55.24, again outperforming GPT-4o (53.97). Notably, driven by the reward design in AutoDriveRL, DriveRX produces more compact reasoning traces and faster responses than existing reasoning-based models—better aligning with the real-time requirements of autonomous driving. Our main contributions are as follows: • We propose AutoDriveRL, a unified training framework that leverages semantic reward modeling to enable structured reasoning across four core autonomous driving tasks: perception, prediction, planning, and behavior. • We develop a cross-task vision-language model, DriveRX, which achieves strong performance across multiple driving tasks and complex traffic scenarios, demonstrating strong generalization and robustness, especially in behavior tasks. • We conduct an in-depth analysis of the training dynamics, identifying cross-task interactions, vision encoder bottlenecks in perception, and the impact of reward-guided reasoning compression. 2 Related Work 2.1 Vision-Language Models for Autonomous Driving Recent works have explored using vision-language models (VLMs) to enhance perception and reasoning in autonomous driving. DriveLM [ 35 ] formulates multiple driving tasks as structured graph-based QA, constructing a multi-task dataset. Dolphins [ 36 ] focuses on dialogue-based reasoning with historical control and counterfactuals. DriveVLM [ 15 ] proposes a three-stage language-driven planning pipeline with templated prompts and task-specific instructions. These methods largely rely on supervised fine-tuning (SFT) over static human-annotated data for task-specific outputs. AlphaDrive [ 16 ] introduces reinforcement learning (RL) but is limited to the planning stage, lacking support for end-to-end reasoning and broader task-level optimization. In contrast, we adopt an RL framework that decomposes driving into four semantically distinct subtasks, each equipped with a task-specific reward model to evaluate response quality and guide policy learning. This