LMAD : Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
Nan Song, Bozhou Zhang, Xiatian Zhu, Jiankang Deng, Li Zhang \corrauthor · 2025
Read LMAD : Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving by Nan Song, Bozhou Zhang, Xiatian Zhu, Jiankang Deng, Li Zhang \corrauthor (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12404.txt.
Introduction
Song and co-authors present a driving model, published in 2025, that steers a car and can explain its choices in words. Vision-language models already help cars describe scenes, which builds trust with passengers. But most systems bolt the language model onto multi-view camera images and fine-tune it on driving text. The authors say that leaves scene grasp shallow and spatial sense weak, exactly where complex driving needs strength. Their design follows modern end-to-end driving practice, with structure built for the task.
The model folds in scene interaction steps before deciding, so it weighs how road users affect each other. Its layout splits the driving job into specialized parts under one language-aware core. The result aims at actions that come with readable reasons. For self-driving to earn public trust, that pairing matters. The paper tests the design on standard driving benchmarks.
Catalog record
| CREATOR | Nan Song, Bozhou Zhang, Xiatian Zhu, Jiankang Deng, Li Zhang \corrauthor |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-12404
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LMAD : Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving Nan Song, Bozhou Zhang, Xiatian Zhu, Jiankang Deng, Li Zhang \corrauthor Abstract Large vision-language models (VLMs) have shown promising capabilities in scene understanding, enhancing the explainability of driving behaviors and interactivity with users. Existing methods primarily fine-tune VLMs on on-board multi-view images and scene reasoning text, but this approach often lacks the holistic and nuanced scene recognition and powerful spatial awareness required for autonomous driving, especially in complex situations. To address this gap, we propose a novel vision-language framework tailored for autonomous driving, called LMAD . Our framework emulates modern end-to-end driving paradigms by incorporating comprehensive scene understanding and a task-specialized structure with VLMs. In particular, we introduce preliminary scene interaction and specialized expert adapters within the same driving task structure, which better align VLMs with autonomous driving scenarios. Furthermore, our approach is designed to be fully compatible with existing VLMs while seamlessly integrating with planning-oriented driving systems. Extensive experiments on the DriveLM and nuScenes-QA datasets demonstrate that LMAD significantly boosts the performance of existing VLMs on driving reasoning tasks, setting a new standard in explainable autonomous driving. Introduction The rapid development of autonomous driving has stimulated the research across various domains, with particular emphasis on enhancing scene understanding and behavior explainability. Recently, large vision-language models (VLMs) ( Liu et al. 2024 ; Zhang et al. 2023 ; Gao et al. 2023 ; Chen et al. 2024b ) have emerged as powerful tools in bridging visual and linguistic information, showcasing considerable potential in explaining driving behaviors and improving interactions with users. Hence, the research on incorporating VLMs into autonomous driving is undergoing extensive exploration. Recent methods directly fine-tune VLMs with proposed datasets ( Wang et al. 2023 ; Nie et al. 2024 ; Sima et al. 2024 ; Tian et al. 2024 ; Huang et al. 2024 ; Xie et al. 2025 ) or employ powerful and off-the-shelf language models to construct agents ( Wen et al. 2024 ; Mao et al. 2023 ; Ma et al. 2023 ) , facilitating the steady progress of explainable autonomous driving. Figure 1: Comparison of (a) previous methods that employ standard VLMs on driving scenarios and (b) our LMAD with a task-specialized structure and the scene preliminary interaction mechanism. Drawing inspirations from autonomous driving frameworks, recent approaches use simple visual representation and intermediate results of driving system to provide scene information, and directly employ or finetune VLMs with optional Chain-of-Though (CoT) for reasoning ( Sima et al. 2024 ; Ma et al. 2023 ; Sha et al. 2023 ; Qian et al. 2025 ) , as shown in Figure 1 (a). However, directly using this information makes it challenging for the model to capture relationships between traffic elements, thereby impeding a holistic understanding of driving scenes. In addition, the VLM still struggles in localization and motion estimation, which are crucial for autonomous driving behavior analysis. During progressive reasoning process, this might cause severe cumulative errors, thus leading to suboptimal performance of existing methods in driving tasks. Addressing these limitations, we propose a novel framework LMAD tailored for autonomous driving scenarios (Figure 1 (b)), leveraging both holistic scene understanding and optimized task-specialized structure of modern end-to-end driving paradigms. We introduce a Preliminary Interaction (PI) mechanism for prior scene relationship modeling and integrate a VLM with a set of specialized expert adapters (e.g., LoRA ( Hu et al. 2022a ) ) in the same driving task structure encompassing perception, prediction and planning in sequence. The PI mechanism can provide rough relational information of traffic participants and alleviate the learning complexity of VLMs, while adapters work collaboratively to address questions of varying complexity, which is achieved by their targeted acquisition of task-specific driving knowledge from driving scenarios. In addition, LMAD also facilitates seamless incorporation of prior knowledge from an end-to-end driving system, further complementing with VLMs and enhancing subsequent reasoning. Our contributions are summarized as follows: (i) We propose leveraging the advanced design philosophy of end-to-end driving paradigms to achieve explainability in autonomous driving. (ii) We introduce LMAD, a novel VLM framework characterized with the scene preliminary interaction mechanism and a task-specific expert structure, allowing seamless incorporation of prior knowledge from an existing end-to-end driving system. (iii) Extensive experiments on the DriveLM and nuScenes-QA reasoning benchmarks demonstrate that LMAD achieves state-of-the-art performance. Related work VLMs for autonomous driving. With the emergence of GPT series ( Achiam et al. 2023 ) , LLMs have been successfully applied across various domains, yielding significant improvements due to their powerful reasoning abilities and extensive world knowledge. Recently, the field has witnessed an increased focus on VLMs, which incorporate vision encoders ( Li et al. 2023 ) and further unleash the capabilities of LLMs. Some approaches ( Zhu et al. 2023 ; Liu et al. 2024 ; Bai et al. 2023 ; Chen et al. 2024b ) leverage more advanced architectures and richer datasets to achieve superior multi-modal performance, though they often neglect the associated resource and time costs. In contrast, to strike a balance between performance and efficiency, Parameter-Efficient Fine-Tuning (PEFT) methods ( Zhang et al. 2023 ; Gao et al. 2023 ; Alayrac et al. 2022 ) tackle this challenge by introducing a limited number of learnable parameters, such as Prompt-Tuning, Adapter, and LoRA ( Hu et al. 2022a ) . In the field of autonomous driving, VLMs can function as human experts, promoting the explainability of driving behaviors and addressing challenging long-tail and zero-shot scenarios. Recently, tens of thousands of driving scenarios with language annotations have been made available ( Chen et al. 2023 ; Nie et al. 2024 ; Qian et al. 2024 ; Sima et al. 2024 ; Xie et al. 2025 ) , enabling the exploration of VLMs in autonomous driving. Some methods ( Mao et al. 2023 ; Wen et al. 2024 ; Wang et al. 2024 ; Xing et al. 2025 ) directly resort to robust GPT series to construct driver agents capable of performing driving tasks, effective but constrained by the limitations of GPT models. To overcome this challenge, latest work ( Shao et al. 2024 ; Xu et al. 2023 ; Wang et al. 2023 ) has focused on training specialized VLM models, ensuring the scalability and generalizability of these frameworks. Especially, DriveLM ( Sima et al. 2024 ) and its follow-up work ( Xie et al. 2025 ) build a distinct Graph-of-Thoughts technique that spans from perception to planning, thus improving comprehensive reasoning capabilities across various tasks in autonomous driving. Recently, DriveMM ( Huang et al. 2024 ) aggregates various driving language datasets to create a general VLM for autonomous driving. However, these fine-tuned VLMs still lack the task-specific knowledge or spatial awareness required to tackle complex scene reasoning. Additionally, several works have sought to bridge classical end-to-end frameworks ( Jiang et al. 2023 ; Hu et al. 2023 ) and language models. VLP ( Pan et al. 2024 ) and VLM-AD ( Xu et al. 2024 ) supervise end-to-end framework with auxiliary contrastive learning between the intermediate features of the driving output and text or the planning results of driving model and VLMs. Moreover, DriveVLM ( Tian et al. 2024 ) , Senna ( Jiang et al. 2024 )