AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving
Lianming Huang, Haibo Hu, Yufei Cui, Jiacheng Zuo, Shangyu Wu, Nan Guan, Chun Jason Xue · 2025
Read AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving by Lianming Huang, Haibo Hu, Yufei Cui, Jiacheng Zuo, Shangyu Wu, Nan Guan, Chun Jason Xue (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05404.txt.
Introduction
Lianming Huang and co-authors teach driving vision-language models to stop thinking once they are sure. These models sharpen a self-driving car's perception, but their slow, layer-by-layer reasoning endangers real-time use, and they often keep computing long after the answer is settled. Their 2025 paper proposes AD-EE, an early-exit framework tuned to driving.
The method uses causal analysis to pick the layer where each kind of input can safely leave the network. It was tested on the Waymo data, on a corner-case set built for hard scenes, and on a real vehicle running an open driving platform. Across several models, delay fell by up to about fifty-eight percent while detection accuracy rose by up to forty-four percent. The authors argue that in driving, knowing when to stop reasoning is itself a safety feature.
Catalog record
| CREATOR | Lianming Huang, Haibo Hu, Yufei Cui, Jiacheng Zuo, Shangyu Wu, Nan Guan, Chun Jason Xue |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05404
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving Lianming Huang, Haibo Hu, Yufei Cui, Jiacheng Zuo, Shangyu Wu, Nan Guan, Chun Jason Xue Abstract With the rapid advancement of autonomous driving, deploying Vision-Language Models (VLMs) to enhance perception and decision-making has become increasingly common. However, the real-time application of VLMs is hindered by high latency and computational overhead, limiting their effectiveness in time-critical driving scenarios. This challenge is particularly evident when VLMs exhibit over-inference—continuing to process unnecessary layers even after confident predictions have been reached. To address this inefficiency, we propose AD-EE, an Early Exit framework incorporates domain characteristics of autonomous driving and leverages causal inference to identify optimal exit layers. We evaluate our method on large-scale real-world autonomous driving datasets, including Waymo and the corner-case-focused CODA, as well as on a real vehicle running the Autoware Universe platform. Extensive experiments across multiple VLMs show that our method significantly reduces latency, with maximum improvements reaching up to 57.58%, and enhances object detection accuracy, with maximum gains of up to 44%. 1 Introduction The integration of Vision-Language Models (VLMs) into autonomous driving systems has become an unstoppable trend, driven by their powerful capability to bridge visual perception and linguistic reasoning. In the industry, Li Auto’s innovative DriveVLM-Dual Tian et al. 2024 system exemplifies how VLMs can effectively assist smaller models in decision-making by providing contextual awareness and semantic understanding. In research field, recent works such as Talk2BEV Choudhary et al. 2024 leverage VLMs to strengthen BEV (Bird’s Eye View) perception, while Text-to-Drive Nguyen et al. 2024 explores VLM-based planning strategies to optimize path generation and obstacle avoidance. These advancements underscore a paradigm shift where VLMs are not merely supplementary but are actively enhancing the core perception Zhou et al. 2025 and planning Ma et al. 2024 modules of autonomous driving systems. This synergy between VLMs and smaller models is rapidly transforming both the scalability and reliability of autonomous driving technologies Zhou et al. 2024 . Particularly in corner case scenarios, as demonstrated in Figure 1 , VLMs significantly enhance detection and perception capabilities by capturing intricate contextual cues that smaller models often miss. This highlights their indispensable role in resolving complex urban scenes and improving the safety and robustness of autonomous systems. Figure 1: The Sparse R-CNN model, trained on Waymo, detects vehicles and e-bikes but misses animals and occluded objects. In contrast, Deepseek_VL2-small accurately identifies both, demonstrating robustness in complex urban settings. Despite the remarkable capabilities of Vision-Language Models (VLMs), significant challenges remain when applying them to real-time autonomous driving scenarios. For instance, while VLMs have been employed to enhance BEV perception generation, discussions on the latency issues of VLMs are often overlooked Choudhary et al. 2024 . Similarly, VLMs have been integrated into high real-time driving modules like MPC, yet the problem of high latency remains unresolved Long et al. 2024 . The latest state-of-the-art models, such as DeepSeek Wu et al. 2024 and LLaVA Liu et al. 2023 , demonstrate impressive capabilities but also still substantial latency and computational overhead due to their inherent complexity and large-scale architectures. This undoubtedly hinders their application in autonomous driving. In real-world driving scenarios, lower latency directly translates to more time for the vehicle to respond and handle events, contributing to safer driving Liu et al. 2025 . Therefore, accelerating the inference speed of VLMs without compromising their comprehensive perception and reasoning capabilities is crucial for advancing autonomous driving technologies. Addressing this challenge calls for innovative solutions to enhance VLM inference efficiency while maintaining robust perception and reasoning performance . Figure 2: Over Inference Recent studies show that LLMs often exhibit over-inference, a challenge that extends to VLMs Sui et al. 2025 ; Chen et al. 2024 . For instance, DeepSeek_VL2-small achieves correct predictions as early as Layer 18, making further processing redundant, as show in Fig 2 , Moreover, since DeepSeek is a Chinese-based foundational model, the Chinese term for "vehicle" appears earlier. In autonomous driving, where object recognition is relatively straightforward, early exit mechanisms can effectively reduce latency by terminating inference at intermediate layers once thresholds are met Huang et al. 2024 ; Zhao et al. 2024 . This approach mitigates over-inference, enhancing efficiency without altering the architecture. Despite its promise, early exit remains underexplored in real-time VLM applications for autonomous driving. Building on our research findings, we propose AD-EE, an integrate an Early Exit mechanism into VLMs, specifically optimized for autonomous driving tasks with detailed analysis and validation. To achieve this, we leverage causal inference techniques to identify and quantify the optimal exit layers, maximizing both efficiency and accuracy across diverse driving scenarios. Unlike conventional early exit strategies, our approach not only determines the most effective early exit layers but also provides robust analysis of their impact on perception and decision-making. Our primary evaluation focuses on 2D detection, a critical perception task for avoiding traffic accidents and guiding vehicle planning Liang et al. 2024 ; Jin et al. 2024 . This task is often challenged by complex edge cases Li et al. 2022 , where enhanced VLM reasoning proves highly beneficial Chen et al. 2025 . Our contributions are threefold: • We integrate the Early Exit mechanism with VLM-based perception tasks in autonomous driving, achieving not only enhanced inference speed but also surprisingly improved accuracy by mitigating "over-inference". • We propose a Causal Inference-based approach to identify and analyze the optimal early exit layers for enhanced VLM inference. • We conduct extensive experiments on real autonomous vehicles, benchmarking against traditional full-layer inference. Our results demonstrate significant gains in both efficiency and accuracy, substantiating the practical value of AD-EE enhanced VLMs in real driving conditions. 2 Related Work 2.1 Vision-Language Models in Autonomous Driving The integration of Vision-Language Models (VLMs) in autonomous driving has garnered increasing attention for their ability to bridge visual perception and linguistic understanding. Recent advancements, such as DriveVLM-Dual Tian et al. 2024 , have demonstrated how VLMs can assist smaller models in decision-making by enhancing contextual awareness and semantic interpretation. Furthermore, Talk2BEV Choudhary et al. 2024 extends VLM capabilities to Bird’s Eye View (BEV) perception, enabling richer environmental understanding through multi-modal learning. Text-to-Drive Nguyen et al. 2024 further explores VLM-driven planning strategies, showcasing how natural language instructions can dynamically adjust vehicle navigation and obstacle avoidance. Despite these advancements, deploying VLMs in real-time driving remains challenging due to the substantial computational cost and inference latency inherent in large-scale models. This latency bottleneck severely constrains their application in safety-critical tasks such as dynamic object detection and path planning, where timely responses are imperative. 2.2 Causal Inference Causal inference is a key technique for understanding cause-and-effect relationships in observational and experimental data, widely applied in fields like healthcare, finance, and public policy Glass et al. 2013 ; Atanasov and Black 2016 ; Imbens and Rubin 2015 . It primarily relies on two frameworks: the Structural Causal Model (SCM), which uses directed acyclic graphs (DAGs) to map causal relationships, and the Potential Outcome Framework, which focuses on treatment effects for counterfactual analysis Pearl 2009 ; Winship and Morgan 1999 . Recent advancements in Large Language Models (LLMs) have enabled new applications of causal inference, especially in handling unstructured, high-dimensional data Feder et al. 2022 ; Kiciman