SafeAuto : Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models
Jiawei Zhang, Xuan Yang, Taiqi Wang, Yu Yao, Aleksandr Petiushko, Bo Li · 2025
Read SafeAuto : Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models by Jiawei Zhang, Xuan Yang, Taiqi Wang, Yu Yao, Aleksandr Petiushko, Bo Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00211.txt.
Introduction
Jiawei Zhang and co-authors add layers of safety knowledge to model-driven self-driving, in this 2025 paper. Driving models that reason in words often fumble exact control values and traffic rules. Their framework attacks both faults.
A new loss sharpens control numbers written as text, weighted by digit place. Traffic rules are turned into logic statements and checked inside a probability network before an action passes. A memory search over past drives, with video and signals, adds lessons from experience. Across several data sets the full stack beat prior systems on accuracy and safety. The authors release the code for others to test.
Catalog record
| CREATOR | Jiawei Zhang, Xuan Yang, Taiqi Wang, Yu Yao, Aleksandr Petiushko, Bo Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00211
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SafeAuto : Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models Jiawei Zhang, Xuan Yang, Taiqi Wang, Yu Yao, Aleksandr Petiushko, Bo Li Abstract Traditional autonomous driving systems often struggle to connect high-level reasoning with low-level control, leading to suboptimal and sometimes unsafe behaviors. Recent advances in multimodal large language models (MLLMs), which process both visual and textual data, offer an opportunity to unify perception and reasoning. However, effectively embedding precise safety knowledge into MLLMs for autonomous driving remains a significant challenge. To address this, we propose SafeAuto , a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge. First, we introduce a Position-Dependent Cross-Entropy (PDCE) loss to improve low-level control signal predictions when values are represented as text. Second, to explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic (e.g., “red light ⟹ \implies stop”) and embeds them into a probabilistic graphical model (e.g., Markov Logic Network) to verify predicted actions using recognized environmental attributes. Additionally, our Multimodal Retrieval-Augmented Generation (RAG) model leverages video, control signals, and environmental attributes to learn from past driving experiences. Integrating PDCE, MLN, and Multimodal RAG, SafeAuto outperforms existing baselines across multiple datasets, enabling more accurate, reliable, and safer autonomous driving. The code is available at https://github.com/AI-secure/SafeAuto . † † affiliationnotice: \icmlintern , \icmluiuc , \icmlatnuro \dagger \ddagger \lx@sectionsign \lx@sectionsign \lx@sectionsign 1 Introduction Autonomous Driving (AD) systems ( Kim et al., 2018 ; Jin et al., 2023 ; Hu et al., 2023 ; Jiang et al., 2023 ; Zhang et al., 2024b ) have made significant strides in recent years, yet they often rely on separate modules for high-level decision-making (e.g., “the car should slow to a stop”) and low-level control signal prediction (e.g., providing the specific speed or steering angle for the next few moments). However, these two aspects are inherently correlated, as high-level actions directly guide low-level control signals. This modular design often overlooks this correlation, leading to inefficiencies and less cohesive driving behaviors. Recent advancements in Multimodal Large Language Models (MLLMs) ( Liu et al., 2023b ; Liu et al., 2023a ; Lin et al., 2023 ) offer a promising avenue to bridge high-level reasoning and low-level control in autonomous driving (AD), which provide a unified framework capable of processing and reasoning over multiple data modalities, such as images, videos, and text. Some recent works ( Wang et al., 2023 ; Xu et al., 2024 ; Wang et al., 2024 ) have begun to leverage MLLMs to generate both high-level action descriptions and low-level control signals in an end-to-end manner. However, they are predominantly data-driven and often fail to perform at human levels due to several limitations. Firstly, for low-level action prediction, current approaches in adapting MLLMs generally follow two fashions. The first fashion treats the prediction of float numbers as text generation ( Gruver et al., 2024 ; Xu et al., 2024 ) , directly training the MLLM using cross-entropy (CE) loss for token prediction. Some variations ( Brohan et al., 2023 ; Sima et al., 2023 ) of this method involve tokenizing the prediction range into several bins and adding new tokens for each bin into the LLM’s vocabulary, allowing the model to predict the corresponding bin token ID. However, these methods remain somewhat coarse compared to traditional regression techniques ( Hu et al., 2023 ) using Mean Squared Error (MSE) loss. Alternatively, another fashion ( Jin et al., 2024b ) employs a linear layer to decode the float number from the output hidden embeddings of the MLLM, enabling the use of MSE loss to train the model. While this approach may improve numerical accuracy, it compromises the autoregressive capability of the LLM, as the model can then only be purely used for numerical prediction and cannot perform any further QA-for example, handling high-level question-answering. Additionally, regarding high-level action prediction, a significant limitation of current methods is their inability to effectively utilize both structured and unstructured knowledge when making decisions. Specifically, existing approaches often focus solely on data-driven techniques, inadequately incorporating structured knowledge such as traffic rules and safety constraints. Although some methods ( Sima et al., 2023 ; Mao et al., 2023 ; Wang et al., 2024 ) attempt to include traffic regulations by embedding them into the model’s context, this implicit approach is insufficient. Due to the inherent tendency of MLLMs to hallucinate, they may still generate unsafe or illegal actions. Meanwhile, while RAG ( Lewis et al., 2020 ) has been employed in language models ( Semnani et al., 2023 ; Zhang et al., 2024a ) to mitigate issues like hallucination by incorporating relevant information from external sources, Yuan et al. (2024) first propose to combine the rich multimodal data inherent in autonomous driving contexts—such as videos, images, and control signals—to learn from past driving experiences as unstructured knowledge. Figure 1: Overview of our SafeAuto pipeline for end-to-end high-level and low-level prediction in autonomous driving, featuring: (1) the Position-Dependent Cross-Entropy Loss ( Section 3.1 ) for improved low-level numerical predictions using soft, weighted digit probability distributions; (2) Knowledge-Enhanced Post-Safety Verification ( Section 3.2 ) with Markov Logic Networks to verify high-level actions against traffic rules; and (3) a Multimodal RAG ( Section 3.3 ) training method that incorporates similar driving experiences via text-based rankings for better context-aware decision-making. To address these challenges, we propose a novel framework SafeAuto that enhances MLLMs for autonomous driving through three key contributions as shown in Figure 1 : (1) Position-Dependent Cross-Entropy (PDCE) Loss: We propose a PDCE loss that retains the autoregressive nature of the MLLM while behaving like an MSE loss during training. This loss function improves numerical prediction accuracy without compromising the model’s language generation abilities. (2) Knowledge-Enhanced Post-Safety Verification: We employ Markov Logic Networks (MLNs) ( Richardson & Domingos, 2006 ) to explicitly encode domain knowledge and structured traffic rules into the decision-making process of the MLLM. This knowledge-enabled reasoning allows us to verify and correct the high-level actions suggested by the MLLM, ensuring they comply with traffic regulations and safety constraints. (3) Multimodal RAG for Autonomous Driving: We introduce a method that utilizes video data, control signals, and the environmental predicates used in the MLN to retrieve similar driving experiences. By learning a joint embedding across these modalities based on the ranking derived from text description of the current scenario—which contain rich semantic information—we can effectively leverage past experiences to inform current decision-making. By integrating these components, SafeAuto provides a comprehensive solution to the challenges faced by current MLLMs in autonomous driving. We evaluate our approach on two benchmark datasets: BDD-X ( Kim et al., 2018 ) and DriveLM ( Sima et al., 2023 ) , both featuring low-level control signals and high-level action descriptions. Our experimental results demonstrate significant improvements in both low-level control accuracy and high-level action prediction. First, for low-level prediction on the BDD-X dataset, it reduces the Root Mean Square Error (RMSE) for speed and course predictions by an additional 5.8% and 14.1% over the state-of-the-art (SOTA) baselines, respectively. Furthermore, on the DriveLM dataset, it decreases the Average Displacement Error (ADE) for motion prediction by 44.4%. Second, for high-level prediction on the BDD-X dataset, our method boosts high-level action performance beyond the