Large Language Model based Interactive Decision-Making for Autonomous Driving

Xinwei DONG 1, Jiyang LI 2, Jiabin XIE 1, Yang YI 3, Tianshang JIA 1, Shiyu FANG 1, Ye TIAN 1, Peng HANG 1 · 2026

Read Large Language Model based Interactive Decision-Making for Autonomous Driving by Xinwei DONG 1, Jiyang LI 2, Jiabin XIE 1, Yang YI 3, Tianshang JIA 1, Shiyu FANG 1, Ye TIAN 1, Peng HANG 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23513.txt.

Introduction

Dong and seven colleagues build a language-model driver aid for dense mixed traffic, where timid defaults and misread intentions cause trouble. Their system models scenes in plain concepts, reads both stated and hinted intentions of nearby drivers, and picks moves that are both safe and efficient. Final choices go out as short messages to nearby road users, and simulator tests beat older baselines.

Catalog record

CREATORXinwei DONG 1, Jiyang LI 2, Jiabin XIE 1, Yang YI 3, Tianshang JIA 1, Shiyu FANG 1, Ye TIAN 1, Peng HANG 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70221
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23513

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Model based Interactive Decision-Making for Autonomous Driving Xinwei DONG 1, Jiyang LI 2, Jiabin XIE 1, Yang YI 3, Tianshang JIA 1, Shiyu FANG 1, Ye TIAN 1, Peng HANG 1 Abstract In high-conflict mixed-traffic scenarios involving human-driven and autonomous vehicles, most existing autonomous driving systems default to overly conservative behaviors, lack proactive interaction, and consequently suffer from limited public acceptance. To mitigate intent misunderstandings and decision failures, we present a Large Language Model based interactive decision-making framework that augments scene understanding and intent-aware interaction to jointly improve safety and efficiency. The approach uses Object-Process Methodology to semantically model complex multi-vehicle scenes, abstracting low-level perceptual data into objects, processes, and relations, thereby streamlining reasoning over latent causal structure. Building on this representation, the Large Language Model parses both explicit and implicit intents of surrounding agents and, under jointly enforced safety and efficiency constraints, selects candidate maneuvers. We further generate perturbed trajectory candidates via Monte Carlo sampling and evaluate them to obtain an optimized executable trajectory. To foster transparency and coordination with nearby road users, the final decision is translated by the Large Language Model into concise natural-language messages and broadcast through an external Human–Machine Interface, completing a closed loop from scene understanding to action to language. Experiments in a cluster driving simulator demonstrate that the proposed method outperforms traditional baselines across safety, comfort, and efficiency metrics, while a Turing-test-style evaluation indicates a high degree of human-likeness in decision making. Besides, these results suggest that coupling semantic scene abstraction with Large Language Model mediated intent reasoning and language-based eHMI communication offers a practical pathway toward interactive, trustworthy autonomous driving in dense mixed traffic. I Introduction In February 2020, eleven Chinese ministries, including the National Development and Reform Commission, jointly issued the Strategy for Innovation and Development of Intelligent Vehicles, which explicitly outlined that by 2025, China aims to achieve large-scale production of conditionally autonomous intelligent vehicles and regional coverage of vehicle wireless communication networks (e.g., LTE-V2X). Consequently, the penetration rate of Connected Automated Vehicles (CAVs) is expected to increase steadily (Olayode et al., 2023; Beijing Innovation Center for Intelligent Vehicles and Mobility Industry, 2022). Despite this progress, autonomous driving systems continue to exhibit insufficient decision-making intelligence in mixed human–autonomous traffic environments, particularly when confronted with complex, high-conflict, and stochastic interaction scenarios. Representative incidents—including multiple Cruise autonomous vehicles becoming immobilized at an intersection in San Francisco in 2022 and decision failures of Baidu Apollo autonomous taxis in Wuhan in 2024—highlight the vulnerability of current systems in such settings. Accident statistics further show that more than 80% of autonomous driving failures occur in complex conflict zones such as intersections and merging areas (Beijing Innovation Center for Intelligent Vehicles and Mobility Industry, 2022). These failures commonly stem from ambiguous right-of-way and dense multi-vehicle interactions, which often trigger overly conservative or unclear behaviors by autonomous vehicles. Such behaviors deviate from human driving expectations, confuse surrounding drivers, and may escalate into secondary conflicts or accidents (Pagale et al., 2024). At their core, these issues reflect the limited ability of existing autonomous systems to proactively understand and interact with human road users, particularly in risk identification and conflict anticipation stages (Ge et al., 2023). For the foreseeable future, road traffic will remain a mixed driving environment where human-driven vehicles (HDVs) and CAVs coexist. In this context, autonomous vehicles must improve not only decision optimality but also interactive intelligence, namely the capability to anticipate, negotiate, and coordinate with human drivers. Existing approaches to autonomous driving decision-making can be broadly classified into three categories: rule-based methods, game-theoretic methods, and machine learning-based methods. Rule-based approaches rely on predefined rules or handcrafted behavioral models and typically yield safe and predictable behavior in routine scenarios. A representative example is the “first-come, first-served” strategy, in which right-of-way is assigned based on arrival order. This principle has been extended into reservation-based and search-based coordination frameworks—such as Monte Carlo Tree Search, Depth-First Search, and Minimum Clique Coverage—that optimize vehicle passage by enforcing exclusive occupancy of conflict zones (Xu et al., 2019). While effective in structured environments or traffic streams dominated by CAVs, these methods struggle in mixed traffic due to the diversity and unpredictability of human driving behavior, often forcing CAVs into passive responses that undermine efficiency and interaction quality. Game-theoretic methods explicitly model traffic as a strategic multi-agent system and derive interactive decisions through equilibrium concepts, enabling a certain degree of proactive behavior. By balancing competition and cooperation, these methods allow CAVs to adjust strategies in response to human driving behaviors (Schwarting et al., 2019). Typical formulations include Stackelberg games (Rahmati et al., 2022), cooperative games (Hang et al., 2021), and hierarchical games with K-value–based right-of-way allocation (Lu et al., 2023; Ji et al., 2023). However, in real mixed traffic, limited observability and heterogeneous driving styles make it difficult to accurately model opponents’ intents and payoffs. Moreover, the computational burden of solving multi-agent games grows rapidly with the number of participants, posing significant challenges for real-time deployment. Machine learning-based methods, particularly imitation learning and reinforcement learning, which has been extensively reviewed in recent surveys on autonomous driving decision-making and planning (Hu et al., 2025), offer a data-driven alternative by learning driving policies from demonstrations or interaction experience. Imitation learning can capture latent social norms from historical data and performs well in common scenarios (Munigety, 2018), but generalizes poorly to long-tail or unfamiliar situations. Reinforcement learning improves adaptability including end-to-end deep reinforcement learning approaches (Peng et al., 2023) through trial-and-error optimization; for example, Liu et al. proposed a MADDPG-based framework with an attention mechanism to better model CAV–HDV interactions and improve intersection efficiency (Liu et al., 2025). Nevertheless, learning-based approaches still face persistent challenges in multi-task transfer, scenario generalization, and most critically interpretability, which limits trust and hampers explicit intent negotiation in highly interactive mixed-traffic scenarios. Human–machine interaction (HMI), especially external human–machine interface (eHMI), has been recognized as an important complementary mechanism for improving mutual understanding between autonomous vehicles and surrounding road users. Prior studies on both in-vehicle HMI and eHMI indicate that explicit intent communication through visual, auditory, or textual cues can reduce uncertainty, improve trust, and enhance safety in interaction-intensive scenarios (Zhang, 2020; Lee et al., 2021; Zhang et al., 2021; Gao and Martens, 2022; Faas and Baumann, 2019). Empirical investigations further confirm that well-designed eHMI signals can significantly accelerate intent inference and coordination among road users, particularly in intersection and shared-space scenarios (Brill et al., 2023). However, most existing HMI and eHMI designs remain largely signal-centric, focusing on interface presentation rather than being tightly coupled with the internal decision-making and reasoning processes of autonomous vehicles, which results in fragmented interaction rather than truly coordinated decision-making. In recent years, large language models (LLMs) have exhibited strong capabilities in semantic understanding, contextual reasoning, and intent inference, offering new opportunities to bridge perception, decision-making, and interaction in autonomous driving systems. Representative studies have explored the use of LLMs for semantic scene interpretation, intent reasoning, and explainable decision support, including language-driven behavior reasoning frameworks (Cui et al., 2024), knowledge-augmented decision support models (Hu et al., 2025), and end-to-end vision–language driving systems (Xu et al., 2024; Yang et al., 2024), collectively demonstrating the potential of LLMs to enhance interpretability and human-likeness. However, existing LLM-based approaches

Read the complete text