A Unified Perception-Language-Action Framework for Adaptive Autonomous Driving

Yi Zhang, Erik Leo Haß, Kuo-Yi Chao, Nenad Petrovic, Yinglei Song, Chengdong Wu and Alois Knoll · 2025

Read A Unified Perception-Language-Action Framework for Adaptive Autonomous Driving by Yi Zhang, Erik Leo Haß, Kuo-Yi Chao, Nenad Petrovic, Yinglei Song, Chengdong Wu and Alois Knoll (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23540.txt.

Introduction

Yi Zhang and co-authors fuse in 2025 perception, language, and action into one framework for self driving cars. Today's systems are fragmented, which limits how well they adapt to odd scenes and explain themselves. Their design joins camera, laser, and radar sensing with a large language model core that reasons in words.

Low level sensing feeds the language core, which weighs context before the driving action is chosen. That coupling aims at human like judgment in open world mess: roadworks, gestures, and strange cargo. The paper reports the design and its tests. Language in the loop also gives passengers and auditors a readable trace of why the car acted. Trust may hinge on such traces.

Catalog record

CREATORYi Zhang, Erik Leo Haß, Kuo-Yi Chao, Nenad Petrovic, Yinglei Song, Chengdong Wu and Alois Knoll
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23540

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Unified Perception-Language-Action Framework for Adaptive Autonomous Driving Yi Zhang, Erik Leo Haß, Kuo-Yi Chao, Nenad Petrovic, Yinglei Song, Chengdong Wu and Alois Knoll Abstract Autonomous driving systems face significant challenges in achieving human-like adaptability, robustness, and interpretability in complex, open-world environments. These challenges stem from fragmented architectures, limited generalization to novel scenarios, and insufficient semantic extraction from perception. To address these limitations, we propose a unified Perception-Language-Action (PLA) framework that integrates multi-sensor fusion (cameras, LiDAR, radar) with a large language model (LLM)-augmented Vision-Language-Action (VLA) architecture, specifically a GPT-4.1-powered reasoning core. This framework unifies low-level sensory processing with high-level contextual reasoning, tightly coupling perception with natural language-based semantic understanding and decision-making to enable context-aware, explainable, and safety-bounded autonomous driving. Evaluations on an urban intersection scenario with a construction zone demonstrate superior performance in trajectory tracking, speed prediction, and adaptive planning. The results highlight the potential of language-augmented cognitive frameworks for advancing the safety, interpretability, and scalability of autonomous driving systems. I Introduction Autonomous driving systems are rapidly evolving, yet fundamental challenges persist in achieving human-like cognitive fidelity during the integration of perception, decision-making, and control under stochastic real-world conditions. Unlike humans, current systems struggle to seamlessly integrate sensory inputs with contextual cues, hindering their ability to manage uncertainties in complex, dynamic environments. This limitation restricts adaptability in partially observable settings, such as urban traffic, where comprehensive scene understanding is critical. These challenges highlight the need for integrated frameworks that emulate human cognitive adaptability, enabling robust, context-aware decision-making. A promising approach to address these challenges is multi-sensor fusion, integrating LiDAR, cameras, and radar to enhance perceptual robustness. While these technologies improve raw data acquisition, a critical gap persists in bridging perceptual inputs with contextual reasoning and actionable control outputs. Vision-language models attempt to mitigate this by grounding visual data in textual instructions, yet they often lack the situational awareness and adaptability required for complex and long-tail driving scenarios. Despite these technological advancements, autonomous driving systems continue to face limitations that hinder their ability to achieve human-like adaptability and safety in complex, dynamic environments. These primary limitations include: • Poor connectivity between functional modules – Isolated perception, language, and planning subsystems hinder cohesive scene understanding, reducing contextual coherence and safety in dynamic driving scenarios like merging or vehicle-following. • Lack of structured semantic understanding in perception data – Raw sensor data (e.g., LiDAR, images) lacks inherent meaning, requiring complex multi-stage processing (e.g., object detection, semantic segmentation, reasoning), which compromises reliability and interpretability in safety-critical decisions. • Limited generalization to unseen scenarios –Rule-based or narrowly trained models struggle with unfamiliar scenarios, such as construction zones or erratic pedestrian behavior, lacking the robust, human-like reasoning needed for diverse conditions. Fig. 1 : Detailed workflow of the proposed framework for complex scene interpretation and motion control in autonomous driving: (1) Perception-layer, (2) Language-layer, and (3) Action-layer. However, large language models (LLMs) demonstrate exceptional capabilities in reasoning, planning, and interactive understanding. So to address these challenges, we propose a unified framework that integrates multi-sensor fusion with an LLM-augmented vision-language-action (VLA) agent. This framework enables autonomous systems to combine low-level perception with high-level cognitive reasoning, achieving explainable, adaptive, and safety-bounded decision-making in open-world driving environments. Our key contributions are outlined below: • Integrated Cognitive Framework – We introduce a Perception-Language-Action (PLA) framework that tightly couples multi-modal perception with LLM-based reasoning and motion planning, enabling coherent and adaptive decision-making in complex urban environments. • Multi-Sensor Semantic Fusion – We develop a robust fusion module that combines LiDAR, radar, and camera data into structured scene descriptions, enhancing both spatial accuracy and semantic richness for downstream reasoning. • Enhanced Generalization through Contextual Reasoning – By incorporating LLM-driven reasoning, our framework improves generalization to unseen scenarios, such as construction zones or unpredictable pedestrian behaviors, enabling robust decision-making in complex environments. • Empirical Validation in Complex Urban Scenarios : We demonstrate the effectiveness and real-time adaptability of our framework through a case study on the nuScenes dataset, focusing on a challenging urban intersection with a construction zone, achieving low prediction errors and robust navigation performance. The rest of the paper is structured as follows. Section II reviews related work, discussing advances in multi-sensor fusion, vision-language models, and large language models for autonomous driving. Section III introduces the Perception-Language-Action framework, including multi-sensor fusion and vision-language-action architecture. Section IV details the experimental setup and a case study on an urban intersection scenario. Section V presents the results, and Section VI concludes with findings and future work. II Related Work The development of autonomous driving systems has spurred significant research across perception, decision-making, and control, with notable advances in multi-sensor fusion, vision-language integration, and reasoning-augmented architectures. This section surveys these efforts, emphasizing their strengths and limitations in addressing modular compartmentalization, data scarcity, and contextual rigidity. II-A Multi-Sensor Fusion for Perception Multi-sensor fusion has emerged as a cornerstone for robust perception in autonomous driving. Techniques integrating LiDAR, cameras, and radar enhance spatial and temporal scene understanding. For instance, PointPainting [ 1 ] fuses LiDAR with camera-derived semantic features for improved 3D object detection, while TransFuser [ 2 ] employs transformers to align multimodal features for trajectory prediction. Radar’s resilience to adverse weather has been leveraged in fusion frameworks like RadarNet [ 3 ] to complement vision-based systems. However, these approaches often treat perception as an isolated module, leading to potential error propagation in dynamic scenarios, such as occlusions or sudden obstacles. II-B Vision-Language Models for Scene Understanding Vision-Language Models (VLMs) have gained traction for grounding visual perception in natural language, offering a bridge between sensory inputs and decision-making. Models like CLIP [ 4 ] and ViLBERT [ 5 ] enable semantic alignment between images and text. In autonomous driving, DriveVLM [ 6 ] explores VLMs to interpret driving scenes via textual prompts and vision inputs, while VLMaps [ 7 ] employs language-guided spatial reasoning for navigation. However, these models often struggle with real-time adaptability and fine-grained situational awareness, such as anticipating pedestrian intent or interpreting ambiguous traffic signals. II-C Large Language Models in Decision-Making Large Language Models (LLMs) like GPT-4 [ 8 ] and LLaMA [ 9 ] have demonstrated potential in reasoning and planning for autonomous systems. GPT-Driver [ 10 ] utilizes in-context learning to generate explainable driving trajectories, while DriveLLM [ 11 ] aligns multimodal motion prediction with language prompts. Despite these advances, LLM-based approaches often operate in isolation from low-level perception and control loops, limiting their ability to adapt to real-time stochastic conditions such as urban intersections or adverse weather [ 12 ] . II-D Integrated Architectures for Autonomous Driving Efforts to unify perception, reasoning, and action have led to hybrid architectures that blend modular and end-to-end learning approaches. Transformer-based frameworks like UniAD [ 13 ] integrate perception, prediction, and planning into a unified pipeline, reducing error propagation while maintaining interpretability. Reason2Drive [ 14 ] and RDA-Driver [ 15 ] introduce reasoning-based decision-making but remain constrained by a lack of joint optimization with perception modules. Meanwhile, Text2motion [ 16 ] integrates neural motion planners with language interfaces, underscoring the growing need for explainability and adaptability in autonomous systems. III Methodology and Workflow In this section, we introduce a framework to tackle the challenges of poor functional module connectivity, limited generalization to unseen scenarios,

Read the complete text