PRAM-R: A Perception-Reasoning-Action-Memory Framework with LLM-Guided Modality Routing for Adaptive Autonomous Driving

Yi Zhang, Xian Zhang, Saisi Zhao, Yinglei Song, Chengdong Wu, Nenad Petrovic and Alois Knoll · 2026

Read PRAM-R: A Perception-Reasoning-Action-Memory Framework with LLM-Guided Modality Routing for Adaptive Autonomous Driving by Yi Zhang, Xian Zhang, Saisi Zhao, Yinglei Song, Chengdong Wu, Nenad Petrovic and Alois Knoll (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04222.txt.

Introduction

Zhang and others present PRAM-R, a perception-thought-action-memory method using LLM-guided routing of modalities for adaptive autonomous driving. Multimodal perception supports robust autonomous driving yet wastes computation when all sensors stay active. An LLM router picks and weights modalities from environmental context and sensor diagnostics, while hierarchical memory keeps time consistency and supports long-term adaptation. A two-stage tests uses synthetic stress tests for balance analysis and real-world checks on the nuScenes dataset.

Catalog record

CREATORYi Zhang, Xian Zhang, Saisi Zhao, Yinglei Song, Chengdong Wu, Nenad Petrovic and Alois Knoll
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31850
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.04222

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PRAM-R: A Perception-Reasoning-Action-Memory Framework with LLM-Guided Modality Routing for Adaptive Autonomous Driving Yi Zhang, Xian Zhang, Saisi Zhao, Yinglei Song, Chengdong Wu, Nenad Petrovic and Alois Knoll Abstract Multimodal perception enables robust autonomous driving but incurs unnecessary computational cost when all sensors remain active. This paper presents PRAM-R, a unified Perception–Reasoning–Action–Memory framework with LLM-Guided Modality Routing for adaptive autonomous driving. PRAM-R adopts an asynchronous dual-loop design: a fast reactive loop for perception and control, and a slow deliberative loop for reasoning-driven modality selection and memory updates. An LLM router selects and weights modalities using environmental context and sensor diagnostics, while a hierarchical memory module preserves temporal consistency and supports long-term adaptation. We conduct a two-stage evaluation: (1) synthetic stress tests for stability analysis and (2) real-world validation on the nuScenes dataset. Synthetic stress tests confirm 87.2% reduction in routing oscillations via hysteresis-based stabilization. Real-world validation on nuScenes shows 6.22% modality reduction with 20% memory recall while maintaining comparable trajectory accuracy to full-modality baselines in complex urban scenarios. Our work demonstrates that LLM-augmented architectures with hierarchical memory achieve efficient, adaptive multimodal perception in autonomous driving. I Introduction Fig. 1 : Perception-Reasoning-Action-Memory Framework with modality routing for complex scene interpretation and motion control in autonomous driving: (1) Perception-layer, (2) Language-layer, (3) Action-layer, and (4) Memory-Layer. Autonomous driving relies on multiple modalities (camera, LiDAR, radar and map) to perceive complex environments. While multimodal fusion enhances generalization and robustness in perception [ 8 , 28 ] , it introduces efficiency challenges: continuously activating all modalities incurs unnecessary computational overhead and latency, with diminishing accuracy gains when redundant or low-quality modalities are included [ 14 ] . Activating all sensors simultaneously is impractical under limited computing resources and real-time constraints. Different scenarios require distinct modality combinations. For example, cameras suffice for highway lane keeping in clear weather, while urban intersections at night or in rain benefit from radar. Thus, adaptively selecting the most informative and reliable modalities is essential for maintaining perception robustness and optimizing computation, energy, and sensor longevity. These factors motivate an intelligent modality routing mechanism that dynamically activates sensors according to scene context and system state. Yet, existing approaches rely on heuristic rules or attention mechanisms that lack high-level reasoning capabilities, and most fusion frameworks either employ fixed modality selection strategies or are tailored to specific driving scenarios [ 19 , 6 , 11 ] , making them difficult to generalize and often unable to balance perception accuracy and computational efficiency. Recent advances in large language models (LLMs) have enabled LLM-guided perception [ 22 ] , extending even to modality routing. However, applying LLMs to real-time driving faces two key challenges: First, their sequential, computation-heavy reasoning introduces latency when invoked per frame; Second, the absence of memory prevents reusing past routing experiences, leading to redundant reasoning and poor long-term adaptation. To address these challenges, following the framework described in [ 32 ] , we propose PRAM-R , a Perception–Reasoning–Action–Memory framework with LLM-Guided Modality Routing for adaptive autonomous driving. Specifically, large language model serves as a modality router , selecting and weighting modalities based on scene context and sensor health. Our contributions are summarized as follows: • We propose PRAM-R , a novel Perception-Reasoning-Action-Memory framework that integrates LLM-guided modality routing into the autonomous driving pipeline for adaptive and efficient multimodal perception. • We introduce a reasoning-driven routing mechanism that dynamically selects and weights sensor modalities through integrated analysis of sensor diagnostics and environmental context, optimizing the trade-off between computational efficiency and perceptual reliability. • We design an asynchronous dual-loop architecture decoupling fast reactive control from deliberative reasoning adaptation, achieving low-latency real-time operation while maintaining robust performance across diverse driving scenarios. • We develop a hierarchical memory module that caches routing decisions to reduce inference overhead, enforces temporal consistency across frames, tracks evolving sensor reliability patterns, and preserves contextual knowledge for continual environmental adaptation. II Related Work II-A Multimodal Perception and Fusion Multimodal fusion has been widely explored for autonomous driving, combining cameras, LiDAR, radar, and HD maps for robust perception. Early fusion methods integrate features at the input or BEV level, while late fusion aggregates detection results from individual modalities. Representative architectures include BEVFusion [ 13 ] , TransFuser [ 4 ] , and MVFNet [ 27 ] , which achieve high detection accuracy by leveraging complementary sensing modalities. However, these approaches assume that all sensors are always reliable and active, leading to redundant computation and degraded performance under sensor failures or adverse conditions. II-B Modality Routing and Sensor Diagnostics Recent research has shifted from static fusion to adaptive, reliability-aware routing that dynamically selects or weights sensor modalities. Methods such as AdaFusion [ 12 ] and DynMM [ 30 ] learn modality gating through attention or uncertainty cues, while SamFusion [ 18 ] adapts fusion weights to adverse weather. In parallel, quantitative diagnostics have been developed to assess sensor health using LiDAR density and echo ratio [ 7 ] , radar cross section [ 23 ] , and vision indicators like brightness and contrast [ 21 ] . These studies provide useful priors for evaluating reliability but largely rely on heuristics and lack reasoning-driven decision integration. II-C LLM and Memory-Augmented Driving Agents Large language and vision-language models have recently been explored for driving-scene reasoning and decision making. Works such as DriveGPT4 [ 29 ] and OmniDrive [ 25 ] employ generative reasoning to describe scenes and propose high-level actions. Meanwhile, temporal-memory agents such as DriveDreamer [ 26 ] , DriveLM [ 24 ] , and Wayformer [ 17 ] capture long-term spatio-temporal dependencies for trajectory prediction and planning. These advances highlight the potential of integrating language-based reasoning and memory, yet current systems operate on fixed sensor inputs and do not adapt sensing configurations according to environmental or diagnostic feedback. However, none of the above approaches jointly integrate LLM-driven modality routing with hierarchical memory for long-term adaptation, which motivates our PRAM-R. III Methodology III-A Framework Overview To enable adaptive autonomous driving and address the challenges of redundant computation as well as inconsistent multimodal reliability under diverse and uncertain conditions, we propose PRAM-R , a unified Perception–Reasoning–Action–Memory framework with LLM-Guided (Qwen3-VL-8B) Modality Routing . As shown in Fig. 1 , the architecture integrates four tightly coupled modules (Perception, Reasoning, Action, and Memory) with an LLM-based Modality Router embedded in the perception layer. The reasoning and action layers retain the designs in [ 32 ] , while the perception layer is enhanced with LLM-guided modality routing and a new fusion strategy for adaptive multimodal perception. In addition, a Memory module is newly introduced to enable contextual adaptation and maintain long-term consistency across time-steps. PRAM-R is designed as an asynchronous dual-loop architecture that integrates a fast reactive loop for real-time perception and control with a slow deliberative loop for modality routing, reasoning, and memory updates. The reactive loop operates at high frequency to ensure immediate responsiveness, while the deliberative loop runs asynchronously at a lower frequency to refine sensing strategies based on accumulated context. Both loops involve memory interaction to maintain temporal consistency and adaptive behavior. At each reactive timestep t , the framework receives the current multimodal sensory streams \mathcal{S}_{t}=\{\mathcal{S}_{t}^{\text{cam}},\mathcal{S}_{t}^{\text{lidar}},\mathcal{S}_{t}^{\text{radar}}\} , together with map data \mathcal{M}_{t}^{\text{map}} , prompts \mathcal{P}_{t} and external information \mathcal{E}_{t} (e.g., traffic signals,

Read the complete text