New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehicles

Aref Ghoreishee, Abhishek Mishra, Lifeng Zhou, John Walsh, Nagarajan Kandasamy · 2025

Read New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehicles by Aref Ghoreishee, Abhishek Mishra, Lifeng Zhou, John Walsh, Nagarajan Kandasamy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01882.txt.

Introduction

Self-driving cars must fuse camera images, laser scans, and heading data into split-second decisions, and the computing budget on board is tight. A 2025 paper by Aref Ghoreishee and four co-authors proposes a new spiking architecture for this multi-modal decision-making, built around reinforcement learning from end to end.

Their perception module uses cross-attention transformers to merge the different sensor streams. Transformers are powerful but costly, which limits their use on small edge hardware in vehicles. Spiking networks, which compute with brief pulses like real neurons, promise a cheaper route. The work tests whether that promise can meet the demands of driving decisions.

Catalog record

CREATORAref Ghoreishee, Abhishek Mishra, Lifeng Zhou, John Walsh, Nagarajan Kandasamy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01882

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehicles Aref Ghoreishee, Abhishek Mishra, Lifeng Zhou, John Walsh, Nagarajan Kandasamy Abstract This work proposes an end-to-end multi-modal reinforcement learning framework for high-level decision-making in autonomous vehicles. The framework integrates heterogeneous sensory input, including camera images, LiDAR point clouds, and vehicle heading information, through a cross-attention transformer-based perception module. Although transformers have become the backbone of modern multi-modal architectures, their high computational cost limits their deployment in resource-constrained edge environments. To overcome this challenge, we propose a spiking temporal-aware transformer-like architecture that uses ternary spiking neurons for computationally efficient multi-modal fusion. Comprehensive evaluations across multiple tasks in the Highway Environment demonstrate the effectiveness and efficiency of the proposed approach for real-time autonomous decision-making. K eywords Multi-Modal Reinforcement Learning, Autonomous Driving Systems, Cross-Attention Transformer Networks 1 Introduction Perception is the cornerstone of modern autonomous agents, transforming raw sensory data into representations suitable for high-level decision making. In real-world scenarios, agents must process heterogeneous data streams, such as camera images, LiDAR point clouds, and IMU signals, and fuse them into a coherent and task-relevant understanding of the environment [ 25 , 32 , 47 ] . Recent work has explored mechanisms for cross-attention [ 41 , 51 ] , contrastive learning [ 58 ] , and representation alignment [ 27 ] to maximize the informativeness and reliability of such fused representations. Among these techniques, cross-attention has emerged as the backbone of modern perception systems, allowing the effective integration of complementary modalities for complex tasks such as autonomous navigation, where the accuracy of decisions is critically dependent on perception quality [ 41 , 51 , 19 , 45 ] . Reinforcement learning (RL) allows agents to learn complex decision-making strategies in dynamic and uncertain environments through direct interaction with the world. Deep reinforcement learning ( DRL ), which integrates deep neural networks to process high-dimensional sensory observations with RL approaches, has achieved impressive results in autonomous driving and robotic navigation [ 60 , 17 , 23 , 55 ] . Since decision-making performance is highly dependent on perception quality, integrating attention-based fusion modules into end-to-end DRL frameworks has become increasingly essential [ 8 ] . Compared to traditional pipelines with separate, hand-engineered perception and control stages, end-to-end learning jointly optimizes both components, enabling the development of robust, task-specific representations and policies directly from raw sensor inputs [ 6 , 36 ] . However, the high computational demands of DRL with attention-based fusion remain a major barrier to the deployment of such systems in autonomous agents with limited hardware resources [ 24 , 48 ] . Spiking Neural Networks ( SNNs ) offer a promising alternative by providing an event-driven, computationally efficient paradigm inspired by biological neurons [ 4 , 52 , 29 , 38 ] . SNNs transmit information through discrete spikes, where sparse neuronal activity and the dominance of addition operations over costly multiplications lead to highly efficient computation [ 42 , 49 ] . These properties make SNNs attractive for resource-limited autonomous systems, but integrating attention mechanisms into spiking architectures introduces new challenges. Attention mechanisms, such as the one proposed in the transformer architecture [ 50 ] , calculate similarity scores using the dot products of the query-key pairs followed by a softmax normalization, which requires multiplications and exponentiations. These operations are incompatible with the multiplication-free, event-driven design principles of SNNs and incur significant computational costs [ 59 ] . To address this, recent work has proposed spiking self-attention ( SSA ) mechanisms, where Query, Key, and Value vectors are encoded as binary or a combination of binary and ternary spikes, enabling attention maps to be computed without softmax normalization [ 59 , 13 ] . This approach substantially reduces computational complexity and has shown promising results in image classification tasks. Since the self-attention mechanism can be adapted for cross-attention by assigning the Query, Key, and Value to different modalities, a similar approach can be applied in the spiking domain. However, existing SSA models disregard temporal dependencies between spike-encoded signals, even though spike encoding naturally introduces a temporal dimension into the system. To overcome this limitation, we propose a temporal-aware spiking cross-attention mechanism based on ternary spiking neurons that explicitly captures spatial dependencies while maintaining temporal dependencies, computational efficiency, and enhancing representational capacity. Building on this module, we design an end-to-end Q-learning architecture for autonomous driving in which the perception stage fuses multiple modalities using event-driven multiplication-free attention. Combined with SNNs, the proposed architectures achieve competitive decision-making performance with significantly reduced computational cost. To the best of our knowledge, this is the first work to integrate spiking cross-attention-based fusion into end-to-end reinforcement learning for autonomous driving. The main contributions of this work are summarized as follows: • End-to-end multi-modal Q-learning network architecture (MM-DQN) , a cross-attention-based fusion framework that integrates bird’s-eye view (BEV) images, and LiDAR and IMU data for autonomous driving decision-making. • Temporal-aware ternary spiking attention (TTSA) , a new attention mechanism that explicitly maintains temporal and sign dependencies when calculating the attention map, thus addressing a key limitation in prior SSA models. • Comprehensive experimental validations demonstrate that multi-modal reinforcement learning achieves a better performance with fewer successive observations compared to single-modal RL. Moreover, while earlier spiking attention mechanisms exhibit a performance gap compared to non-spiking models, our proposed temporal-aware spiking attention substantially reduces this gap. The corresponding computational cost is also quantified. The remainder of the paper is organized as follows. Section 2 discusses related work, and Section 3 introduces the basic concepts underlying Deep-Q and spiking neural networks. Section 4 develops the non-spiking MM-DQN architecture, while Section 5 develops the corresponding spiking version using the TTSA mechanism. Section 6 evaluates the performance of the various architectures. We conclude the paper in Section 7 . 2 Related Work We discuss related work in the area of multimodal RL and the use of SNNs to develop the attention mechanisms required to fuse multiple modalities of information. 2.1 MultiSensor and Multimodal RL Chen et al. propose an RL framework for multisensor agents (vision, touch, proprioception) to maximize the shared information between modalities [ 7 ] . Noting that naive multisensor RL often over-relies on a single modality or learns inconsistent representations when inputs are noisy or missing, the authors introduce a latent state-space model trained with a mutual information objective that explicitly aligns embeddings across modalities. In robotics, Becker et al. tackle the challenge of learning effective state representations from heterogeneous inputs (high-dimensional visual data and low-dimensional signals) by applying distinct self-supervised losses to each modality, respecting their differing information content [ 1 ] . Jangir et al. develop a transformer-based RL framework that fuses a static third-person camera with a robot-mounted egocentric camera using cross-view attention, allowing features from one view to query and fuse with the other before feeding into a soft-actor-critical policy [ 26 ] . Su et al. propose an algorithm for autonomous driving, building a spatio-temporal graph of nearby vehicles and combining it with camera images through cross-attention [ 46 ] . Finally, Lu et al. develop a deep RL system for adaptive scheduling in global logistics that integrates heterogeneous IoT sensor data (RFID, GPS, and environmental sensors) with external context (such as traffic, weather) through

Read the complete text