Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition

Tauseef Ahmed 1 2 3 , Tao Sun 1 , Jeronimo Castrillon 3 4 , Kanishkan Vadivel 2 , Guangzhi Tang 1 · 2026

Read Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition by Tauseef Ahmed 1 2 3 , Tao Sun 1 , Jeronimo Castrillon 3 4 , Kanishkan Vadivel 2 , Guangzhi Tang 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01135.txt.

Introduction

Ahmed and co-authors make a speech recognizer lean with spiking and event driven design in this 2026 paper. Deep speech models work well on phones and home devices. Their compute and energy costs strain such small hardware. Neuromorphic design promises relief: sparse, spike based computation in place of dense math. No study had weighed the hardware gains of the rival strategies for speech.

Their base is SpeechMamba, a top speech model. An event driven variant with a special activation passes 60 percent activation sparsity. It loses less than one percent accuracy on LibriSpeech. A spiking variant passes 70 percent sparsity with 30 percent fewer parameters than peer spiking nets. A cycle accurate simulator lets them explore design and hardware together. It finds bottlenecks and adds over ten percent more efficiency. The paper maps which neuromorphic route pays for edge speech.

Catalog record

CREATORTauseef Ahmed 1 2 3 , Tao Sun 1 , Jeronimo Castrillon 3 4 , Kanishkan Vadivel 2 , Guangzhi Tang 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01135

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition Tauseef Ahmed 1 2 3 , Tao Sun 1 , Jeronimo Castrillon 3 4 , Kanishkan Vadivel 2 , Guangzhi Tang 1 Abstract Deep learning has greatly advanced automatic speech recognition (ASR), enabling widespread deployment on edge devices such as smartphones and smart home systems. However, the computational and energy demands of deep neural networks pose significant challenges for such resource-constrained deployments, introducing latency and limiting real-time interaction. Neuromorphic computing offers a promising solution by introducing activation sparsity through spiking neural networks (SNNs) and event-driven neural networks, converting dense operations into sparse computations. However, a study that evaluates the hardware benefits of different neuromorphic strategies remains lacking for ASR. This paper explores spiking and event-driven neuromorphic neural networks to improve activation sparsity in the state-of-the-art SpeechMamba model for ASR. We introduce an event-driven SpeechMamba with FATReLU activation, achieving over 60% activation sparsity with less than 1% accuracy degradation on LibriSpeech. We also propose a spiking SpeechMamba that attains over 70% sparsity while using 30% fewer parameters than comparable SNNs. Finally, we develop a cycle-accurate event-driven simulator enabling flexible algorithm-hardware co-exploration, which helps us identify computational bottlenecks and yields over 10% additional efficiency improvements. ©2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. I Introduction Deep learning has greatly advanced the performance of automatic speech recognition (ASR) in recent years [ 10 , 7 ] , enabling its widespread adoption in real-world applications. ASR is now increasingly deployed on edge devices [ 28 ] , such as smartphones and smart home devices, where it supports real-time interaction with users. However, the extensive computation required by deep neural networks (DNNs) poses significant challenges for such deployments. The high computational and energy demands of dense matrix multiplication in DNNs limit the applicability of high-performance models on edge devices with strict power budgets, and introduce latency that disrupts natural user interaction [ 24 ] . Therefore, there is a need for computationally efficient solutions for ASR. Digital neuromorphic computing offers a promising solution to address these efficiency challenges by introducing activation sparsity through spiking neural networks (SNNs) [ 6 ] or event-based neural networks [ 29 ] . Unlike conventional DNNs that perform dense matrix multiplications at every layer, neuromorphic approaches convert these operations into sparse computations, where only active neurons contribute to the forward pass. This strategy has been successfully scaled to complex tasks in computer vision [ 27 ] and language processing [ 12 ] . In the speech domain, neuromorphic approaches have demonstrated success in simpler tasks such as keyword spotting [ 31 ] and denoising [ 23 ] , and recent work has explored large-vocabulary ASR using SNNs for acoustic modeling [ 22 ] . However, a study that evaluates the realistic hardware benefits of different neuromorphic computing strategies, including spiking and event-driven approaches, for complex ASR remains absent. Furthermore, existing neuromorphic works in speech processing primarily report efficiency using algorithmic metrics, such as synaptic operations and theoretical energy estimates [ 31 , 22 ] . These metrics assume ideal sparse computation and do not capture the true costs of execution on real digital neuromorphic hardware, including irregular sparsity that is difficult to exploit efficiently [ 15 ] and memory access patterns with significant data movement overhead [ 24 ] . While existing neuromorphic hardware captures these costs, they impose constraints that limit algorithmic exploration. For example, Loihi [ 5 ] supports only specific neuron models and computation paradigms. This disconnect between abstract metrics and constrained hardware hinders effective algorithm-hardware co-exploration. Algorithmic metrics obscure true hardware bottlenecks, while platform-specific constraints limit the freedom to iterate on network designs. This paper explores different spike-based and event-driven neuromorphic strategies to improve activation sparsity in the state-of-the-art SpeechMamba model [ 7 ] for complex ASR 1 1 1 https://github.com/ERNIS-LAB/speech-asr-neuromorphic-mamba . Our contributions are as follows: • We introduce an event-driven SpeechMamba with FATReLU activation function [ 13 ] . To maximize activation sparsity while minimizing accuracy degradation, we propose a multi-stage training pipeline. The resulting model achieves an average activation sparsity over 60% with minimal accuracy loss ( < 1%) on the LibriSpeech dataset. • We propose a spiking SpeechMamba using binary spikes for computation. With sparsity-aware training, the spiking network achieves over 70% activation sparsity and delivers competitive performance compared to state-of-the-art SNNs while using 30% fewer parameters. • We develop a neuromorphic simulator for cycle-accurate event-driven dataflow execution on a RISC-V Ibex core [ 16 ] . Unlike existing simulators that focus on analytical modeling or coarse event-driven scheduling, our approach provides hardware-aware yet flexible simulation for neuromorphic workloads. Using this simulator, we identify computational bottlenecks in event-driven SpeechMamba and improve its efficiency by over 10%. II Background and Related Work II-A Deep Learning Advances for Speech Recognition Transformer-based architectures have significantly advanced speech recognition by effectively modeling long-range dependencies in sequential data [ 10 ] . However, self-attention mechanisms lack explicit state compression and incur quadratic computational complexity with respect to sequence length [ 1 ] . In contrast, state space models (SSMs) [ 9 ] address these limitations by maintaining compact, input-dependent latent states, enabling more efficient long-context modeling. Recent studies demonstrate that SSM-based speech models can achieve lower word error rates with fewer parameters and reduced computational cost compared to transformer-only architectures [ 14 ] . SpeechMamba [ 7 ] introduced a speech recognition architecture that integrates Mamba blocks with self-attention. The model adopts an encoder-decoder architecture, which consists of multiple repeated units. Within each unit, two Mamba blocks are interleaved with self-attention layers. While the Mamba blocks capture long-range temporal dependencies, the self-attention layers model lower-level temporal representations. Owing to its compact architecture and strong speech recognition performance, we select SpeechMamba as the foundational architecture supporting our proposed approaches. II-B Attention-based Spiking Neural Networks Spikformer [ 34 ] and SpikMamba [ 4 ] are recent spiking neural architectures that combine the efficiency of SNNs with attention-based sequence modeling. Spikformer introduces Spiking Self Attention (SSA), where Query, Key, and Value are represented as spikes and softmax is replaced with sparse, multiplication-free operations. SpikMamba incorporates spiking Mamba blocks and a spiking window-based linear attention mechanism to capture both global and local temporal dependencies in event-based data. In our work, we adopt SSA for the spiking attention module and leverage spiking Mamba blocks as the foundation of our spiking SpeechMamba. II-C Event-driven Digital Neuromorphic Processing Event-driven digital neuromorphic processing exploits activation sparsity by computing and communicating only non-zero activations. When activations fall below a threshold, downstream computations and memory accesses are skipped entirely, where cost scales with active neurons rather than total network size [ 29 ] . This paradigm originates from spiking neural networks, where neurons emit binary spikes only when their membrane potential exceeds a firing threshold, naturally inducing high sparsity. Recent work has generalized beyond binary spikes to graded activations, retaining the computational benefits of event-driven processing while

Read the complete text