SpikeMLLM: Spike-based Multimodal Large Language Models via Modality-Specific Temporal Scales and Temporal Compression
Han Xu 1,2,3,† Zhiyong Qin 1,2,† Di Shang 1,2 Jiahong Zhang 1,2 Xuerui Qiu 1,2,4 Bo Lei 3 Tiejun Huang 3,5 Bo Xu 1,2∗ Guoqi Li 1,2,6,7∗ · 2026
Read SpikeMLLM: Spike-based Multimodal Large Language Models via Modality-Specific Temporal Scales and Temporal Compression by Han Xu 1,2,3,† Zhiyong Qin 1,2,† Di Shang 1,2 Jiahong Zhang 1,2 Xuerui Qiu 1,2,4 Bo Lei 3 Tiejun Huang 3,5 Bo Xu 1,2∗ Guoqi Li 1,2,6,7∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18610.txt.
Introduction
Xu and eight colleagues offer SpikeMLLM, the first spike-based frame for multimodal language models, where spike nets save power but mixed inputs resist a shared code. Their frame folds ANN quantization into spike form, adds per-type time scales, and squeezes LIF units to cut steps from linear to log scale. Four test models keep near-full scores under step cuts, with gaps of 0.72 percent and about 1 percent on the FP16 base. A chip for the spike path gives about 9 times the throughput and about 26 times the power efficiency.
Catalog record
| CREATOR | Han Xu 1,2,3,† Zhiyong Qin 1,2,† Di Shang 1,2 Jiahong Zhang 1,2 Xuerui Qiu 1,2,4 Bo Lei 3 Tiejun Huang 3,5 Bo Xu 1,2∗ Guoqi Li 1,2,6,7∗ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73119 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18610
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpikeMLLM: Spike-based Multimodal Large Language Models via Modality-Specific Temporal Scales and Temporal Compression Han Xu 1,2,3,† Zhiyong Qin 1,2,† Di Shang 1,2 Jiahong Zhang 1,2 Xuerui Qiu 1,2,4 Bo Lei 3 Tiejun Huang 3,5 Bo Xu 1,2∗ Guoqi Li 1,2,6,7∗ Abstract . Multimodal Large Language Models (MLLMs) have achieved remarkable progress but incur substantial computational overhead and energy consumption during inference, limiting deployment in resource-constrained environments. Spiking Neural Networks (SNNs), with their sparse event-driven computation, offer inherent energy efficiency advantages on neuromorphic hardware, yet extending them to MLLMs faces two key challenges: heterogeneous modalities make uniform spike encoding insufficient, and high-resolution image inputs amplify timestep unfolding overhead. We propose SpikeMLLM , the first spike-based framework for MLLMs, which unifies existing ANN quantization methods in the spiking representation space and incorporates Modality-Specific Temporal Scales (MSTS) guided by Modality Evolution Discrepancy (MED) and Temporally Compressed LIF (TC-LIF) for timestep compression from T = L − 1 T=L-1 to T = log 2 ( L ) − 1 T=\log_{2}(L)-1 . Experiments on four representative MLLMs across diverse multimodal benchmarks show that SpikeMLLM maintains near-lossless performance under aggressive timestep compression ( T v / T t = 3 / 4 T_{v}/T_{t}=3/4 ), with average gaps of only 0.72 % 0.72\% and 1.19 % 1.19\% relative to the FP16 baseline on InternVL2-8B and Qwen2VL-72B. We further develop a dedicated RTL accelerator tailored to the spike-driven datapath, observing 9.06 × 9.06\times higher throughput and 25.8 × 25.8\times better power efficiency relative to an FP16 GPU baseline under a deployment-oriented co-design setting, suggesting the promise of algorithm–hardware co-design for efficient multimodal intelligence. 1. Introduction Multimodal Large Language Models (MLLMs) have achieved remarkable progress in multimodal understanding and reasoning, emerging as a core component of modern AI systems ( 63 ; 50 ; 6 ; 62 ; 2 ) . However, their inference typically incurs substantial computational overhead and energy consumption, posing significant challenges in resource-constrained environments and limiting their potential as intelligent infrastructure. Spiking Neural Networks (SNNs) transmit information via discrete spikes, transforming dense multiplications into a sparse event-driven paradigm ( 57 ; 35 ; 9 ; 42 ) , which demonstrates significant energy efficiency advantages when combined with neuromorphic hardware ( 12 ; 43 ; 23 ; 39 ) . Moreover, biological neural systems are known to exhibit temporal processing mechanisms, where different sensory modalities often operate at distinct temporal scales ( 47 ; 40 ) . SNNs inherently possess temporal dynamics, making them mechanistically well-suited for multimodal temporal processing, thus providing a promising paradigm for energy-efficient multimodal intelligent systems. In recent years, SNNs have expanded from early visual recognition to natural language processing and large-scale language models, with researchers exploring paradigms including direct training ( 53 ; 8 ) , ANN-to-SNN conversion ( 20 ; 41 ) , and integer-to-spike unfolding ( 59 ; 24 ; 60 ; 57 ) — the latter gaining increasing attention for its compatibility with both sequential and parallel temporal patterns. In the direction of language modeling, recent efforts have begun to scale SNNs to large language models (LLMs) ( 57 ; 55 ) , demonstrating promising performance in language tasks. However, existing efforts on scaling SNNs toward LLMs remain largely confined to unimodal text scenarios, and SNNs have yet to be introduced into MLLMs, where multimodal inputs introduce new challenges and open questions. Bringing SNNs into MLLMs introduces two key challenges: (i) Visual and language modalities differ significantly in information density, activation distribution, and computational characteristics. A uniform spike encoding strategy fails to accommodate their distinct requirements — struggling to meet the precision demands of the language modality while introducing unnecessary redundancy for the visual modality. (ii) In the integer-to-spike unfolding paradigm, discrete activations with L quantization levels require T=L-1 timesteps ( 59 ; 57 ) . In MLLMs, high-resolution image inputs generate substantially more tokens than unimodal text-based LLMs, further amplifying temporal unfolding overhead and posing a significant efficiency bottleneck for multimodal inference. To address these challenges, we propose SpikeMLLM , the first spike-based framework for MLLMs, which unifies existing ANN quantization methods within the spiking representation space. Within this framework, we further develop two key mechanisms. For challenge (i) , we propose Modality-Specific Temporal Scales (MSTS) , which characterizes cross-layer representational evolution discrepancy across modalities via Modality Evolution Discrepancy (MED), and adaptively allocates differentiated timesteps accordingly — assigning more timesteps to the more dynamically evolving text modality while reducing timesteps for the spatially redundant visual modality, thereby improving overall performance without increasing effective inference overhead. For challenge (ii) , we propose Temporally Compressed LIF (TC-LIF) , which compresses timesteps from T=L-1 to T=\log_{2}(L)-1 via temporal weighted firing while preserving spike-driven sparse addition, substantially reducing spiking inference overhead without sacrificing representational capacity. Experiments on four mainstream MLLMs and scalability validation on Qwen2VL-72B ( 50 ) demonstrate that SpikeMLLM maintains near-lossless performance relative to the FP16 baseline under aggressive timestep compression (i.e., T_{v}/T_{t}=3/4 for visual/text modalities), with average performance gaps of only 0.72% and 1.19% on InternVL2-8B ( 6 ) and Qwen2VL-72B, respectively. In addition, to study whether these algorithmic simplifications can translate into practical system benefits, we further develop a dedicated RTL accelerator tailored to the induced spike-driven datapath. Under this deployment-oriented co-design setting, we observe favorable throughput and power characteristics relative to an FP16 GPU reference, suggesting the promise of algorithm–hardware co-design for efficient multimodal intelligence. The main contributions of this paper are summarized as follows: • We introduce SNNs into MLLMs for the first time, proposing SpikeMLLM, a spike-based framework that unifies existing ANN quantization methods within the spiking representation space. • We propose MSTS, which improves multimodal model performance through modality- and layer-wise differentiated timestep allocation without increasing effective timesteps. • We propose TC-LIF, which compresses the timesteps of integer-to-spike unfolding from T=L-1 to T=\log_{2}(L)-1 , effectively alleviating temporal unfolding overhead in spiking multimodal inference. • Extensive experiments on four mainstream MLLMs across diverse multimodal reasoning tasks, with scalability further demonstrated on Qwen2VL-72B, validate the effectiveness, compatibility, and scalability of SpikeMLLM under aggressive timestep compression. • We validate the energy efficiency advantages of SpikeMLLM through a dedicated RTL accelerator under hardware-software co-design, providing insights for next-generation neuromorphic chip design. 2. Preliminaries 2.1. Spiking Neuron SNNs transmit information via discrete spikes in the temporal dimension. The Leaky Integrate-and-Fire (LIF) model is widely adopted for its balance between efficiency and biological plausibility ( 35 ; 25 ) : (1) \mathbf{U}^{l}[t]=\lambda\mathbf{U}^{l}[t-1]+\mathbf{X}^{l}[t]-V_{th}\cdot\mathbf{S}^{l}[t-1] (2) \mathbf{S}^{l}[t]=\Theta(\mathbf{U}^{l}[t]-V_{th}) where \lambda is the leakage coefficient, V_{th} is the firing threshold, and \Theta(\cdot) is the Heaviside step function. At each timestep t , the neuron accumulates \mathbf{X}^{l}[t] into membrane potential \mathbf{U}^{l}[t] and emits spike \mathbf{S}^{l}[t] if the threshold is exceeded. The standard non-polar mode has \mathbf{S}^{l}[t]\in\{0,1\} , where 1 denotes an excitatory spike and 0 the resting state. The polar firing mode ( 16 ; 56 ) extends this to \mathbf{S}^{l}[t]\in\{-1,0\} or \{0,1\} via a dual-threshold mechanism: an excitatory spike ( +1 ) is fired when the membrane potential exceeds V_{th} , an inhibitory spike ( -1 ) when it falls below -V_{th} , and 0 otherwise. This enables richer encoding per timestep while preserving event-driven sparse addition. 2.2. Integer-to-Spike LIF However, both standard and polar LIF neurons produce discrete binary spike outputs, which