BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation
Sihang Guo, Chenlin Zhou, Jiaqi Wang, Kehai Chen, Qingyan Meng, Zhengyu Ma · 2026
Read BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation by Sihang Guo, Chenlin Zhou, Jiaqi Wang, Kehai Chen, Qingyan Meng, Zhengyu Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13859.txt.
Introduction
Guo and five colleagues present BiSpikCLM, claimed as the first fully binary spiking language model that avoids floating-point matrix multiplication. Its softmax-free spiking attention drops both softmax and floating-point steps from autoregressive modeling, cutting energy use. A distillation method called Spike-Aware Alignment Distillation trains the spiking student against a conventional teacher across embeddings, attention maps, features and outputs.
Catalog record
| CREATOR | Sihang Guo, Chenlin Zhou, Jiaqi Wang, Kehai Chen, Qingyan Meng, Zhengyu Ma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91663 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13859
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation Sihang Guo, Chenlin Zhou, Jiaqi Wang, Kehai Chen, Qingyan Meng, Zhengyu Ma Abstract Spiking Neural Networks (SNNs) offer promising energy-efficient alternatives to large language models (LLMs) due to their event-driven nature and ultra-low power consumption. However, to preserve capacity, most existing spiking LLMs still incur intensive floating-point matrix multiplication (MatMul) and nonlinearities, or training difficulties arising from the complex spatiotemporal dynamics. To address these challenges, we propose BiSpikCLM, the first fully binary spiking MatMul-free causal language model. BiSpikCLM introduces Softmax-Free Spiking Attention (SFSA), eliminating softmax and floating-point operations in autoregressive language modeling. For efficient training, we introduce Spike-Aware Alignment Distillation (SpAD), which aligns ANN teacher and SNN student across embeddings, attention maps, intermediate features, and output logits. SpAD framework allows BiSpikCLM to reach comparable performance to ANN counterparts using substantially fewer training tokens (e.g., only 5.6% of the tokens for the 1.3B model). As a result, BiSpikCLM achieves competitive performance at only 4.16%–5.87% of the computational cost on natural language generation tasks. Our results highlight the feasibility and effectiveness of fully binary spike-driven LLMs and establish the distillation as a promising pathway for brain-inspired spiking NLP. Figure 1 : Overview of Softmax-Free Spiking Attention (SFSA) . Left: Comparison between Vanilla Causal Self-Attention (CSA) (bottom) and SFSA (top). CSA uses softmax and additive masks, while SFSA employs spike-based activation and binary causal masking. Right: Detailed SFSA pipeline, showing spike-form Q, K, V computation, masked integer attention, spiking activation, and spike-based output, enabling fully discrete and energy-efficient attention modeling. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing, powering a wide range of applications from conversational agents to code generation ( 6 ; 1 ) . However, these models typically require extensive computational resources and energy consumption during both training and inference. For example, GPT-3 was trained with 175 billion parameters using hundreds of petaflop/s-days of compute ( 6 ) . In addition, inference also incurs substantial energy costs, as serving a single query can involve billions of operations and significant GPU utilization ( 43 ; 41 ) , raising concerns about their scalability and environmental impact ( 43 ) . Unlike ANN-based LLMs, the human brain achieves superior intelligence with far lower energy consumption, operating on just 20 watts to power approximately 86 billion neurons ( 17 ; 13 ) . Inspired by this energy-efficiency, Spiking Neural Networks (SNNs) ( 29 ; 13 ) communicate through binary spike events, enabling event-driven and low-power computation ( 56 ; 40 ) , particularly on neuromorphic hardware, making SNNs a promising alternative to traditional ANNs. While recent efforts have shown promising results of SNNs in computer vision tasks ( 61 ; 26 ; 27 ) , extending SNNs to natural language processing (NLP), especially LLMs, remains largely underexplored. A central challenge is the design of spiking attention mechanisms. In contrast to vision models, where representations are often bidirectional and spatially local, autoregressive LLMs require causal attention to ensure that each token prediction depends only on its preceding context. However, conventional causal attention relies on floating-point matrix multiplications and the softmax operation, both of which are computationally intensive and fundamentally incompatible with spike-based processing. Existing attempts either retain these components ( 63 ) or introduce multi-threshold neurons and integer activations ( 51 ) , which still incur substantial floating-point overhead. Designing a spike-driven causal attention mechanism is therefore critical: it must eliminate softmax while preserving the autoregressive representational capacity of binary spike trains. This challenge directly motivates our Softmax-Free Spiking Attention (SFSA) , which enables spike-based language modeling for spiking LLMs. Moreover, training spiking LLMs introduces additional difficulties beyond those in vision tasks. The inherent temporal dynamics of SNNs already leads to complex computational graphs and high computational cost during backpropagation. Scaling up the architecture further exacerbates this, making full end-to-end training inefficient or even infeasible. Consequently, prior works mostly resort to ANN-to-SNN conversions ( 51 ; 39 ) . However, such methods typically require large time steps to approximate ANN activations, resulting in high inference cost. Integer-based conversions further scale the operations by T\times N , which compromises the potential energy benefits of event-driven spiking computation. To address these challenges, we propose BiSpikCLM, a binary spiking causal language model built on two key components: a spike-driven attention mechanism (SFSA), schematically depicted in Figure 1 , and a spike-aware knowledge distillation scheme (SpAD), presented in Figure 3 . Overall, our contributions can be summarized as follows: • We propose BiSpikCLM , the first binary spiking MatMul-free causal language model equipped with a fully spike-driven attention mechanism. Our SFSA replaces the causal self-attention, which relies on floating-point operations and softmax, enabling efficient autoregressive language modeling with binary spikes. The overall design follows the OPT-family architecture ( 58 ) , adapted to the spiking domain. • We introduce SpAD , a novel training framework that enables BiSpikCLM to be directly trained from random initialization. SpAD distills hierarchical knowledge covering embeddings, attention maps, intermediate features, and output logits from the teacher model, thereby accelerating convergence and reducing the amount of training data required for large-scale spiking LLMs. • With only 10B training tokens, significantly fewer than the 180B tokens used to train OPT-1.3B, our SpAD framework enables BiSpikCLM-1.3B to achieve 42.19% zero-shot accuracy on common reasoning benchmarks using 4 time steps, approaching the 49.73% of OPT-1.3B, while consuming just 10.6% of the energy per inference. Remarkably, even at 2 time steps, the model maintains 41.33% accuracy with only 5.88% of the energy cost. Table 1 : Comparison of spiking language models. The properties “Softmax-Free”, “Exp-Free (exponential-function-free)”, and “FP-Mul-Free (floating-point multiplication-free)” are assessed at the attention-module level. Model Params Spike Form Training Method Architecture Softmax-Free Exp-Free FP-Mul-Free SpikeBERT ( 28 ) 109M Binary Knowledge distillation Encoder-only ✓ ✓ \times SpikingBERT ( 2 ) 50M Binary Knowledge distillation Encoder-only \times \times \times SpikeLM (BERT) ( 52 ) 110M Ternary Train from scratch Encoder-only \times \times \times SpikeLM (BART) ( 52 ) 139M Ternary Train from scratch Encoder–Decoder \times \times \times SpikeLM (mBART) ( 52 ) 680M Ternary Train from scratch Encoder–Decoder \times \times \times SpikeGPT ( 63 ) 216M Binary Train from scratch Decoder-only ✓ \times \times SpikeLLM ( 51 ) 7-70B Integer Post-training quantization Decoder-only \times \times \times BiSpikCLM (ours) 1.3B Binary SpAD Decoder-only ✓ ✓ ✓ 2 Related Work 2.1 SNNs in Downstream Tasks Recent works show SNNs achieving competitive performance in vision tasks with lower computational consumption. In image classification, advances in surrogate gradients, attention have boosted accuracy and efficiency on CIFAR-10/100 and ImageNet ( 35 ; 62 ; 60 ; 61 ; 26 ) . In object detection, models like SFOD and SpikeYOLO reduce energy cost while closing the gap with ANNs ( 44 ; 4 ; 10 ; 27 ) . For event-based vision, architectures such as MG-SNN effectively handle gesture, motion, and optical flow tasks ( 24 ; 12 ; 34 ; 59 ) . In contrast, the application of SNNs in natural language processing (NLP) is still largely underexplored, with only a few attempts adapting language models to spike-based computation. For example, 28 employs a two-stage distillation pipeline to train an encoder-only spiking student from