Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers

Xinzhe Yuan, Xiang Peng, Bin Gu, Huan Xiong · 2026

Read Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers by Xinzhe Yuan, Xiang Peng, Bin Gu, Huan Xiong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20289.txt.

Introduction

Xinzhe Yuan and three co-workers present spiking operators for spiking transformers. ANN-to-SNN conversion gives a training-free path to spiking language models, but pipelines cover mainly linear steps. That gap limits neuromorphic runs, where nonlinearities need division, exponentiation, or norms. Their method splits these into division, exponentiation, and l2 norms, built with leaky integrate-and-fire neuron groups and bit-shift scaling. These blocks support Softmax, SiLU, and normalization without fine-tuning. Tests show replacing nonlinear operators costs less than one percent accuracy on all tasks.

Catalog record

CREATORXinzhe Yuan, Xiang Peng, Bin Gu, Huan Xiong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53986
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20289

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers Xinzhe Yuan, Xiang Peng, Bin Gu, Huan Xiong Abstract ANN-to-SNN conversion offers a practical, training-free route to spiking large language models. However, current pipelines primarily focus on spike-driven realizations for Transformer linear-algebra operations, while providing limited support for key nonlinear operators. This gap limits compatibility with neuromorphic-style execution constraints, where such nonlinearities typically require division, exponentiation, or norm computations that are not naturally supported by standard leaky integrate-and-fire dynamics. To solve this problem, we propose a plug-and-play framework that implements spike-friendly approximations for Transformer nonlinearities and integrates into existing ANN-to-SNN pipelines. Our method decomposes these nonlinear computations into three recurring primitives—division, exponentiation, and ℓ 2 \ell_{2} norms—and realizes them via population computation using LIF neuron groups, combined with lightweight bit-shift scaling to avoid floating-point arithmetic. By composing these primitives as modular operator blocks, our framework supports common Transformer nonlinearities (e.g., Softmax, SiLU, and normalization) without any fine-tuning. Experiments on a range of LLMs Transformers show that selectively replacing the targeted nonlinear operators incurs less than a 1 % 1\% accuracy drop across all evaluated tasks. 1 Introduction Recently, spiking neural networks (SNNs) have gained increasing attention for energy-efficient, event-driven computation on neuromorphic hardware ( 15 ; 5 ; 1 ) . These models have shown promising results in both computer vision and natural language processing ( 2 ; 27 ; 12 ; 28 ) . Meanwhile, Transformer-based foundation models, particularly large language models (LLMs), have become the dominant inference workloads. Their costs are largely driven by dense linear algebra and associated memory traffic ( 9 ) . Motivated by this, recent spiking Transformers and SNN-LLM efforts have increasingly adopted ANN-to-SNN conversion to reduce inference cost without expensive retraining ( 16 ; 3 ; 4 ; 25 ) . These methods primarily convert Transformer linear computations, such as attention matrix multiplications and feed-forward projections, into spike-driven implementations that leverage event sparsity for improved energy efficiency ( 16 ; 23 ) . However , this spike-centric research paradigm remains incomplete for Transformer architectures, as existing works rarely provide spike-based realizations of key nonlinear operators such as activation functions, normalization layers, and Softmax ( 27 ; 18 ) . Although these components are often considered secondary from an energy-accounting perspective ( 9 ) , they become decisive under strict spike-only deployment. They are difficult to represent with standard leaky integrate-and-fire (LIF) neurons, since they commonly require division or norm computations that do not naturally arise from LIF dynamics. More critically, on neuromorphic hardware platforms that support only spike-based primitives, continuous-valued states are unavailable, making conventional implementations a fundamental obstacle to deployment ( 5 ; 6 ) . Consequently, existing ANN-to-SNN approaches often fall short of truly end-to-end spiking Transformers under strict spike-only constraints, including their nonlinearities ( 28 ; 12 ) . To address this limitation, a few works approximate Transformer nonlinearities with spiking computation. But they typically require additional training, limiting their compatibility with standard ANN-to-SNN conversion pipelines ( 19 ) . Motivated by these observations, we ask the following natural and important question: Can we design plug-and-play spike-based conversion for nonlinear operators in ANN-to-SNN pipelines? In this work, we answer this question affirmatively by developing training-free spike-based replacements that are compatible with standard LIF dynamics. We identify three primitive nonlinear computations that repeatedly arise in Transformer inference: division, exponentiation, and \ell_{2} norms, which form the computational core of Softmax, SiLU, and RMSNorm. Our spike-friendly realizations are built from population computation using LIF neuron groups, together with simple shift-based scaling that avoids floating-point arithmetic. By constructing approximations for these primitives and composing them as modular operator blocks, we obtain fully spiking realizations of the above Transformer nonlineariti es under strict spike-only constraints, without any additional fine-tuning. These operator-level modules are composable and can be plugged into existing ANN-to-SNN conversion pipelines to enable end-to-end spiking Transformer blocks, while respecting the spike-only primitives and lightweight digital operations natively supported by neuromorphic hardware. Our main contributions are summarized as follows: • We propose a spike-friendly approach to approximate Transformer nonlinear operators. Our method operates at the operator level and can be seamlessly integrated into existing ANN-to-SNN conversion pipelines without requiring any modifications to model weights. • We provide theoretical guarantees on the conversion error. We show that our spike-based approximations of nonlinear operators admit provably bounded conversion errors under mild conditions. Additionally, we identify specific configurations that can achieve high approximation accuracy. • We empirically demonstrate the applicability of the proposed method across different models. We test our method on two widely used ANN-to-SNN conversion frameworks. Additionally, we apply our approach to well-known models that have not previously been converted to SNNs, such as Qwen3, by selectively replacing their nonlinear operators while keeping the other computational parts unchanged, to verify the broad applicability of our method. 2 Related Work Early ANN-to-SNN conversion methods approximate continuous ANN activations using firing-rate coding and scale alignment. Diehl et al. introduce weight and threshold balancing to enable fast and accurate inference in deep spiking networks ( 7 ) . Sengupta et al. further extend this conversion paradigm to deeper architectures, demonstrating that VGG and residual networks can be converted to SNNs without retraining while maintaining competitive performance ( 17 ) . As Transformers became the dominant model class, You et al. propose SpikeZIP-TF, which systematically applies ANN-to-SNN conversion to Transformer architectures by spiking the linear projections in attention and feed-forward layers ( 25 ) . At the scale of large language models, Xing et al. introduce SpikeLLM, which constructs large spiking language models using saliency-driven spike allocation and reports efficiency–accuracy trade-offs compared with standard low-bit inference pipelines ( 22 ) . Distinct from the above works that primarily focus on dense linear algebra, Sorbet constructs spike-based realizations of nonlinear functions using shift-based discrete operations, and applies knowledge distillation and fine-tuning to align the spiking model with the original BERT behavior ( 19 ) . Although it identifies the issue of non-linear operators, its method requires training and is incompatible with other pipelines. 3 Preliminary Spiking Neurons and Temporal Accumulation SNNs process information over discrete timesteps t=1,\dots,T . At each timestep, neurons emit binary spikes and communicate through temporally accumulated activity. As a result, real-valued quantities can be represented implicitly by spike counts over a finite temporal window. In practice, ANN-to-SNN conversion methods typically rely on rate-based representations, where the accumulated spike activity approximates the activation of an artificial neuron: a_{\mathrm{SNN}}\approx\sum_{t=1}^{T}s^{t}\cdot\theta, (1) where s^{t}\in\{0,1\} denotes the spike at timestep t , and \theta is the firing threshold. Under appropriate scaling, the expected spike count matches the quantized ANN activation, enabling a direct correspondence between ANN activations and SNN spike statistics ( 16 ) . Leaky Integrate-and-Fire Neuron LIF neuron is the most commonly used neuron model in SNNs. In discrete time, its dynamics are given by \displaystyle v(t) \displaystyle=\lambda v(t-1)+I(t), (2) \displaystyle s(t) \displaystyle=\mathbb{I}\left[v(t)\geq\theta\right], (3) \displaystyle v(t) \displaystyle\leftarrow v(t)-s(t)\theta, (4) where v(t) denotes the membrane potential, I(t) the input current, \lambda\in(0,1] the leak factor, and \theta the firing threshold, \mathbb{I}[\cdot] denotes the indicator function. This accumulate–threshold–reset mechanism allows LIF neurons to approximate linear transformations through temporal spike accumulation, forming the basis of most existing ANN-to-SNN conversion methods. 4 Method LLMs

Read the complete text