Towards High-performance Spiking Transformers from ANN to SNN Conversion Conference: Proceedings of the 32nd ACM International Conference on Multimedia; October 28-November 1, 2024; Melbourne, VIC, Australia Proceedings of the 32nd ACM International Conference on Multimedia (MM ’24), October 28-Nove
Zihan Huang, Xinyu Shi, Zecheng Hao, Tong Bu, Jianhao Ding, Zhaofei Yu, Tiejun Huang · 2025
Read Towards High-performance Spiking Transformers from ANN to SNN Conversion Conference: Proceedings of the 32nd ACM International Conference on Multimedia; October 28-November 1, 2024; Melbourne, VIC, Australia Proceedings of the 32nd ACM International Conference on Multimedia (MM ’24), October 28-Nove by Zihan Huang, Xinyu Shi, Zecheng Hao, Tong Bu, Jianhao Ding, Zhaofei Yu, Tiejun Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-21193.txt.
Introduction
Zihan Huang and co-authors present, in a 2025 paper, a better way to convert a standard neural network into a spiking one, including for transformer models. Spiking networks fire only in brief pulses, which promises low energy use, fast work, and toughness. Building them by conversion is simpler than training them directly, which eats memory.
Conversion has mostly been shown for convolutional networks. Transformers resist it, due to their non linear parts. The authors add a compensation module that uses the outputs of past time steps to estimate the expected output now. That preserves accuracy through the conversion. They also propose a neuron with several thresholds, plus a matching normalization scheme. The pair cuts the long time windows that high accuracy used to need, lowering delay. The result aims at high performance spiking transformers that can actually be deployed.
Catalog record
| CREATOR | Zihan Huang, Xinyu Shi, Zecheng Hao, Tong Bu, Jianhao Ding, Zhaofei Yu, Tiejun Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-21193
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards High-performance Spiking Transformers from ANN to SNN Conversion Conference: Proceedings of the 32nd ACM International Conference on Multimedia; October 28-November 1, 2024; Melbourne, VIC, Australia Proceedings of the 32nd ACM International Conference on Multimedia (MM ’24), October 28-November 1, 2024, Melbourne, VIC, Australia DOI: 10.1145/3664647.3680620 ISBN: 979-8-4007-0686-8/24/10 CCS: Computing methodologies Artificial intelligence Zihan Huang, Xinyu Shi, Zecheng Hao, Tong Bu, Jianhao Ding, Zhaofei Yu, Tiejun Huang Abstract . Spiking neural networks (SNNs) show great potential due to their energy efficiency, fast processing capabilities, and robustness. There are two main approaches to constructing SNNs. Direct training methods require much memory, while conversion methods offer a simpler and more efficient option. However, current conversion methods mainly focus on converting convolutional neural networks (CNNs) to SNNs. Converting Transformers to SNN is challenging because of the presence of non-linear modules. In this paper, we propose an Expectation Compensation Module to preserve the accuracy of the conversion. The core idea is to use information from the previous T time-steps to calculate the expected output at time-step T. We also propose a Multi-Threshold Neuron and the corresponding Parallel Parameter normalization to address the challenge of large time steps needed for high accuracy, aiming to reduce network latency and power consumption. Our experimental results demonstrate that our approach achieves state-of-the-art performance. For example, we achieve a top-1 accuracy of 88.60% with only a 1% loss in accuracy using 4 time steps while consuming only 35% of the original power of the Transformer. To our knowledge, this is the first successful Artificial Neural Network (ANN) to SNN conversion for Spiking Transformers that achieves high accuracy, low latency, and low power consumption on complex datasets. The source codes of the proposed method are available at https://github.com/h-z-h-cell/Transformer-to-SNN-ECMT. 1. Introduction Spiking neural networks(SNNs) are a type of neural network model that imitates the mechanisms of biological neurons ( Bohte et al. 2000 ; Gerstner et al. 2014 ) . They are called the third generation of neural networks ( Maass 1997 ) due to their biological plausibility and computational efficiency ( Tavanaei et al. 2019 ; Zenke et al. 2021 ) . Neurons in SNNs do not produce output at each time step. Instead, they become active and emit spikes only when their membrane potential reaches a specific threshold. The sparse activity of spikes leads to significantly higher computational efficiency than traditional neural networks ( Roy et al. 2019 ) , especially on neuromorphic chips ( Merolla et al. 2014 ; Davies et al. 2018 ; DeBole et al. 2019 ; Pei et al. 2019 ) . However, training large-scale, high-precision, and low-latency SNNs remains challenging due to the non-differentiable nature of spikes. Currently, there are two main approaches to train SNNs. The first approach is direct training using backpropagation or local learning ( Lee et al. 2016 ; Wu et al. 2018 ; Neftci et al. 2019 ; Zhang and Li 2020 ; Fang et al. 2021 ; Wu et al. 2022 ; Zhu et al. 2022 ; Yin et al. 2024 ; Xu et al. 2024 ; Duan et al. 2022 ) . These methods utilize differentiable continuous functions or spike-time-dependent plasticity strategies to replace the non-differentiable spike emission rules. However, this training process still relies on standard GPUs that are not well-suited for the unique characteristics of SNNs, leading to significant resource consumption and limited performance. The second approach is ANN to SNN conversion ( Cao et al. 2015 ; Rueckauer et al. 2017 ; Li et al. 2021 ; Deng and Gu 2021 ; Bu et al. 2022b ) . This conversion method does not require any additional training. Instead, it uses pre-trained ANNs and replaces activation functions with spiking neurons, leveraging the similarity between ReLU activation functions and spike emission rates of integrate-and-fire models. The result SNN model preserves the original ANN’s performance but often leads to longer inference times, and the modules that can be successfully converted are limited. As is well known, Transformers have demonstrated exceptional performance in various vision tasks ( Dosovitskiy et al. 2021 ; Radford et al. 2021 ; Jain et al. 2023 ; Liu et al. 2023 ; Chen et al. 2024 ) . Despite numerous efforts to convert CNNs to SNNs, converting Transformer models remains a challenge. This is due to unique nonlinear modules such as layernorm and GELU in Transformers that differ from the ReLU function in CNNs. These modules require interaction between neurons within the same layer and exhibit non-linear characteristics, making it challenging to achieve accurate conversion through the linear piecewise quantization of individual neurons. This paper proposes a new method to convert Transformers to SNNs. The main challenge lies in handling non-linear modules. To address this, we propose an Expectation Compensation Module (ECM) that calculates expectations and replaces each non-linear module. Specifically, a customized ECM is employed to substitute the matrix product, performing most operations through accumulations. This reduces power consumption and ensures the total output matches the expected result at each time step. To improve the efficiency of minimal spikes, we introduce Multi-Threshold Neurons and the corresponding Parallel Parameter normalization, significantly reducing the required latency and power consumption for inference with comparable accuracy. Our main contributions are summarized as follows: • We analyze the challenges of non-linear module conversion in Transformer and introduce a novel solution called the Expectation Compensation Module, which uses the information from the previous time steps to calculate the expected output at the current time step. This module overcomes the limitations of traditional methods with minimal power consumption increase. • To overcome the issue of slow accuracy improvement over time during Transformer conversion, we propose a Multi-Threshold Neuron and the corresponding Parallel Parameter normalization, substantially reducing power consumption requirements and significantly decreasing latency. • The proposed method is effective on the ImageNet1k dataset, outperforming existing SNN models in accuracy and significantly reducing power consumption compared to other Transformer models. It achieves a top-1 accuracy of 88.60%, with only a 1% accuracy loss compared to ANN, while reducing energy consumption by 65%. 2. Related Works ANN-SNN Conversion The ANN-SNN conversion methods aim to replicate the performance of ANNs by converting pre-trained ANN weights into synaptic weights of SNNs. Cao et al. ( Cao et al. 2015 ) initially proposed training an ANN with ReLU activation function and then replacing the activation layer with IF neurons. Diehl et al. ( Diehl et al. 2015 ) further narrowed the gap by scaling and normalizing the weights. To address the spike count errors resulting from the hard reset mechanism, soft reset neurons were proposed in Rueckauer et al. ( Rueckauer et al. 2017 ) and Han et al. ( Han et al. 2020 ) . Further research has aimed to minimize conversion errors through various optimization: (1) Optimizing thresholds: Sengupta et al. ( Sengupta et al. 2019 ) and Zhang et al. ( Zhang et al. 2023 ) proposed dynamic threshold adjustment strategies during the conversion process. (2) Optimizing membrane potential: Bu et al. ( Bu et al. 2022a ) demonstrated that setting the initial membrane potential at half the threshold can reduce errors. Hao et al. ( Hao et al. 2023 ) further suggested analyzing residual membrane potential to eliminate conversion errors.