Spike2Former: Efficient Spiking Transformer for High-performance Image Segmentation
Zhenxin Lei, Man Yao, Jiakui Hu, Xinhao Luo, Yanye Lu, Bo Xu, Guoqi Li · 2024
Read Spike2Former: Efficient Spiking Transformer for High-performance Image Segmentation by Zhenxin Lei, Man Yao, Jiakui Hu, Xinhao Luo, Yanye Lu, Bo Xu, Guoqi Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14587.txt.
Introduction
Zhenxin Lei and co-authors present Spike2Former, a spiking transformer for image segmentation, in a paper from 2024. Spiking networks sip power, which suits cameras and edge devices. They lag badly at segmentation, the pixel-by-pixel labeling of scenes. Direct conversion of dense networks often collapses their firing.
The team found which modules choke spike flow and redesigned them. A new neuron type, the normalized integer spiking neuron, steadied training. On three standard datasets, scores jumped by nine to fourteen points over prior spiking bests. Power efficiency rose five to six fold at the same time. The results narrow the gap between spiking and standard vision models.
Catalog record
| CREATOR | Zhenxin Lei, Man Yao, Jiakui Hu, Xinhao Luo, Yanye Lu, Bo Xu, Guoqi Li |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14587
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spike2Former: Efficient Spiking Transformer for High-performance Image Segmentation Zhenxin Lei, Man Yao, Jiakui Hu, Xinhao Luo, Yanye Lu, Bo Xu, Guoqi Li Abstract Spiking Neural Networks (SNNs) have a low-power advantage but perform poorly in image segmentation tasks. The reason is that directly converting neural networks with complex architectural designs for segmentation tasks into spiking versions leads to performance degradation and non-convergence. To address this challenge, we first identify the modules in the architecture design that lead to the severe reduction in spike firing, make targeted improvements, and propose Spike2Former architecture. Second, we propose normalized integer spiking neurons to solve the training stability problem of SNNs with complex architectures. We set a new state-of-the-art for SNNs in various semantic segmentation datasets, with a significant improvement of + 12.7 % \mathbf{+12.7\%} mIoU and 5.0 × \mathbf{5.0\times} efficiency on ADE20K, + 14.3 % \mathbf{+14.3\%} mIoU and 5.2 × \mathbf{5.2\times} efficiency on VOC2012, and + 9.1 % \mathbf{+9.1\%} mIoU and 6.6 × \mathbf{6.6\times} efficiency on CityScapes. Our code is available at https://github.com/BICLab/Spike2Former Introduction Spiking Neural Networks (SNNs) emulate the spatiotemporal dynamics and spike-based communication of biological neurons. The former ensures the network’s representation ( Maass 1997 ) , while the spike-driven paradigm introduced by the latter allows SNNs to perform sparse computing when deployed on neuromorphic chips ( Merolla et al. 2014 ; Davies et al. 2018 ) , thereby benefiting from low power consumption ( Roy, Jaiswal, and Panda 2019 ; Schuman et al. 2022 ) . A notable example is the sensing-computing neuromorphic chip Speck, which operates at a power level as low as 0.7mW in typical visual scenes ( Yao et al. 2024d ) . The complex neuronal dynamics and binary activations make it challenging to train large-scale SNNs. It took a long time for the SNN field to effectively address this issue through surrogate gradient training ( Wu et al. 2018 ; Neftci, Mostafa, and Zenke 2019a ) and residual learning design ( Fang et al. 2021 ; Hu et al. 2024b ) . Currently, SNNs have achieved commendable performance on simple image classification tasks ( Yao et al. 2024b ; Yao et al. 2024a ) . Unfortunately, when it comes to complex visual tasks that require the use of sophisticated neural network architectures, SNNs fall short. For instance, in image segmentation, an additional segmentation head is required alongside the backbone used for image classification, resulting in a network structure that is significantly more complex than that for classification. Simply converting complex Artificial Neural Networks (ANNs) into spiking versions or directly applying residual designs from classification tasks often leads to a notable drop in performance. Consequently, the few existing SNN models ( Kim, Chough, and Panda 2022 ; Zhang, Fan, and Zhang 2023 ; Yao et al. 2024a ; Su et al. 2024 ; Patel et al. 2021 ) that tackle image segmentation tasks tend to perform poorly. This work explores the application of SNNs with more complex architectures to image segmentation tasks. Specifically, Mask2Former ( Cheng et al. 2022 ) is a classic Transformer-based per-mask classification architecture consisting of three parts: a backbone network; a Feature Pyramid Network (FPN) pixel decoder with a multi-scale deformable transformer encoder block; and a transformer decoder block. Directly converting the Mask2Former architecture into a spiking version results in obvious performance degradation and non-convergence. To address this, we investigated which modules in the spiking Mask2Former contribute to the significant loss of information, where the spiking neurons in these modules nearly cease to fire. The first module with severe spike degradation is the deformable attention transformer encoder block in the FPN decoder. In the vanilla Mask2Former, the query operation in the deformable attention block is inherently sparse; if the queried information consists of sparse spikes, this could result in significant information loss. To address this, we incorporate convolution blocks in encoder blocks and redesign the deformable attention blocks to preserve more effective information and energy efficiency. The second key module that leads to information loss is the final mask embedding layer. This layer is crucial as it outputs the final segmentation results; however, being at the deepest part of the network, the semantic information that reaches this point is already greatly diminished. To end this, we build an auxiliary information branch to enhance the representation of mask embedding. Another challenge is that, beyond architectural design, spiking neurons inherently introduce information loss by converting continuous values into binary spikes. This issue has long plagued the SNN field, leading to the development of various methods aimed at refining the membrane potential distribution, such as attention mechanism ( Yao et al. 2023 ) and information maximization loss ( Guo et al. 2022 ) . Recently, Luo et al. 2025 proposed an integer training and spike inference method in spiking CNNs to address this challenge. However, this approach does not extend to more complex architectures like Mask2Former, which require numerical stability and precise representation in the interaction of cross-modal features ( Cheng, Schwing, and Kirillov 2021 ; Carion et al. 2020 ; Li et al. 2023 ) . To address this, we propose a novel normalization method that normalizes integers during training to facilitate training, while ensuring that the spike-driven nature during inference remains unaffected. The proposed methods are validated on the popular datasets ADE20k, CityScapes, Pascal VOC2012 for semantic segmentation. Spike2Former far exceeds the best models in SNNs in both performance and energy consumption and is comparable to the ANNs. The main contributions are: 1) Spike2Former . We analysis the challenge of applying SNNs to complex architecture and propose a transformer-based image segmentation method, Spike2Former, that incorporates the Spike-driven Deformable Transformer Encoder (SDTE) and Spike-Driven Mask Embedding (SDME) module to resolve the information deficiency and improve model performance. 2) NI-LIF Spiking Neuron . We design a novel spiking neuron, NI-LIF, to increase the training stability and reduce the information deficiency in complex segmentation method. The NI-LIF normalizes the integer activation during training and can be equivalent to spike-driven in inference, capitalizing on the low power nature of SNNs. 3) Performance : The proposed Spike2Former obtains a remarkable performance improvement in popular segmentation tasks with low power consumption, demonstrating the potential of SNNs in complex scenarios. Spike2Former achieves a new state-of-the-art ADE20k ( \mathbf{+12.7\%} mIoU and \mathbf{5.0\times} ), Pascal VOC2012 ( \mathbf{+14.3\%} mIoU and \mathbf{5.2\times} ), CityScapes ( \mathbf{+9.1\%} mIoU and \mathbf{6.6\times} ). Figure 1: (A) The architecture of Spike2Former, we propose Spike-Driven Mask Embedding(SDME) and Spike-driven Deformable Transformer Encoder to introduce the transformer-based method to SNNs. (B) Micro Design in Spike-Driven Deformable Transformer Encoder(SDTE) including Spike-driven Deformable attention(SDDA) and Spike Separable Convolution(ESC). (C) Comparison between NI-LIF and I-LIF. Integer activation results in information loss, especially in cross-model interaction. NI-LIF normalizes the integer during training to preserve information and converts them to spikes during inference with only sparse addition. Related Works SNN Training Training methods in Spiking Neural Networks (SNNs) mainly fall into two categories: ANN-to-SNN conversion and direct training. While ANN-to-SNN conversion inherits from traditional neural networks, it suffers from longer time steps and limited real-time processing capabilities ( Li et al. 2022 ; Bu et al. 2022 ; Wu et al. 2021 ) . Direct training with surrogate gradients ( Neftci, Mostafa, and Zenke 2019b )