Spiking Transformers Need High Frequency Information

Yuetong Fang, Deming Zhou, Ziqing Wang, Hongwei Ren, Zecui Zeng, Lusong Li, Shibo Zhou, Renjing Xu · 2025

Read Spiking Transformers Need High Frequency Information by Yuetong Fang, Deming Zhou, Ziqing Wang, Hongwei Ren, Zecui Zeng, Lusong Li, Shibo Zhou, Renjing Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18608.txt.

Introduction

Yuetong Fang and co-authors present a 2025 study of spiking neural networks, brain-like systems that send data as one-bit spikes. Such nets promise low energy use, yet they trail standard nets in accuracy. The common blame is data loss from sparse spikes. The team argues the true flaw is a bias: the nets pass low-frequency parts of a signal and mute the fine, high-frequency parts.

Swapping a smoothing step for a peak-keeping step lifted scores in their tests. Their design, Max-Former, adds peak steps at the input and swaps attention for a local filter. It reached about 82 percent top scores on a large image set with fewer parts than a rival. A plain net variant set records on two smaller sets. Restoring fine detail, the work suggests, is the key fix for spiking nets.

Catalog record

CREATORYuetong Fang, Deming Zhou, Ziqing Wang, Hongwei Ren, Zecui Zeng, Lusong Li, Shibo Zhou, Renjing Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18608

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spiking Transformers Need High Frequency Information Yuetong Fang, Deming Zhou, Ziqing Wang, Hongwei Ren, Zecui Zeng, Lusong Li, Shibo Zhou, Renjing Xu Abstract Spiking Transformers offer an energy-efficient alternative to conventional deep learning by transmitting information solely through binary (0/1) spikes. However, there remains a substantial performance gap compared to artificial neural networks. A common belief is that their binary and sparse activation transmission leads to information loss, thus degrading feature representation and accuracy. In this work, however, we reveal for the first time that spiking neurons preferentially propagate low-frequency information. We hypothesize that the rapid dissipation of high-frequency components is the primary cause of performance degradation. For example, on Cifar-100, adopting Avg-Pooling (low-pass) for token mixing lowers performance to 76.73%; interestingly, replacing it with Max-Pooling (high-pass) pushes the top-1 accuracy to 79.12%, surpassing the well-tuned Spikformer baseline by 0.97%. Accordingly, we introduce Max-Former that restores high-frequency signals through two frequency-enhancing operators: extra Max-Pooling in patch embedding and Depth-Wise Convolution in place of self-attention. Notably, our Max-Former (63.99 M) hits the top-1 accuracy of 82.39% on ImageNet, showing a +7.58% improvement over Spikformer with comparable model size (74.81%, 66.34 M). We hope this simple yet effective solution inspires future research to explore the distinctive nature of spiking neural networks, beyond the established practice in standard deep learning. Figure 1: Spiking Transformer architectures: (a) Avg-Pool vs. (b) Max-Pool for token mixing, with (c) detailed implementation of the Spiking MLP (S-MLP) block. In mainstream (non-spiking) Vision Transformer research, Avg-Pool that captures global low-frequency patterns is a more common token mixing strategy than Max-Pool (high-pass) [ 1 , 2 ] . Surprisingly, in Spiking Transformers, replacing Avg-Pool with Max-Pool yields a +2.39% improvement on Cifar-100. 1 Introduction Spiking neural networks (SNNs) offer an energy-efficient alternative to conventional artificial neural networks (ANNs) [ 3 ] . Their efficiency stems primarily from spiking neurons that utilize spatiotemporal dynamics to mimic the operational principles of human brains [ 4 ] . In ANNs, all neurons within the same layer must await the complete processing of real-valued, dense tensors before any information can flow to the subsequent layer. In contrast, SNNs transmit information asynchronously, with spiking neurons consuming energy only when receiving or emitting spikes (“1”), otherwise remaining inactive [ 5 , 6 ] . This binary activation pattern enables SNNs to replace the energy-intensive multiply-and-accumulate (MAC) operations that are essential in ANNs with much simpler spike-based accumulation. Leveraging the energy-efficiency benefits, Spiking Transformers, which integrate the powerful Transformer architecture with spike-based computing, have gained growing attention [ 7 , 8 ] . Figure 2: Comparison between ReLU and spiking neuron (S-Neuron): (a) Input images; (b) Fourier spectrum analysis of output features processed as input \rightarrow activation \rightarrow weighting, with high-frequency regions marked (red dashed boxes: regions >0.55 \times max amplitude) and (c) the corresponding relative log amplitude; (d) GradCAM comparison with identical architectural setting following [ 8 ] , with the converted Spiking Transformer using 256 timesteps. Spiking neurons cause the rapid dissipation of high-frequency components, which consequently leads to the degradation of feature representations. However, the discrete nature of spike-based computation presents both opportunities and challenges. Currently, the most significant obstacle facing SNNs is the considerable performance gap compared to ANNs. For this discrepancy, the prevailing explanations have centered on the “representation error” [ 9 , 10 , 11 ] , a fundamental constraint in which spiking neurons transmit information via binary spike trains rather than continuous activation values. This seems inconsistent with the established consensus in the standard deep learning literature, where networks with low-bit or even binary activations can maintain comparable performance [ 12 , 13 ] . Further, it should be noted that SNNs operate on temporal sequences: despite spiking neurons strictly transmitting binary signals at each individual time step, a train of spikes spanning n simulation timesteps can encode activation values with at least \log(n) -bit precision [ 14 , 15 ] . The conflicting observation suggests unexplored opportunities for unlocking Spiking Transformers’ performance limit. It is natural to think from the frequency domain. Spiking neurons produce discrete, pulse-like activations, fundamentally distinguishing their frequency response from continuous activation functions commonly used in standard networks (e.g., ReLU [ 16 ] ). Prior works have suggested that spiking neurons may enrich signals with local details (high frequencies) [ 17 , 18 ] . However, in Figure 2 (b-c), we observed an intriguing phenomenon: when examining the end-to-end information flow of input \rightarrow activation \rightarrow weighting, rather than focusing solely on the property of activation functions, spiking neurons tend to propagate low-frequency information more prominently than ReLU. With this insight, we hypothesize that the rapid dissipation of high-frequency components is the primary cause of feature degradation, preventing Spiking Transformers from learning local, detailed information (Figure 2 (d)). To preliminarily validate this hypothesis, we conduct a simple experiment only using the non-parametric pooling operators as shown in Figure 1 . From the frequency domain, Max-Pooling excels at capturing local high-frequency details (e.g., local edges/textures), whereas average pooling favors global low-frequency patterns. Notably, Spiking Transformers exhibit opposite preferences to ANNs in token mixing. While average pooling is more common for token mixing in standard Transformers [ 1 , 2 ] , in Spiking Transformers, replacing it with Max-Pooling yields a +2.39% improvement on CIFAR-100. In this work, we provide both theoretical and empirical evidence that high-frequency information is essential for Spiking Transformers: • To identify the cause of feature degradation, we reveal and theoretically prove for the first time that spiking neurons function as low-pass filters at the network level. • We propose Max-Former to validate our hypothesis, which restores high-frequency information through two lightweight operators: extra Max-Pooling in patch embedding and Depth-Wise Convolution (DWC) in place of self-attention. • Restoring high-frequency information significantly improves performance while saving energy cost. On ImageNet, Max-Former achieves 82.39% top-1 accuracy (+7.58% over Spikformer) with 30% energy savings at similar model size (63.99M vs. 66.34M). We believe this straightforward yet powerful solution will motivate future research to explore the unique properties of SNNs, beyond the established practice in standard deep learning. 2 Preliminary and Related Works 2.1 Spiking Neuron Models SNNs implement spike-driven processing through biologically-inspired neuron models for non-linear activations. The Leaky Integrate-and-Fire (LIF) model is a widely adopted abstraction of this behavior, offering an effective balance between biological plausibility and computational efficiency [ 19 ] . The discretized LIF model under each simulation timestep n can be formulated as: \displaystyle U[n] \displaystyle=f(V[n-1],I[n]), (1) \displaystyle S[n] \displaystyle=H\bigl(U[n]-V_{\mathrm{th}}\bigr), (2) \displaystyle V[n] \displaystyle=\begin{cases}U[n]-V_{\mathrm{th}},&S[n]=1,\\ U[n],&S[n]=0,\end{cases} (3) where \beta is the decay factor, V_{\mathrm{th}} is the firing threshold, and H(\cdot) refers to the Heaviside step function that determining spike generation: S[n]=H\bigl(U[n]-V_{\mathrm{th}}\bigr)) = 1 when U[n]\geq V_{\mathrm{th}} , otherwise remains inactive ( S[n] = 0). The charging process of the LIF neuron is determined by f(\cdot) : \displaystyle f(V[n-1],I[n]) \displaystyle=\beta V[n-1]+(1-\beta)I[n] (4) At each timestep n , the current membrane potential U[n] is updated by integrating the time-domain signal I[n] corresponding to input data or intermediate operations like Conv and MLP. If U[n] exceeds the threshold V_{\text{th}} , the neuron fires a spike ( S[n] = 1). V[n] records membrane potential over time given the decay factor \beta and output spike activity. If the

Read the complete text