Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks

Kai Sun, Peibo Duan, Yongsheng Huang, Guowei Zhang, Benjamin Smith, Nanxu Gong, Levin Kuhlmann · 2026

Read Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks by Kai Sun, Peibo Duan, Yongsheng Huang, Guowei Zhang, Benjamin Smith, Nanxu Gong, Levin Kuhlmann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14252.txt.

Introduction

Sun and six co-authors propose Selective Alignment Knowledge Distillation for spiking neural networks with unequal treatment. Current distillation aligns all steps, assuming all predictions should be treated equally. SNN predictions vary over time, and middle steps need not each be correct. SeAl-KD aligns class and temporal knowledge, equalizing rival logits at wrong steps and reweighting alignment by score. Tests on image and neuromorphic datasets show improvements over older distillation methods.

Catalog record

CREATORKai Sun, Peibo Duan, Yongsheng Huang, Guowei Zhang, Benjamin Smith, Nanxu Gong, Levin Kuhlmann
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42210
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14252

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks Kai Sun, Peibo Duan, Yongsheng Huang, Guowei Zhang, Benjamin Smith, Nanxu Gong, Levin Kuhlmann Abstract Spiking neural networks (SNNs), which are brain-inspired and spike-driven, achieve high energy efficiency. However, a performance gap between SNNs and artificial neural networks (ANNs) still remains. Knowledge distillation (KD) is commonly adopted to improve SNN performance, but existing methods typically enforce uniform alignment across all timesteps, either from a teacher network or through inter-temporal self-distillation, implicitly assuming that per-timestep predictions should be treated equally. In practice, SNN predictions vary and evolve over time, and intermediate timesteps need not all be individually correct even when the final aggregated output is correct. Under such conditions, effective distillation should not force every timestep toward the same supervision target, but instead provide corrective guidance to erroneous timesteps while preserving useful temporal dynamics. To address this issue, we propose S elective A lignment K nowledge D istillation (SeAl-KD) , which selectively aligns class-level and temporal knowledge by equalizing competing logits at erroneous timesteps and reweighting temporal alignment based on confidence and inter-timestep similarity. Extensive experiments on static image and neuromorphic event-based datasets demonstrate consistent improvements over existing distillation methods. The code is available at https://github.com/KaiSUN1/SeAl . 1 Introduction Spiking Neural Networks (SNNs), often regarded as the third generation of neural networks, are biologically inspired models that communicate through discrete spikes rather than continuous-valued activations ( 16 ). By processing information in an event-driven manner with sparse spikes over time, SNNs offer the potential for high energy efficiency, especially when deployed on neuromorphic hardware ( 8 ). Nevertheless, training SNNs to achieve accuracy comparable to Artificial Neural Networks (ANNs) remains challenging, because useful evidence is accumulated progressively through spikes, making different timesteps unevenly informative during learning ( 1 ; 18 ). To narrow the performance gap between ANNs and SNNs, knowledge distillation (KD) has become a widely adopted strategy ( 22 ; 20 ). Early distillation methods typically supervise only the final aggregated output by matching averaged logits, which may mix inconsistent temporal information during optimization ( 31 ; 3 ). More recent approaches move toward timestep-wise distillation by injecting teacher supervision at each timestep and aligning logits throughout the temporal dimension ( 29 ; 28 ). In parallel, several methods further exploit the temporal structure of SNNs through self-distillation ( 5 ) or temporal consistency regularization ( 31 ; 4 ), encouraging predictions or features from different timesteps to be consistent in order to stabilize optimization and improve performance ( 5 ; 29 ). The consistency assumption is partially misaligned with the intrinsic properties of SNNs and their prediction mechanism. Due to membrane potential integration and reset, spike firing may induce abrupt changes in membrane states, making it difficult for SNNs to maintain identical predictions across all timesteps, as detailed in Appendix A . Meanwhile, since the final decision is determined by temporal accumulation rather than any single timestep, an incorrect intermediate prediction does not necessarily imply an incorrect final outcome. As illustrated in the toy case in Figure 1 (a), some intermediate timesteps are misclassified while the final temporally aggregated prediction remains correct. Figure 1 (b) further shows that per-timestep accuracy is consistently lower than temporally aggregated accuracy. Moreover, Figure 1 (c) reveals that more than 12% of correctly classified samples are misclassified at some intermediate timesteps, and some samples are never correctly classified at any individual timestep but still become correct after temporal aggregation. These observations suggest that a timestep should not be judged solely by whether it is already correct, but by whether it contributes useful evidence to the final temporal accumulation. Figure 1 : Mismatch between intermediate and final predictions in SNNs. (a) A toy example where intermediate predictions are wrong, but the final aggregated prediction is correct. (b) Per-timestep accuracy and final aggregated accuracy on CIFAR100. (c) Distribution of CIFAR100 samples with correct final predictions by the number of correctly predicted timesteps out of T . However, existing timestep-wise distillation strategies do not explicitly account for this distinction. By encouraging each timestep to align with the teacher signal, they may overlook what corrective evidence is actually needed for improving temporal accumulation and which temporal sources can provide reliable support. This perspective leads to two key questions for timestep-wise distillation: what corrective evidence should be injected into a currently erroneous timestep , and from which temporal sources should it be drawn ? For a currently erroneous timestep dominated by a confusing wrong class, directly transferring teacher preference over the full class distribution may dilute the needed correction and interfere with the adjustment of the most critical class relation, namely that between the ground-truth class and the wrongly favored class. Moreover, in temporal self-distillation, not all source timesteps are equally reliable: some provide confident and compatible evidence, while others may introduce noisy or misleading temporal signals. To address these issues, we propose SeAl-KD , a selective distillation framework with two components. Error-aware Logit Alignment (ELA) refines the class evidence received by erroneous timesteps, while Selective Temporal Alignment (STA) emphasizes reliable and compatible source timesteps during alignment. Figure 2 : SeAl-KD framework. SNNs learn from the same copied ANN output across timesteps. ELA equalizes the true and predicted-false logits at erroneous timesteps before teacher–student KL. STA reweights temporal KL using confidence and inter-timestep similarity. C denotes the ground-truth class. Left-bottom illustration shows that our method aims to follow temporal diversity to obtain correct predictions, even when some intermediate predictions are incorrect. Our main contributions are summarized as follows: (1) We reveal that erroneous timesteps are prevalent in SNNs and show that intermediate misclassifications do not necessarily impair the final prediction under temporal aggregation, exposing the mismatch between consistency and temporal evidence accumulation. (2) We propose SeAl-KD, a selective KD framework that improves corrective supervision for erroneous timesteps through ELA and STA. We further provide theoretical analysis to justify the proposed selective alignment. (3) We conduct extensive experiments on both static and neuromorphic image datasets. The results show that SeAl-KD preserves richer temporal distributions across timesteps and consistently improves performance. 2 Related Works 2.1 Knowledge Distillation for SNNs KD improves SNN training by transferring final outputs, logits, or features from ANNs to narrow the gap with real-valued ANNs ( 22 ; 11 ). Along this line, distillation strategies have evolved from global supervision on temporally aggregated outputs ( 30 ) to timestep-wise supervision that aligns logits or representations at each timestep ( 29 ; 28 ). In addition, self-distillation and temporal consistency regularization further exploit the student’s temporal structure to align outputs across timesteps ( 20 ; 34 ). Despite these advances, most existing methods impose supervision on different timesteps in a largely uniform manner, without distinguishing whether a timestep is currently erroneous, what corrective information it actually needs, or whether the temporal source used for supervision is reliable. 2.2 Temporal Discrepancy in SNNs Temporal discrepancy is intrinsic to SNNs, as membrane integration and spike-triggered resets cause neuronal states and predictions to evolve across timesteps ( 1 ). Prior work has improved temporal modeling and training stability through learnable membrane dynamics ( 7 ), temporal normalization ( 32 ; 6 ), and surrogate-gradient design ( 14

Read the complete text