TEFormer: Structured Bidirectional Temporal Enhancement Modeling in Spiking Transformers

Sicheng Shen, Mingyang Lv, Bing Han, Dongcheng Zhao, Guobin Shen, Feifei Zhao, Yi Zeng · 2026

Read TEFormer: Structured Bidirectional Temporal Enhancement Modeling in Spiking Transformers by Sicheng Shen, Mingyang Lv, Bing Han, Dongcheng Zhao, Guobin Shen, Feifei Zhao, Yi Zeng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18274.txt.

Introduction

Sicheng Shen and co-authors add a temporal module to spiking transformers. Their 2026 preprint presents TEFormer. Spiking neural networks promise energy-efficient sequence modeling, and spiking transformers are a rising design in that space. Yet they lack a principled way to fuse information across time. That gap limits how well they use spatiotemporal patterns. TEFormer adds structured bidirectional temporal enhancement, letting each moment draw on both past and future context.

Bidirectional flow suits spikes, whose meaning often depends on what follows. Better temporal fusion could close the gap with standard transformers. It would do so at a fraction of the energy cost. Efficient sequence models matter for edge devices and sensors.

Catalog record

CREATORSicheng Shen, Mingyang Lv, Bing Han, Dongcheng Zhao, Guobin Shen, Feifei Zhao, Yi Zeng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18274

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TEFormer: Structured Bidirectional Temporal Enhancement Modeling in Spiking Transformers Sicheng Shen, Mingyang Lv, Bing Han, Dongcheng Zhao, Guobin Shen, Feifei Zhao, Yi Zeng Abstract In recent years, Spiking Neural Networks (SNNs) have achieved remarkable progress, with Spiking Transformers emerging as a promising architecture for energy-efficient sequence modeling. However, existing Spiking Transformers still lack a principled mechanism for effective temporal fusion, limiting their ability to fully exploit spatiotemporal dependencies. Inspired by feedforward–feedback modulation in the human visual pathway, we propose TEFormer , the first Spiking Transformer framework that achieves bidirectional temporal fusion by decoupling temporal modeling across its core components. Specifically, TEFormer employs a lightweight and hyperparameter-free forward temporal fusion mechanism in the attention module , enabling fully parallel computation, while incorporating a backward gated recurrent structure in the MLP to aggregate temporal information in reverse order and reinforce temporal consistency. Extensive experiments across a wide range of benchmarks demonstrate that TEFormer consistently and significantly outperforms strong SNN and Spiking Transformer baselines under diverse datasets. Moreover, through the first systematic evaluation of Spiking Transformers under different neural encoding schemes, we show that the performance gains of TEFormer remain stable across encoding choices, indicating that the improved temporal modeling directly translates into reliable accuracy improvements across varied spiking representations. These results collectively establish TEFormer as an effective and general framework for temporal modeling in Spiking Transformers. 1 Introduction Spiking Neural Networks (SNNs) are increasingly regarded as a promising foundation for next-generation artificial intelligence. By representing information with sparse binary spikes, SNNs inherently offer advantages in energy efficiency, computational throughput, and hardware compatibility ( 22 ; 45 ) , making them particularly attractive for neuromorphic intelligence. This potential has been further amplified by recent advances in neuromorphic processors ( 28 ) and event-driven sensing technologies ( 13 ) , which provide both efficient computing substrates and natural input modalities for spiking computation. Together, these developments elevate SNNs from purely biologically inspired models to a practical and scalable paradigm for large-scale intelligent systems. Early SNN models were primarily driven by neuroscience insights and biologically faithful designs ( 5 ; 46 ; 7 ) . While such approaches offer strong interpretability, they face substantial challenges in scaling to deep architectures and supporting efficient training. To overcome these limitations, subsequent research increasingly adopted architectural advances from ANNs, leading to spiking counterparts of mainstream models such as CNNs, RNNs, and GNNs ( 10 ; 15 ; 21 ; 54 ) . These efforts significantly improved performance and scalability, accelerating the development of modern SNN architectures. Despite these advances, most existing SNN architectures remain task-specific and lack a unified modeling paradigm. In contrast, Transformers exhibit strong scalability, effective sequence modeling capability, and natural extensibility to multimodal and large-scale settings ( 35 ; 8 ) . Introducing this paradigm into the spiking domain is therefore a critical step toward general-purpose neuromorphic intelligence. This motivation has led to the emergence of Spiking Transformers, which aim to combine the sparse, event-driven computation of SNNs with the expressive modeling capacity of Transformers. Since the introduction of Spikformer ( 52 ) , a growing body of work has steadily enriched this research direction, including the Spike-driven Transformer family ( 42 ; 41 ) , DISTA ( 38 ; 39 ) , QKFormer ( 49 ) , and Spike2Former ( 20 ) . More recently, STEP ( 31 ) has provided a unified evaluation framework, promoting reproducibility and fair comparison. However, while Spiking Transformers have shown promising performance, their ability to model temporal dependencies remains limited. Recent works, such as TIM ( 32 ) , STAtten ( 19 ) , and ST-SSA ( 51 ) , attempt to enhance temporal dynamics by introducing explicit temporal aggregation or recurrent operations within the attention mechanism. These designs either rely on heuristic, task-sensitive hyperparameters or impose sequential dependencies across time steps, thereby compromising the inherent parallelism of attention and increasing training and inference latency. In contrast, the human visual pathway exhibits pervasive feedforward–feedback modulation, where temporal information is integrated in a fundamentally bidirectional manner ( 29 ; 30 ) . The lack of mechanisms that reflect this property suggests that temporal modeling in Spiking Transformers remains insufficiently explored, motivating more principled designs that can capture bidirectional temporal dependencies without sacrificing computational efficiency. To address these challenges, we propose a Temporal Enhanced Spiking Transformer (TEFormer) , which adopts a structured and lightweight temporal modeling strategy. Instead of introducing sequential operations into attention, TEFormer integrates a hyperparameter-free module into spiking attention to enable parallel forward temporal fusion , preserving computational efficiency. In addition, we redesign the upsampling pathway of the multi-layer perceptron to support backward temporal tracing , complementing forward attention with reverse-time information aggregation. Together, these components form a unified framework for bidirectional temporal fusion in Spiking Transformers, advancing temporal modeling beyond prior unidirectional and heuristic approaches. The main contributions of this work are summarized as follows: • We propose a Temporal Enhanced Attention (TEA) module that enables parallel forward temporal fusion in spiking attention without additional hyperparameters, improving temporal modeling while preserving Transformer efficiency. • We introduce a gated temporal mechanism in a restructured MLP ( T-MLP ) to support backward temporal tracing, which complements forward attention and enables bio-inspired effective bidirectional temporal fusion in Spiking Transformers. • Extensive experiments across multiple datasets and neuron encoding schemes demonstrate that the proposed bidirectional temporal fusion yields robust and transferable performance gains, validating the effectiveness of TEFormer. 2 Related Work 2.1 Temporal Enhancement in SNNs Enhancing temporal dynamics is a central theme in improving information processing in spiking neural networks. At the neuron level, prior work has focused on strengthening temporal integration by introducing learnable dynamics. PLIF ( 11 ) incorporates learnable membrane time constants to enhance temporal accumulation, while IIRSNN ( 9 ) improves spatiotemporal processing through explicit synaptic modeling. In addition, a series of LIF neuron variants with learnable dynamics further extend the capability of modeling long-term dependencies. At the network level, TA-SNN ( 40 ) reinforces temporal information by adaptively weighting time steps, TCJA ( 53 ) proposes joint attention for adaptive temporal modeling, and STSC-SNN ( 43 ) combines temporal convolution with attention mechanisms, offering network-level solutions to temporal enhancement. Temporal enhancement in Spiking Transformers has been explored in only a few representative works. TIM ( 32 ) strengthens temporal information via step-wise convolution on queries, while STAtten ( 19 ) and ST-SSA ( 51 ) design specialized attention mechanisms to improve temporal expressiveness. Despite their effectiveness, these methods share common characteristics: temporal information is mainly propagated in a forward direction, key design choices rely on heuristic and dataset-sensitive hyperparameters, and temporal modeling is tightly coupled with attention computation, which may limit parallel efficiency. These observations indicate that temporal modeling in Spiking Transformers remains an open and evolving research direction. 2.2 Frontiers in Spiking Transformers Brain-inspired Spiking Neural Networks (SNNs) emulate spike-based information transmission and synaptic plasticity mechanisms of biological neural systems, enabling low power consumption and high computational efficiency. These characteristics make SNNs naturally well suited for neuromorphic hardware, while the introduction of surrogate gradient methods has made the training of large-scale SNN models feasible ( 24 ) . As the field matures, Spiking Transformers have emerged as a promising direction, integrating global modeling

Read the complete text