SpikGPT: A High-Accuracy and Interpretable Spiking Attention Framework for Single-Cell Annotation

Min Huang, Rishikesan Kamaleswaran · 2025

Read SpikGPT: A High-Accuracy and Interpretable Spiking Attention Framework for Single-Cell Annotation by Min Huang, Rishikesan Kamaleswaran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03286.txt.

Introduction

Naming cell types from single-cell data is a key step in biology. Min Huang and Rishikesan Kamaleswaran present SpikGPT for that task in this 2025 work. Annotation gets hard when data sets differ by batch or hold cell types never seen before.

SpikGPT blends cell embeddings from a model called scGPT with a spiking Transformer design. Spiking networks fire in brief pulses, like real neurons, which can save energy. The authors aim for accuracy that is also easy to interpret. Trustworthy labels help researchers read tissue and disease data correctly. Batch effects fool many tools, so robustness across labs is a central test for SpikGPT.

Catalog record

CREATORMin Huang, Rishikesan Kamaleswaran
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-03286

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SpikGPT: A High-Accuracy and Interpretable Spiking Attention Framework for Single-Cell Annotation Min Huang, Rishikesan Kamaleswaran Abstract Accurate and scalable cell type annotation remains a challenge in single-cell transcriptomics, especially when datasets exhibit strong batch effects or contain previously unseen cell populations. Here we introduce SpikGPT, a hybrid deep learning framework that integrates scGPT-derived cell embeddings with a spiking Transformer architecture to achieve efficient and robust annotation. scGPT provides biologically informed dense representations of each cell, which are further processed by a multi-head Spiking Self-Attention mechanism, energy-efficient feature extraction. Across multiple benchmark datasets, SpikGPT consistently matches or exceeds the performance of leading annotation tools. Notably, SpikGPT uniquely identifies unseen cell types by assigning low-confidence predictions to an “Unknown” category, allowing accurate rejection of cell states absent from the training reference. Together, these results demonstrate that SpikGPT is a versatile and reliable annotation tool capable of generalizing across datasets, resolving complex cellular heterogeneity, and facilitating discovery of novel or disease-associated cell populations. Keywords scRNA-seq annotation \cdot Spiking attention \cdot transformer 1 Introduction Single-cell RNA sequencing (scRNA-seq) has powerful ability to dissect cellular heterogeneity, enabling the profiling of gene expression at single-cell resolution across diverse tissues and conditions 12 4 8 . A critical step in scRNA-seq analysis is cell-type annotation by clustering, which assigns biological meaning to expression profiles and lays the foundation for downstream interpretation and discovery 3 14 . In traditional way, researchers label cell types manually based on the clustering and prior knowledge 15 7 . However, labor annotation is time consuming and non-reproducible. Therefore, scRNA-seq cell-type annotation tools were developed and achieved superior performance. Despite numerous advances, automated cell-type annotation remains challenging due to the high dimensionality, sparsity, and noise inherent in scRNA-seq data. Conventional machine learning methods often struggle to generalize across datasets 2 13 9 1 , while deep learning models can be computationally intensive and lack biological interpretability 16 11 . Recent transformer-based approaches have shown promise by learning rich embeddings from scRNA-seq data 5 17 . However, these models are not optimized for sparse biological signals and often require extensive computational resources for training and inference. To address these limitations, we developed a novel cell-type annotation framework called SpikGPT that integrates the strengths of biological language modeling with biologically inspired attention mechanisms. Our model is predicated on the paradigm of biological language models. This approach treats gene expression profiles as structured sequences, analogous to sentences in natural language, thereby facilitating the learning of context-aware representations that intrinsically capture the co-expression relationships and underlying regulatory logic among genes. By leveraging this representation paradigm, we utilize scGPT-generated 6 embeddings to encode rich semantic information from single-cell gene expression data. To further enhance computational efficiency and biological plausibility, Spiking Self-Attention (SSA) mechanism that mimics the sparse, event-driven nature of neuronal firing were introduced. Our multi-head spiking attention module processes the scGPT embeddings through dynamic spiking neuron models, enabling sparse, energy-efficient, and interpretable information flow. This hybrid architecture effectively combines the sequence modeling power of language models with biologically grounded sparse computation, yielding a model that is both accurate and computationally lightweight. We demonstrate that our approach, SpikeGPT, significantly outperforms existing annotation tools in both accuracy and computational efficiency across multiple benchmark scRNA-seq datasets. Crucially, SpikeGPT exhibits superior generalization capability for predictions on external and heterogeneous datasets, affirming its robustness. Furthermore, by learning deep, universal representations of cellular states, our model possesses the unique ability to accurately predict and characterize novel or ’unseen’ cell types not present in the training data, overcoming a major limitation of conventional methods. By grounding its design in biological language and neural computation principles, SpikeGPT offers a powerful, highly generalizable, and interpretable solution for large-scale single-cell analysis. 2 Method SpikGPT model We propose SpikGPT, which incorporates the scGPT embedding into Spiking self attention. For each cell, gene expression values from the scRNA-seq matrix are first embedded into a dense vector using scGPT, a transformer-based model pretrained on large-scale single-cell datasets. This embedding captures the global transcriptional profile of the cell and serves as input to our annotation framework. E=scGPT(X) (1) Where X\in\mathbb{R}^{n\times g} is the original scRNA-seq matrix, with n cells and g genes, E\in\mathbb{R}^{n\times k} is the resulting dense embedding matrix, where each cell is represented by a k-dimensional feature vector. After obtaining the cell embedding matrix E\in\mathbb{R}^{n\times k} from scGPT, we apply two expansion steps to adapt the data for input into the Spikformer model and to enhance its capacity for robust and expressive learning. First, each individual cell embedding e_{i}\in\mathbb{R}^{k} is repeated m times along a new axis, resulting in a tensor E^{\prime}\in\mathbb{R}^{n\times m\times k} , where m is a hyper-parameter that can be manually set, with a default of 300. This expansion allows the model to construct multiple parallel feature representations for each cell, increasing the richness and flexibility of representation. Although the repeated vectors are initially identical, subsequent non-linear transformations (such as spiking activations and attention mechanisms) introduce diversity and enable independent adaptation across feature channels. Second, the entire E^{\prime} matrix is repeated T times along a new dimension, forming the final input tensor E^{\prime\prime}\in\mathbb{R}^{T\times n\times m\times k} . T is a hyper-parameter that can be manually set, with a default of 4. This dimension can be interpreted as multiple stochastic views of the same dataset, promoting robustness and generalization by allowing the model to learn invariant features across repeated input exposures. Together, these two expansion steps enhance the model’s ability to represent complex cellular states and improve training stability under sparse and spiking activation dynamics. The final input tensor E^{\prime\prime}\in\mathbb{R}^{T\times n\times m\times k} , obtained through the aforementioned embedding expansion, is then fed into the Spiking Self-Attention (SSA) module, which is organized into H parallel attention heads. For each attention head i (i=1,2,\dots,H) , the Query ( Q_{i} ), Key ( K_{i} ), and Value ( V_{i} ) matrices are computed using separate learnable linear projections: Q_{i}=E^{\prime\prime}W_{Q}^{(i)},\quad K_{i}=E^{\prime\prime}W_{K}^{(i)},\quad V_{i}=E^{\prime\prime}W_{V}^{(i)} (2) where W_{Q}^{(i)},W_{K}^{(i)},W_{V}^{(i)}\in\mathbb{R}^{k\times d_{h}} are the projection weight matrices for the i -th attention head, and d_{h} denotes the dimensionality of each head. Using independent projections per head allows the model to capture diverse patterns across different subspaces, which helps stabilize training and improve representational capacity. Next, the projected matrices pass through distinct spiking neuron layers to generate sparse, spike-form representations. To introduce biologically inspired sparse activation into the attention mechanism, we adopt Leaky Integrate-and-Fire (LIF) neurons in all spiking neuron layers of SpikGPT. Specifically, each Query ( Q_{i} ), Key ( K_{i} ), and Value ( V_{i} ) matrix is passed through a corresponding spike neuron layer—denoted as \text{SN}_{Q} , \text{SN}_{K} , \text{SN}_{V} —before computing the attention outputs: Q_{i}^{\prime}=\text{SN}_{Q}(Q_{i}),\quad K_{i}^{\prime}=\text{SN}_{K}(K_{i}),\quad V_{i}^{\prime}=\text{SN}_{V}(V+i) (3) The LIF neurons are implemented using the SpikingJelly framework. For parameters of it, the time constant \tau=2.0 controls the rate of membrane potential decay. The step_mode=’m’ setting enables multi-step (temporal) mode, allowing the neuron to integrate inputs across multiple discrete time steps. All spikes are binary, temporally sparse events, enhancing both computational efficiency and model generalizability. The spiking self-attention operation is then computed as: \text{SSA}_{i}=\text{SN}(Q_{i}^{\prime}K_{i}^{\prime T}V_{i}^{\prime}) (4) Here, SN denotes the spiking neuron function applied to the output of each attention head. For each of the H attention heads, the query ( Q_{i}^{\prime} ), key

Read the complete text