Generalizable, real-time neural decoding with hybrid state-space models

Avery Hee-Woon Ryoo, Nanda H Krishna, Ximeng Mao, Mehdi Azabou, Eva L Dyer, Matthew G Perich, Guillaume Lajoie · 2025

Read Generalizable, real-time neural decoding with hybrid state-space models by Avery Hee-Woon Ryoo, Nanda H Krishna, Ximeng Mao, Mehdi Azabou, Eva L Dyer, Matthew G Perich, Guillaume Lajoie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05320.txt.

Introduction

Avery Hee-Woon Ryoo and co-authors present POSSM, a hybrid model for decoding neural activity in real time, in a 2025 paper. Real-time decoding is central to neuroscience and brain-computer interfaces, where latency limits are strict. Simple recurrent models are fast but generalize poorly to new data.

Transformer models generalize well after large pretraining, but they are heavy and not always fit for low-resource, real-time use. POSSM joins spike tokenization through cross-attention with a recurrent state-space backbone. That pairing aims at fast, causal online prediction and efficient generalization to new sessions and individuals. The design bridges two families.

Catalog record

CREATORAvery Hee-Woon Ryoo, Nanda H Krishna, Ximeng Mao, Mehdi Azabou, Eva L Dyer, Matthew G Perich, Guillaume Lajoie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05320

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generalizable, real-time neural decoding with hybrid state-space models Avery Hee-Woon Ryoo, Nanda H Krishna, Ximeng Mao, Mehdi Azabou, Eva L Dyer, Matthew G Perich, Guillaume Lajoie Abstract Real-time decoding of neural activity is central to neuroscience and neurotechnology applications, from closed-loop experiments to brain-computer interfaces, where models are subject to strict latency constraints. Traditional methods, including simple recurrent neural networks, are fast and lightweight but often struggle to generalize to unseen data. In contrast, recent Transformer-based approaches leverage large-scale pretraining for strong generalization performance, but typically have much larger computational requirements and are not always suitable for low-resource or real-time settings. To address these shortcomings, we present POSSM , a novel hybrid architecture that combines individual spike tokenization via a cross-attention module with a recurrent state-space model (SSM) backbone to enable (1) fast and causal online prediction on neural activity and (2) efficient generalization to new sessions, individuals, and tasks through multi-dataset pretraining. We evaluate POSSM ’s decoding performance and inference speed on intracortical decoding of monkey motor tasks, and show that it extends to clinical applications, namely handwriting and speech decoding in human subjects. Notably, we demonstrate that pretraining on monkey motor-cortical recordings improves decoding performance on the human handwriting task, highlighting the exciting potential for cross-species transfer. In all of these tasks, we find that POSSM achieves decoding accuracy comparable to state-of-the-art Transformers, at a fraction of the inference cost (up to 9 × \times faster on GPU). These results suggest that hybrid SSMs are a promising approach to bridging the gap between accuracy, inference speed, and generalization when training neural decoders for real-time, closed-loop applications. 1 Introduction Neural decoding – the process of mapping neural activity to behavioural or cognitive variables – is a core component of modern neuroscience and neurotechnology. As neural recording techniques evolve and datasets grow in size, there is increasing interest in building generalist decoders that scale and flexibly adapt across subjects and experiments. Several important downstream applications – including closed-loop neuroscience experiments and brain-computer interfaces (BCIs) – require fine-grained, low-latency decoding for real-time control. Advances in these technologies would enable next-generation clinical interventions in motor decoding [ 1 ] , speech prostheses [ 2 ] , and closed-loop neuromodulation [ 3 , 4 ] . Building towards these applications will require neural decoders that meet three requirements: (i) robust and accurate predictions, (ii) causal, low-latency inference that is viable in an online setting, and (iii) flexible generalization to new subjects, tasks, and experimental settings. Although recent developments in machine learning (ML) have enabled significant strides in each of these axes, building a neural decoder that achieves all three remains an open challenge. Figure 1: Existing deep learning models for neural decoding. (a) Recurrent neural networks (RNNs). (b) Attention-based models such as Transformers. Recurrent neural networks (RNNs) [ 5 ] and attention-based models such as Transformers [ 6 ] have shown significant promise for neural decoding tasks [ 7 , 8 , 9 ] . RNNs (Figure 1 a) offer fast, low-latency inference on sequential data and strong performance when trained on specific tasks [ 9 ] . However, their ability to generalize to new subjects is limited due to their rigid input format. Specifically, their reliance on fixed-size, time-binned inputs means that they typically cannot learn from new sessions with different neuron identities or sampling rates without full re-training and/or modifying the architecture. In contrast, Transformer-based architectures (Figure 1 b) offer greater flexibility thanks to more adaptable neural tokenization approaches [ 10 , 11 , 12 ] . Nonetheless, they struggle with applications involving real-time processing due to their quadratic computational complexity, in addition to the overall computational load of the attention mechanism. Recent efforts in sequence modelling with large language models have explored hybrid architectures that combine recurrent layers, such as gated recurrent units (GRUs) [ 13 ] or state-space models (SSMs) [ 14 , 15 , 16 , 17 ] , with attention layers. These models show encouraging gains in long-context understanding and computational efficiency [ 18 , 19 , 20 ] . While hybrid attention-SSM approaches offer a promising solution for real-time neural decoding, to the best of our knowledge, they remain unexplored in this area. We address this gap with POSSM 1 1 1 POSSM stands for POYO -SSM, and is pronounced like the animal (“possum”; IPA: /”p6s9m/). , a hybrid model that combines the flexible input-processing of Transformers [ 21 ] with the efficient, online inference capabilities of a recurrent SSM backbone. Unlike traditional methods that rely on rigid time-binning, POSSM operates at millisecond-level resolution by tokenizing individual spikes. In essence, POSSM builds on a POYO -style cross-attention encoder [ 12 ] that projects a variable number of spike tokens to a fixed-size latent space. The resulting output is then fed to an SSM that updates its hidden state across consecutive chunks in time. This architecture, as illustrated in Figure 2 , offers two key benefits: (i) the recurrent backbone allows for lightweight, constant-time predictions over consecutive chunks of time and (ii) by adopting POYO ’s spike tokenization, encoding, and decoding schemes, POSSM can effectively generalize to different sessions, tasks, and subjects. In this paper, we introduce the POSSM architecture and evaluate its performance on intracortical recordings of spiking activity from experiments in both non-human primates (NHPs) and humans. Although our current evaluations are conducted offline, POSSM is designed for real-time inference and can be readily implemented for online experiments. Finally, while this paper is concerned with invasive electrophysiology recordings, this method could be extended further to other modalities using POYO -style tokenization (see Section 5 for a discussion). Our contributions are summarized as follows: • Performance and efficiency: We evaluate POSSM against other popular models using NHP datasets that contain multiple sessions, subjects, and different reaching tasks (centre-out, random target, maze, etc.). We find that POSSM matches or outperforms other models on all these tests, doing so with greater speed and significantly reduced computational cost. • Multi-dataset pretraining improves performance: Through large-scale pretraining, we find that POSSM delivers improved performance on NHP datasets across sessions, subjects, and even tasks. • Cross-species transfer learning: Pretraining on diverse NHP datasets and then finetuning POSSM leads to state-of-the-art performance when decoding imagined handwritten letters from human cortical activity [ 22 ] . This cross-species transfer not only outlines the remarkable transferability of neural dynamics across different primates, but also shows the potential for leveraging abundant non-human data to augment limited human datasets for improved decoding performance. • Long sequence complex decoding: When trained on human motor-cortical data during attempted speech [ 2 ] , POSSM achieves strong decoding performance. In contrast, attention-based models struggle with the task’s long-context demands, making them computationally prohibitive. Figure 2: An architecture for generalizable, real-time neural decoding. POSSM combines individual spike tokenization [ 11 ] and input-output cross-attention [ 21 ] with a recurrent SSM backbone. In this paper, we typically consider k=3 and T_{c}= 50ms. 2 Methods We focus on decoding neuronal spiking activity, which consists of discrete events triggered when excitatory input to a neuron exceeds a certain threshold. The timing and frequency of spikes encode information conveyed to other neurons throughout the brain, underlying a communication system that is central to

Read the complete text