Modern Neuromorphic AI: From Intra-Token to Inter-Token Processing

Osvaldo Simeone · 2026

Read Modern Neuromorphic AI: From Intra-Token to Inter-Token Processing by Osvaldo Simeone (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00245.txt.

Introduction

Osvaldo Simeone, writing alone, reviews how modern AI quietly absorbs brain inspired design ideas. AI growth has driven energy costs sharply upward. Neuromorphic computing promises brain like efficiency in return. Its tools include sparse, discrete spikes and feedback loops. Newer AI models already echo those traits. Quantized activations and state space dynamics are two examples.

His lens splits work inside one token from work across tokens. Early spiking networks handled the inside token case. They remixed channels of a single input vector. Recent work pushes the same ideas across tokens. Sparse attention and state space memory pick what to combine. Associative memory sits at the core of that shift. Training methods get a full review as well. They range from surrogate gradients to local reinforcement rules. The paper maps one field onto another.

Catalog record

CREATOROsvaldo Simeone
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00245

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modern Neuromorphic AI: From Intra-Token to Inter-Token Processing Osvaldo Simeone Abstract The rapid growth of artificial intelligence (AI) has brought novel data processing and generative capabilities but also escalating energy requirements. This challenge motivates renewed interest in neuromorphic computing principles, which promise brain-like efficiency through discrete and sparse activations, recurrent dynamics, and non-linear feedback. In fact, modern AI architectures increasingly embody neuromorphic principles through heavily quantized activations, state-space dynamics, and sparse attention mechanisms. This paper elaborates on the connections between neuromorphic models, state-space models, and transformer architectures through the lens of the distinction between intra-token processing and inter-token processing. Most early work on neuromorphic AI was based on spiking neural networks (SNNs) for intra-token processing, i.e., for transformations involving multiple channels, or features, of the same vector input, such as the pixels of an image. In contrast, more recent research has explored how neuromorphic principles can be leveraged to design efficient inter-token processing methods, which selectively combine different information elements depending on their contextual relevance. Implementing associative memorization mechanisms, these approaches leverage state-space dynamics or sparse self-attention. Along with a systematic presentation of modern neuromorphic AI models through the lens of intra-token and inter-token processing, training methodologies for neuromorphic AI models are also reviewed. These range from surrogate gradients leveraging parallel convolutional processing to local learning rules based on reinforcement learning mechanisms. I Introduction Modern artificial intelligence (AI) tools are widely believed to have the potential to transform our economies and societies, yet this progress comes at a well-documented steep cost in terms of energy resources for both training [ 15 ] and inference [ 54 ] . The quest for a large “intelligence-to-joule ratio” [ 50 ] has long been the objective of neuromorphic engineering , which takes inspiration from the most energy-efficient intelligent system known: the human brain [ 40 ] . Over more than three decades of research, neuromorphic engineering has identified a number of principles from neuroscience that may be responsible for the energy efficiency of the brain [ 17 , 47 ] , including (see Figure 1 ): sparse communication through discrete spikes (i.e., binary signals) with event-driven processing, dynamic state-space computations in neuronal membranes, analog in-memory computing, massively parallel computing, and local learning rules. Fig. 1: Over more than three decades of research, neuromorphic engineering has identified a number of principles from neuroscience that may be responsible for the energy efficiency of the brain, including: sparse communication through discrete spikes (i.e., binary signals) with event-driven processing, dynamic state-space computations in neuronal membranes, analog in-memory computing, massively parallel computing, and local learning rules. Fig. 2: Modern AI systems pre-process text – and similarly other types of data – via tokenization and embedding. These steps produce a matrix \bm{E} with N columns (here N=5 ), each corresponding to a token embedding containing D channels. Recent developments in mainstream AI have incorporated a number of lessons from neuromorphic engineering [ 47 ] . Most notably, state-of-the-art large language models (LLMs) have been demonstrated with activations quantized to integers with resolutions as low as 4 bits [ 39 , 38 ] – moving towards the neuromorphic regime of spike-based (i.e., binary) communication. Furthermore, some of the best-performing sequence-to-sequence models are designed based on state dynamics [ 26 , 25 ] – thus reproducing the stateful operation of typical neuromorphic models. This convergence reflects fundamental computational principles: efficient systems must exploit temporal structure if present and avoid unnecessary computation through sparsity. Despite the outlined convergence of AI along neuromorphic principles, the research on neuromorphic AI has remained relatively separate from mainstream AI. The state of the art up to around 2022 is well summarized in a number of tutorial papers, including the November 2019 special issue of IEEE Signal Processing Magazine on “Learning Algorithms and Signal Processing for Brain-Inspired Computing” [ 41 , 34 , 31 , 46 ] , and later reviews on event-driven sensing [ 13 ] and neuroscience-inspired learning algorithms [ 44 ] . Other surveys have organized the field around algorithm-hardware co-design for spike-based machine intelligence [ 49 ] , and specific platforms such as Intel’s Loihi processor [ 17 ] . These reviews have established neuromorphic computing as a mature field with demonstrated applications in vision, robotics, and signal processing. However, the last few years have seen an important shift in the design of neuromorphic AI platforms, which is highlighted in this article. As illustrated in Figure 2 , modern AI models, including transformers and state-space models (SSMs), operate on sequences of tokens – discrete informational units representing a (sub)word or an image patch as a vector consisting of multiple features or channels . Early work on neuromorphic AI has focused on the task of intra-token processing, i.e., of transforming individual tokens by combining different features, targeting applications such as image classification. In contrast, recent research points to the potential of neuromorphic principles for sequence-to-sequence mapping via inter-token processing. This ongoing shift is the main focus of this article. Intra-token processing (Figure 3 ) transforms individual tokens independently by mixing information across channels within each token. This type of operation is exemplified by linear layers and, more broadly, conventional neural networks. Early neuromorphic systems, summarized in the previous reviews mentioned above, address this type of computation, focusing on tasks such as image classification. In these systems, the input data are represented and processed via sparse spiking (binary) signals defined along a virtual time axis – an internal computational dimension (see Figure 1 ). Typical results, sketched in Figure 4 and exemplified in Table I , show that, as computations are carried out along this virtual time axis, neuromorphic systems consume more energy, processing more and more spikes, eventually approaching the energy consumption of a conventional system based on standard neural networks. The benefits of neuromorphic processing in this context materialize in a flexible trade-off between accuracy and energy: running the neuromorphic system for a shorter (virtual) time may cause a potentially small loss in accuracy but at much reduced energy consumption. Inter-token processing (Figure 5 ) mixes information between different tokens in a sequence, separately for each channel. Notably, in transformers, self-attention mechanisms implement inter-token processing by computing pairwise relationships between all tokens; while in SSMs, state-space dynamic updates carry information from past to future tokens. Recent work has explored the adoption of neuromorphic principles for the implementation of inter-token processing, encompassing both self-attention [ 67 , 55 , 62 , 42 ] and SSMs [ 56 , 5 , 65 ] . In such systems, as illustrated in the bottom part of Figure 4 and exemplified in Table II , improved energy-accuracy trade-offs arise from the sparsity of the inner representations leveraged by neuromorphic AI systems. This article traces the evolution of neuromorphic systems from intra-token processing to inter-token processing, providing a unified introduction to this emerging topic that targets signal processing researchers. I-A Organization This paper is organized as follows. Section II establishes the mathematical foundations of token-based processing, formally defining intra-token and inter-token operations and explaining their role in modern AI architectures. Section III introduces neuromorphic processing units (NPUs), a general model for neuromorphic AI systems, and its constituent blocks referred to as neuromorphic processing elements (NPEs), formulating them as recurrent

Read the complete text