Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2

Steven Abreu, Sumit Bam Shrestha, Rui-Jie Zhu, Jason Eshraghian · 2025

Read Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2 by Steven Abreu, Sumit Bam Shrestha, Rui-Jie Zhu, Jason Eshraghian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18002.txt.

Introduction

Steven Abreu and co-authors run language models on a brain-like chip in a 2025 paper. Big models burn vast power on stock gear. Intel's Loihi 2 chip works by spikes, with low-bit math and state held on chip. Their build drops matrix multiplication from the model frame. A 370-million-part model, shrunk to low bits on GPU, lost no score in tests.

Early chip reads claim up to three times the flow at half the power of an edge GPU. Scale climbs in their favor as models grow. More chip craft should widen the gap, they say. The aim is long text at low cost on small gear. Brain-form chips may host that path.

Catalog record

CREATORSteven Abreu, Sumit Bam Shrestha, Rui-Jie Zhu, Jason Eshraghian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18002

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2 Steven Abreu, Sumit Bam Shrestha, Rui-Jie Zhu, Jason Eshraghian Abstract Large language models (LLMs) deliver impressive performance but require large amounts of energy. In this work, we present a MatMul‐free LLM architecture adapted for Intel’s neuromorphic processor, Loihi 2. Our approach leverages Loihi 2’s support for low-precision, event-driven computation and stateful processing. Our hardware-aware quantized model on GPU demonstrates that a 370M-parameter MatMul-free model can be quantized with no accuracy loss. Based on preliminary results, we report up to 3 × \times higher throughput with 2 × \times less energy, compared to transformer-based LLMs on an edge GPU, with significantly better scaling. Further hardware optimizations will increase throughput and decrease energy consumption. These results show the potential of neuromorphic hardware for efficient inference and pave the way for efficient reasoning models capable of generating complex, long-form text rapidly and cost-effectively. 1 Introduction Large language models (LLMs) have revolutionized machine learning—but their computational and energy demands are enormous. This challenge motivates the development of efficient and scalable foundation models that are optimized not only algorithmically but co-designed with novel hardware architectures. In this paper, we propose a hardware-aware approach that integrates an efficient LLM architecture with Intel’s neuromorphic processor, Loihi 2 ( Davies et al., 2021 ) . Although originally designed for event-based, sparse computations in spiking neural networks, Loihi 2’s support for low-precision arithmetic and unstructured weight sparsity makes it an attractive platform for reducing energy consumption and latency in LLM inference. See Appendix A.1 for more details on Loihi 2. Although LLMs have been dominated by self-attention ( Vaswani et al., 2017 ) with quadratic runtime complexity, LLMs based on state space models (SSMs) offer linear scaling with competitive performance ( Gu & Dao, 2023 ) . SSMs rely on element-wise recurrence ( Gupta et al., 2022 ) , and use stateful neurons that align well with compute-near-memory architectures like Loihi 2. Advances in quantization of LLMs ( Dettmers et al., 2022 ; Frantar et al., 2023 ; Xiao et al., 2024 ) have culminated in extreme quantization at scale with LLMs using binary activations ( Zhu et al., 2023 ) or ternary weight matrices as seen in BitNet ( Ma et al., 2024 ) , piecewise affine transformers ( Kosson & Jaggi, 2023 ) , ShiftAddLLM ( You et al., 2024 ) , and earlier work on binarized neural networks ( Courbariaux et al., 2016 ) . Building on these ideas, Zhu et al. (2024) introduced the MatMul-free LLM, which replaces traditional matrix multiplications (MatMuls) with ternary matrices and element-wise operations, while also using a subquadratic SSM layer based on the HGRN model ( Qin et al., 2023 ; Qin et al., 2024 ) . Loihi 2 is optimized for sequence-based processing, element-wise recurrence, low-precision arithmetic, and weight sparsity, all of which are features of the MatMul-free model. These benefits have been demonstrated on signal processing tasks ( Orchard et al., 2021b ; Shrestha et al., 2024 ) with orders of magnitude better latency and efficiency than state-of-the-art solutions. This paper presents a work-in-progress of adapting and deploying the MatMul-free language model from Zhu et al. (2024) to Intel Loihi 2, opening a pathway that bridges neuromorphic computing with state-of-the-art efficient LLMs. This required a microcode implementation to map the MMF model to Loihi 2’s architecture, along with a detailed ablation study to evaluate the optimal bit-precision for all operators in the MMF model. We are able to run the MMF model fully on-chip, using fixed-point arithmetic to optimize for energy and latency. 2 Model Architecture The model architecture is based on the 370M parameter 1 1 1 Available on HuggingFace: huggingface.co/ridger/MMfreeLM-370M . MatMul-free language model ( Zhu et al., 2024 ) . It uses a combination of ternary weights and specialized layers to replace all matrix multiplications with additions, bit shifts, and elementwise operators. The overall model architecture follows the Metaformer ( Yu et al., 2022 ) paradigm, consisting of alternating token mixers and channel mixers. Figure 1 provides a high-level overview of this structure. Figure 1: Model architecture of the MatMul-free language model from Zhu et al. (2024) . The model introduces two key innovations. The BitLinear Layer combines a ternarized linear transformation with a preceding RMSNorm operation, to stabilize the activation distribution ( Ma et al., 2024 ; Zhang et al., 2023 ; Zhu et al., 2024 ) . Formally, the input vector x\in\mathbb{R}^{d} (where d=1,024 for the 370M model), \mathbf{W}\in\{-c,0,c\}^{d\times d^{\prime}} is the ternary weight matrix with a scaling factor c\in\mathbb{R} . The RMSNorm ( Zhang & Sennrich, 2019 ) and subsequent BitLinear layer are implemented as: \displaystyle\text{RMSNorm}(x;g,\epsilon) \displaystyle=\frac{x}{\sqrt{\epsilon+\sum_{i}^{d}x_{i}^{2}}}\odot g (1) \displaystyle\text{BitLinear}(x;\mathbf{W},g,\epsilon) \displaystyle=\text{RMSNorm}(x;g,\epsilon)\circledast\mathbf{W} (2) where \odot denotes element-wise multiplication, \epsilon=10^{-6} is a small constant, g\in\mathbb{R}^{d} is a learned scaling parameter, and \circledast performs the accumulation of ternary weights and inputs 2 2 2 As shown by Kosson & Jaggi (2023) , this can be done using only additions and negations. . The use of ternary weights naturally leads to synaptic sparsity; the 370M MMF model has 35.4\%\pm 2.5\% weights of magnitude zero across all ternary weight matrices 3 3 3 Note that this synaptic sparsity does not yield memory savings because sparse encoding of a 1024\times 1024 ternary matrix takes \approx 7.5\times more memory than dense encoding. However, zero weights do allow for energy savings by skipping calculations. . The MatMul-free language model further uses a BitLinear version of the GLU (Gated Linear Unit ( Dauphin et al., 2017 ) ): \displaystyle g_{t}=x_{t}\circledast\mathbf{W}_{g},\quad u_{t}=x_{t}\circledast\mathbf{W}_{u},\quad p_{t}=\tau(g_{t})\odot u_{t},\quad d_{t}=p_{t}\circledast\mathbf{W}_{d} where g_{t},u_{t},p_{t}\in\mathbb{R}^{d} are intermediate activations, and \tau denotes the SiLU activation function \tau(x)=x\odot\sigma(x) where \sigma(x)=1/(1+e^{x}) is the sigmoid function. The MLGRU (MatMul-free Linear Gated Recurrent Unit) acts as the token mixer, replacing the computationally expensive self-attention mechanism in traditional transformers. It utilizes a variant of the GRU ( Cho et al., 2014 ) , inspired by the HGRN ( Qin et al., 2023 ; Qin et al., 2024 ) , modified to use only additions and element-wise products. This is achieved by employing BitLinear layers for all linear transformations within the MLGRU cell. Formally, we denote h_{t}\in\mathbb{R}^{d} as the hidden state at time t and x_{t}\in\mathbb{R}^{d} as the input at time t . The dynamics of the MLGRU layer at time step t are given by: \displaystyle f_{t} \displaystyle=\sigma\left(\text{BitLinear}(x_{t};\mathbf{W}_{f},g_{f},\epsilon)\right), \displaystyle c_{t} \displaystyle=\tau\left(\text{BitLinear}(x_{t};\mathbf{W}_{c},g_{c},\epsilon)\right), \displaystyle h_{t} \displaystyle=f_{t}\odot h_{t-1}+(1-f_{t})\odot c_{t}, \displaystyle g_{t} \displaystyle=\text{RMSNorm}\left(\text{BitLinear}(x_{t};\mathbf{W}_{g},g_{g},\epsilon);g_{g^{\prime}},\epsilon\right), \displaystyle o^{\prime}_{t} \displaystyle=g_{t}\odot\tau(h_{t}), \displaystyle o_{t} \displaystyle=\text{BitLinear}(o^{\prime}_{t};\mathbf{W}_{o},g_{o},\epsilon) where \mathbf{W}_{c},\mathbf{W}_{f},\mathbf{W}_{o},\mathbf{W}_{g}\in\mathbb{R}^{d\times d} are ternary weights, f_{t},g_{t},c_{t},o^{\prime}_{t} are intermediate activations at time step t , h_{t} is the hidden state, and o_{t} is the final output at time step t . The initial hidden state h_{0} is set to zero. Similarly to the HGRN ( Qin et al., 2023 ) , the MLGRU also employs the cummax operation to bias the forget gate values in deeper layers closer to 1, which we omit for brevity. 3 Model Adaptation for Loihi 2 Quantization of weights and activations As a first step, we quantize the original half-precision model from Zhu et al. (2024) . Loihi 2 supports 8-bit weights and 24-bit activations. As done by previous work ( Ma et al., 2024 ; Zhu et al., 2024 ) , we evaluate the

Read the complete text