Training Deep Normalization-Free Spiking Neural Networks with Lateral Inhibition

Peiyu Liu, Jianhao Ding, Zhaofei Yu · 2025

Read Training Deep Normalization-Free Spiking Neural Networks with Lateral Inhibition by Peiyu Liu, Jianhao Ding, Zhaofei Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23253.txt.

Introduction

Deep spiking networks usually lean on normalization, a scaling step brains do not use in that form. Peiyu Liu, Jianhao Ding, and Zhaofei Yu swap it for lateral inhibition, a circuit trait from the cortex. Their layers split into two neuron groups, one that excites and one that calms. The calm side sets how lively cells are and how much they gain. Two tricks steady the training.

One balances the two inputs at the start, with gain control built in. The other splits the training signal flow from the forward pass. Across data sets and designs, deep spiking nets trained in a stable way. Scores stayed close to rivals. The setup also serves as a test bed for cortex study.

Catalog record

CREATORPeiyu Liu, Jianhao Ding, Zhaofei Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training Deep Normalization-Free Spiking Neural Networks with Lateral Inhibition Peiyu Liu, Jianhao Ding, Zhaofei Yu Abstract Spiking neural networks (SNNs) have garnered significant attention as a central paradigm in neuromorphic computing, owing to their energy efficiency and biological plausibility. However, training deep SNNs has critically depended on explicit normalization schemes, such as batch normalization, leading to a trade-off between performance and biological realism. To resolve this conflict, we propose a normalization-free learning framework that incorporates lateral inhibition inspired by cortical circuits. Our framework replaces the traditional feedforward SNN layer with a circuit of distinct excitatory (E) and inhibitory (I) neurons that complies with Dale’s law. The circuit dynamically regulates neuronal activity through subtractive and divisive inhibition, which respectively control the activity and the gain of excitatory neurons. To enable and stabilize end-to-end training of the biologically constrained SNN, we propose two key techniques: E-I Init and E-I Prop. E-I Init is a dynamic parameter initialization scheme that balances excitatory and inhibitory inputs while performing gain control. E-I Prop decouples the backpropagation of the E-I circuits from the forward propagation and regulates gradient flow. Experiments across several datasets and network architectures demonstrate that our framework enables stable training of deep SNNs with biological realism and achieves competitive performance without resorting to explicit normalizations. Therefore, our work not only provides a solution to training deep SNNs but also serves a computational platform for further exploring the functions of lateral inhibition in large-scale cortical computation. 1 Introduction Inspired by computational principles of biological neurons, spiking neural networks (SNNs) stand at the intersection of artificial intelligence and neuroscience ( Maass 1997 ; Ghosh-Dastidar & Adeli 2009 ) . They not only enable highly energy-efficient computation on neuromorphic hardware ( Roy et al. 2019 ; Xiao et al. 2025 ) but also provide models for understanding the cortical computation across multiple scales ( Kumarasinghe et al. 2021 ; Korcsak-Gorzo et al. 2022 ; N’dri et al. 2024 ) . This duality places SNNs at the heart of the emerging field of NeuroAI ( Sadeh & Clopath 2025 ) , fostering a synergy between artificial intelligence and neuroscience. On the one hand, a deeper understanding of brain’s computational principles offers direct inspirations for new learning algorithms and architectures in SNNs ( Fang et al. 2021b ; Pan et al. 2025 ) . On the other hand, deep learning techniques have made it possible to obtain large-scale SNNs with high performance ( Wu et al. 2018 ; Fang et al. 2021a ; Bu et al. 2022 ) . Some of these well-trained models, in turn, can serve as in silico platforms for investigating multi-scale cortical computations that are difficult to access through wet-lab experiments ( Bellec et al. 2018 ) . Despite the promising synergy, realizing the potential of SNNs as an ideal platform for exploring both machine and biological intelligence is hindered by a trade-off between computational performance and biological plausibility. Many SNN learning algorithms achieve high performance by adopting backpropagation-based techniques, treating spiking neurons as recurrent units unrolled over time ( Neftci et al. 2019 ; Fang et al. 2021a ) . While this strategy yields models whose performance are comparable to their ANN counterparts, it reduces SNNs to mere deep learning artifacts, sacrificing basic biological properties. As a result, these models often ignore fundamental principles in neuroscience like the excitation-inhibition (E-I) dynamics, which are crucial in gain control ( Goldwyn et al. 2018 ; Del Rosario et al. 2025 ) , neural oscillation ( Buzsáki & Draguhn 2004 ) , selective attention ( Zhang et al. 2014 ) , and so on. Consequently, deriving meaningful insights for neuroscience from these biologically unfaithful models becomes a challenge. However, approaches that prioritize biological realism also face the challenges of unstable training. Biologically plausible learning rules such as spike-timing-dependent plasticity (STDP) ( Gerstner et al. 1996 ; Bi & Poo 1998 ) , often struggle with the instability during training and thus can only be applied to shallow SNNs ( Habenschuss et al. 2012 ; Beyeler et al. 2013 ) . In the field of deep learning, this training instability issue is partly overcome by explicit normalization schemes, most notably batch normalization (BN) ( Ioffe & Szegedy 2015 ) . However, while BN is a powerful tool, it depends on global statistics across mini-batches and has no known biological analogue, making it implausible for brain-inspired models. This widens the gap between high performance and biological plausibility, highlighting the need for a computationally effective and biologically grounded alternative. Here, we address the challenge of training deep biologically plausible SNNs by introducing lateral inhibition. We propose a normalization-free learning framework for deep SNNs, based on an excitatory-inhibitory (E-I) circuit, as shown in Figure 1 . Our framework presents a viable, brain-inspired alternative to standard normalization schemes, bridging the gap between high-performance deep learning and biologically plausible neural computation. The main contributions of our work can be summarized as follows: 1. We propose incorporating E-I circuits, composed of distinct excitatory and inhibitory neuron populations, into deep SNNs to enable normalization-free training. 2. To enable the training of this biologically-constrained SNN, we introduce a dynamic initialization scheme to ensure effective learning from the very beginning of training. 3. We construct a framework for stable training by integrating our initialization scheme with adaptive stabilization of divisive inhibition and a straight-through estimator (STE) method. These techniques have proven to be essential for stable learning of deep SNNs with E-I circuits. 4. We empirically validate our framework across various deep architectures. Experiments demonstrate that our biologically constrained learning framework can achieve competitive performance, indicating the viability of our brain-inspired learning algorithm. Figure 1: The proposed feedforward E-I circuit (left), compared with BN-equipped architecture (right). Neurons in layer l-1 and l+1 are not shown. In the E-I circuit, excitatory and inhibitory neurons are explicitly separated. In contrast, the BN architecture applies normalization operations between layers without distinguishing neuron types. 2 Related Work 2.1 Normalization methods in spiking neural networks Recent methods for training deep SNNs can be broadly categorized into two approaches, ANN-to-SNN conversion (ANN2SNN) ( Rueckauer et al. 2017 ; Sengupta et al. 2019 ; Han et al. 2020 ; Han & Roy 2020 ; Ding et al. 2021 ; Stöckl & Maass 2021 ; Bu et al. 2022 ) and direct end-to-end training through backpropagation ( Lee et al. 2016 ; Neftci et al. 2019 ; Li et al. 2021 ; Fang et al. 2021a ; Fang et al. 2021b ; Guo et al. 2022b ; Xiao et al. 2022 ; Zhu et al. 2024 ) . Normalization is critical in both ANN2SNN and training from scratch. Many ANN2SNN methods merge parameters of normalization layers in ANNs into synaptic weights of spiking neurons ( Sengupta et al. 2019 ; Han et al. 2020 ; Bu et al. 2022 ) . In contrast, training SNNs from scratch usually directly adopt normalization schemes developed for ANNs, especially BN ( Ioffe & Szegedy 2015 ) . There are also BN-derived normalization schemes designed for SNNs, like BNTT ( Kim & Panda 2021 ) , tdBN ( Zheng et al. 2021 ) , and TEBN ( Duan et al. 2022 ) . However, these strategies still inherit

Read the complete text