DelRec: learning delays in recurrent spiking neural networks

Alexandre Queant, Ulysse Rançon, Benoit R Cottereau, Timothée Masquelier · 2025

Read DelRec: learning delays in recurrent spiking neural networks by Alexandre Queant, Ulysse Rançon, Benoit R Cottereau, Timothée Masquelier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24852.txt.

Introduction

Alexandre Queant, Ulysse Rançon, Benoit R Cottereau, and Timothée Masquelier teach recurrent spiking networks to learn their own signal delays. Their 2025 paper works with networks of artificial neurons that fire brief spikes, prized for low energy use. In brains, both connection strength and transmission delay adapt, and theory says delays add expressive power.

Until now, delay learning mostly covered one-way networks. Their method tunes delays in recurrent loops together with the weights, using a smooth approximation that gives usable gradients for fractional delays. On a chaotic series prediction task it beat all baselines. On four other time-based data sets it stayed competitive. Studying the trained networks shows delay tuning builds depth-dependent timing structure, echoing how real circuits organize signals in time.

Catalog record

CREATORAlexandre Queant, Ulysse Rançon, Benoit R Cottereau, Timothée Masquelier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24852

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DelRec: learning delays in recurrent spiking neural networks Alexandre Queant, Ulysse Rançon, Benoit R Cottereau, Timothée Masquelier Abstract Spiking neural networks (SNNs) are a bio-inspired alternative to conventional real-valued deep learning models, with the potential for substantially higher energy efficiency. Interest in SNNs has recently exploded due to a major breakthrough: surrogate gradient learning (SGL), which allows training SNNs with backpropagation, strongly outperforming other approaches. In SNNs, each synapse is characterized not only by a weight but also by a transmission delay. While theoretical works have long suggested that trainable delays significantly enhance expressivity, practical methods for learning them have only recently emerged. Here, we introduce “DelRec”, the first SGL-based method to train axonal or synaptic delays in recurrent spiking layers, compatible with any spiking neuron model. DelRec leverages a differentiable interpolation technique to handle non-integer delays with well-defined gradients at training time. We show that trainable recurrent delays outperform feedforward ones, leading to new state-of-the-art (SOTA) on two challenging temporal datasets (Spiking Speech Command, an audio dataset, and Permuted Sequential MNIST, a vision one), and match the SOTA on the now saturated Spiking Heidelberg Digit dataset using only vanilla Leaky-Integrate-and-Fire neurons with stateless (instantaneous) synapses. Our results demonstrate that recurrent delays are critical for temporal processing in SNNs and can be effectively optimized with DelRec, paving the way for efficient deployment on neuromorphic hardware with programmable delays. Our code is available at https://github.com/alexmaxad/DelRec . 1 Introduction Inspired by the architecture and dynamics of biological neural networks, Recurrent Spiking Neural Networks (RSNNs) provide a compelling and energy-efficient framework for processing time-varying data. Their recurrent structure enables them to maintain an internal state, integrating information over extended periods, an essential capability for tasks involving temporal dependencies, such as speech recognition and time-series prediction ( Bellec et al. 2018 ) . However, despite their promise, RSNNs remain underutilized in machine learning due to significant training challenges, particularly the pervasive issues of vanishing and exploding gradients. Recent advancements have sought to address these limitations by enhancing the models of spiking neurons ( Yin et al. 2021 ; Bittar & Garner 2022 ; Baronig et al. 2025 ) . Innovations such as adaptive leaky integrate-and-fire (AdLIF) models and other sophisticated neuron dynamics have led to notable improvements, achieving state-of-the-art performance on widely used spiking neural network benchmarks. Yet, these neuron-centric approaches put their emphasis on modulating current inputs, rather than reactivating past signals, which limits their ability to model extended temporal dynamics. An alternative approach involves incorporating transmission delays in synaptic connections, a feature observed in biological systems. In the brain, these delays are modulated by myelin, an insulating sheet around axons that accelerates conduction speeds. Evidence further indicates that myelin levels are plastic ( Monje 2018 ) . Delays enhance network expressivity: neurons detect coincident spike time latencies, which, in the presence of heterogeneous delays, correspond to arbitrary spike onset latency sequences. While previous studies have successfully demonstrated the benefits of using delays in feedforward connections ( Shrestha & Orchard 2018 ; Sun et al. 2022 ; Sun et al. 2023 ; Timcheck et al. 2023 ; Hammouamri et al. 2024 ; Deckers et al. 2024 ; Göltz et al. 2025 ; Ghosh et al. 2025 ) , their potential in recurrent connections remains largely unexplored. Recurrent delays could offer even greater advantages, facilitating self-sustained activity (see Fig. 1 A), modeling long-term dependencies, and supporting complex patterns like oscillations and polychronization. Theoretical work by Izhikevich 2006 demonstrated that recurrent delays transform a neuron’s differential equation, expanding the range of possible solutions and enabling richer dynamics. Additionally, recurrent delays may mitigate gradient challenges by implementing temporal skip connections, improving gradient propagation during training (see Fig 1 B). Figure 1: A : The optimization of a single delay in a recurrent connection can transform two recurrently connected neurons in a pattern generator. Two different behaviors of two neurons with the same inputs. Each neuron is recurrently connected to itself and to the other neuron, with a weight equal to 1. The recurrent connections each have a delay, indicated by the circled number on the connection. The neurons spike if they receive inputs strictly superior to 1 spike. At time t-1 , the neurons do not receive any input. The blue neuron receives an input spike at times t and t+3 , while the pink neuron only receives two spikes at time t+1 . Top : The inputs trigger the firing of one spike per neuron, working as a coincidence detector for spikes reaching the two neurons in a short time interval. Bottom : When the delay of the pink neuron’s recurrent connection (blue arrow) is increased from 1 to 3 time steps, the same input triggers a regular and sustained firing pattern. B : Delays in recurrent connections reduce the risks of exploding or vanishing gradients by bridging distant time steps. Computational graphs of a vanilla RSNN with a intrinsic delay of 1 time step in all recurrent connections ( Top ), and of a RSNN with different and longer delays in the recurrent connections ( Bottom ). To date, however, only a handful of studies have explored the potential of delays in recurrent connections, and even fewer have focused on learning optimal delay configurations. Recent approaches have introduced algorithms to optimize these temporal parameters, demonstrating promising improvements in temporal tasks. For instance, Xu et al. 2025 achieved state-of-the-art results by learning a single recurrent delay parameter per layer using backpropagation. Their approach selects delays from a fixed set via a softmax function with a decreasing temperature, showcasing the potential of more flexible, parameterized methods. To the best of our knowledge, only Mészáros et al. 2025 have proposed an algorithm specifically designed to learn optimal delays in recurrent connections. Their method, tailored for the EventProp algorithm ( Wunderlich & Pehle 2021 ) , leverages exact gradient computation. However, it inherits common limitations of EventProp, including scalability challenges and suboptimal performance on real-world temporal benchmarks. Currently, all state-of-the-art spiking approaches on these benchmarks rely on surrogate gradient learning (SGL). In this paper, we introduce ”DelRec”, the first method to train axonal or synaptic delays in recurrent connections using surrogate gradient learning (SGL) and backpropagation. Our method operates in discrete time and eliminates the need to predefine a maximum delay range. During training, we relax the constraint of integer delays by employing a differentiable interpolation process, then round delays to the nearest integer for inference. Our approach is implemented using the Pytorch-based Spikingjelly library ( Fang et al. 2023 ) , and is compatible with any spiking neuron model. We achieved new state-of-the-art results on two challenging temporal datasets (Spiking Speech Command, an audio dataset, and Permuted Sequential MNIST, a vision one), even when using simple Leaky-Integrate-and-Fire neurons. Additionally, our study is the first to combine the optimization of feedforward delays using DCLS ( Hammouamri et al. 2024 ) and delays in recurrent connections. We believe this work establishes a foundation for leveraging delays in recurrent networks and provides an accessible tool to explore the potential improvements offered by such methods. Our results highlight the essential role of recurrent delays in temporal processing for spiking neural networks (SNNs). Using DelRec, these delays can be optimized effectively, opening

Read the complete text