A scalable event-driven spatiotemporal feature extraction circuit Thanks: We thank Ton Juny Pina and Philipp Klein for their help with PCB soldering, and Thorben Schoepe for the chip photography. The work was funded by grants: EU H2020: NeuTouch (813713), BeFerroSynaptic (871737) and MANIC (861153); DFG: memTDE (441959088 - SPP 2262 MemrisTec 422738993) and NMVAC (432009531). We acknowledge the financial support of the CogniGron research center and the Ubbo Emmius Funds (University of Groningen). CRedit: Conceptualisation: H. G., M. M.; Methodology: E. C., H. G., M. M.; Software/Hardware: M. C., H. G., E. J., M. M., O. R., W. S. G.; Investigation: H. G., M. M.; Writing: E.C., M. C., H. G., M. M., O. R.; Visualisation: M. C., H. G., M. M., O. R.; Supervision: E. C. Affiliations: 1 Bio-Inspired Circuits and Systems (BICS) Lab, Zernike Institute for Advanced Materials, University of Groningen, Netherlands 2 Groningen Cognitive Systems and Materials Center (CogniGron), University of Groningen, Netherlands 3 Micro- and Nanoelectronic Systems (MNES), Technische Universität Ilmenau, Germany 4 Istituto Italiano di Tecnologia, Genova, Italy * Equal contribution Corresponding authors: {h.r.greatorex, m.mastella}@rug.nl Abstract Event-driven sensors, which produce data only when there is a change in the input signal, are increasingly used in applications that require low-latency and low-power real-time sensing, such as robotics and edge devices. To fully achieve the latency and power advantages on offer however, similarly event-driven data processing methods are required. A promising solution is the Time Difference Encoder (TDE) : an event-based processing element which encodes the time difference between events on different channels into an output event stream. In this work we introduce a novel TDE implementation on CMOS. The circuit is robust to device mismatch and allows the linear integration of input events. This is crucial for enabling a high-density implementation of many TDEs on the same die, and for realising real-time parallel processing of the high-event-rate data produced by event-driven sensors. I Introduction The emergence of event-based sensors has opened up new possibilities for applications that demand low-latency and energy-efficient data processing. Unlike traditional frame-based sensors that capture information at fixed intervals, event-based sensors transmit data only when a change is detected. This makes them highly suitable for real-time decision-making tasks across various sensory modalities, such as vision, audio, and tactile sensing [ 1 , 2 , 3 ] . These sensors not only offer low latency but also enable high temporal resolution and low power consumption, provided the event streams remain sparse. As a result, there has been growing interest in developing event-based processing units and circuits that can handle these streams while retaining the inherent benefits of the event-driven paradigm [ 4 , 5 , 6 ] . In this paper, we present a novel circuit that implements an improved version of the event-based processing element known in literature as the TDE [ 7 , 8 , 9 , 10 , 11 , 12 ] . While there has been a growing propensity to design event-based circuits that operate in the digital domain [ 13 , 14 ] , the original breakthroughs in this area were achieved through analog implementations that leverage the underlying physics of the devices [ 15 , 16 ] . Our design stays true to this analog approach, using transistors operating in subthreshold with currents in the range of \text{\,}\mathrm{pA} . The improved circuit in this work is designed to overcome issues related to mismatch, making it more scalable for large arrays. We demonstrate the circuit’s application to an event-vision task, highlighting its ability to extract spatiotemporal patterns from event-based data [ 17 ] . This makes it a strong candidate for edge applications, where low energy consumption and minimal latency are crucial for real-time sensory data processing. Figure 1 : Photograph of the realised “cognigr1” ASIC , fabricated in the XFAB 180\text{\,}\mathrm{nm} technology. The boxes highlight the location of the structures on the die. The total size of the TDE circuit is 19\text{\,}\mathrm{\SIUnitSymbolMicro m} \times 56\text{\,}\mathrm{\SIUnitSymbolMicro m} including guard rings. II Methods Figure 2 : The circuit schematics of the TDE synapse. The circuit initially proposed in [ 18 ] consists of a single discharge branch in the facilitatory block and a DPI [ 19 ] in the trigger block. The improved circuit presented in this work makes use of a DPI circuit in both blocks to ensure linear integration of events. Modifications with respect to the old circuit are highlighted. II-A Improved TDE Circuit The circuit proposed in this work (Fig. 2 ) is compared to the TDE circuit originally proposed by [ 18 ] , following previous implementations of CMOS motion detectors [ 20 , 21 ] . The TDE circuit generates an exponentially decaying current with an initial magnitude proportional to the negative exponential of the time difference between two spikes [ 22 ] . This current is fed into a Leaky Integrate and Fire (LIF) neuron [ 23 ] to produce output spikes with an instantaneous frequency with the same proportionality to the input time difference. This is achieved by the two distinct “facilitatory” and “trigger” integrator blocks in the TDE circuit, which we refer to collectively as the TDE synapse. The inputs to each block are the two input channels of the TDE . Figure 3 : Silicon measurements of the TDE circuit. The TDE received two sequential FAC and TRG input events with a time difference of 12\text{\,}\mathrm{m}\mathrm{s} . The response of the trigger block, V_{\text{TRG}} , generates an EPSC through transistor \text{M}_{10} which is integrated by the neuron, eliciting a spiking response observed through the membrane potential V_{\text{mem}} . Both blocks include a time-decaying voltage trace initiated by a digital pulse (or event). When an event arrives at the facilitatory input (FAC), a decaying voltage trace is initiated. When an event arrives at the trigger input (TRG), a synaptic current with amplitude proportional to the immediate value of the trigger trace (relative to the power supply) is generated. This current is integrated by a neuron circuit. The closer the facilitatory and trigger spikes are in time, the higher the amplitude of the facilitatory trace at the arrival of the TRG spike and of the consequent EPSC . Therefore, the resulting output spikes encode the input time difference both in the number of output spikes and their Interspike Interval (ISI) , or equivalently their instantaneous firing rate. In this way the TDE circuit acts as an asymmetric correlation detector on the two inputs and encodes the time difference between input events in an analog fashion. II-B Testing setup The cognigr1 Integrated Circuit (IC) [ 24 , 25 , 26 ] was fabricated in CMOS (Fig. 1 ) using a 180\text{\,}\mathrm{nm} technology node. The biases of the circuit are provided using a subthreshold Digital to Analog Converter (DAC) present on the die. All tests were carried out using a custom setup with oscilloscopes and a microcontroller with a Python interface supplying configuration information and events. II-C Optical flow task Figure 4 : Left: Simulated event-based camera data of a textured surface moving in a vertical direction. Center: The x-y space was randomly and sparsely sampled by 100 TDEs with an equal ratio of cardinal orientations. Right: Arrows represent the connectivity of FAC and TRG circuit inputs to neighbouring event-based camera pixels. The TDE circuit was applied to an event-based vision task. We simulated event-based camera [ 1 ] data (Fig. 4 ) to evaluate the circuit’s capability to detect optical flow. A synthetic textured surface consisting of multiple irregularities and patterns moving upwards was used to generate a stream of x-y address events. Each TDE in the network was configured to receive the events from two particular pixels on its FAC and TRG inputs. In this way, each TDE was configured to have a certain direction sensitivity, determined by its 2-pixel receptive field. One hundred TDEs were randomly distributed across the visual field, with equal proportions oriented in the four cardinal directions. Since the cognigr1 IC hosts only a single TDE circuit, the event-based camera data was sequentially applied to its input. If multiple TDE circuits are present in an on-chip array, further analysis would be needed to examine the impact of mismatch on task performance. III Results and Discussion Figure 5 : Monte Carlo simulation results comparing our circuit with [ 18 ] . For a given time difference, \Delta t , the charge transmitted by the TDE synapse was measured for 2000 simulated instances of the circuit. This charge was then normalized relative to the average charge for a given \Delta t . III-A Circuit measurements Figure 3 shows measurements of input and output signals of the TDE circuit. Digital pulses, representing events or spikes, were applied to the FAC and TRG transistors of the circuit ( \text{M}_{1} and \text{M}_{6} respectively). The time difference between the two pulses is encoded by V_{\text{TRG}} which generates an EPSC with an amplitude that encodes the time between input events. This current is then fed into the neuron circuit eliciting a spiking response that can be observed through the membrane potential V_{\text{mem}} . III-B Monte Carlo analysis To validate the improvement of our circuit with respect to the circuit proposed in [ 18 ] , Monte Carlo simulations were performed and analysed using the Cadence Spectre tool. We simulated 2000 different instances of both circuits over a range of \Delta t values, sampling from a realistic distribution of physical transistor variations. Figure 5 shows the distribution of charge sourced by both circuits, calculated by integrating the current I^{\text{TDE}} (see Fig. 2 ). Figure 6 shows the mean and standard deviation of each Monte Carlo simulation, demonstrating the improved robustness of the newly proposed circuit to device mismatch. We found an average reduction in the coefficient of variation of 61% averaged across \Delta t . III-C Optical flow task Figure 7 shows the spiking response of the 100 TDE units in the network run on the cognigr1 IC . The response of each orientation as a percentage of the total network activity shows a significant preference towards the direction of optical flow. The TDEs oriented perpendicular to the direction of optical flow (left and right) exhibit a greater response in comparison to those orientanted in the null direction (down). This can be explained by considering that an ideal sensor would produce exactly coincident spikes from pixels aligned along a moving edge. Figure 6 : The average charge transmitted plotted against \Delta t , error bars show the standard deviation across Monte Carlo simulations. Figure 7 : Left: A raster plot of the measured spiking responses of the TDEs in the network. Right: The measured number of spikes for the four TDE orientations is expressed as a fraction of the total network activity during the event data duration. Instead, the inherent asynchrony of a physical Address Event Representation (AER) [ 27 ] sensor produces some jitter which was also introduced into the synthetic data. This results in very small time differences at the input of a TDE with a receptive field aligned to a moving edge, leading to high activation of TDEs oriented perpendicular to the direction of motion. However, when sampled across the entire visual field, this effect averages out, resulting in roughly equal responses in the orthogonal directions. IV Conclusion In this work, we designed and silicon-verified an enhanced version of the TDE event-based computational primitive. The modifications we made to the circuit significantly improved its robustness to device mismatch, as confirmed by comprehensive Monte Carlo simulations. These simulations demonstrated the circuit’s suitability for integration into high-density array structures on CMOS technology, thereby paving the way for efficient scaling in practical applications. To explore the circuit’s potential for highly parallel real-time processing, we tested it on an optical flow detection task utilizing event-based synthetic data derived from a moving texture. This application highlighted the circuit’s capabilities not only in handling real-time data but also in its efficiency in processing the information generated by event-driven sensors.