SpikeStereoNet: A Brain-Inspired Framework for Stereo Depth Estimation from Spike Streams

Zhuoheng Gao, Yihao Li, Jiyao Zhang, Rui Zhao, Tong Wu, Hao Tang, Zhaofei Yu, Hao Dong, Guozhang Chen, Tiejun Huang · 2025

Read SpikeStereoNet: A Brain-Inspired Framework for Stereo Depth Estimation from Spike Streams by Zhuoheng Gao, Yihao Li, Jiyao Zhang, Rui Zhao, Tong Wu, Hao Tang, Zhaofei Yu, Hao Dong, Guozhang Chen, Tiejun Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19487.txt.

Introduction

Depth from two eyes depends on matching points between views, a task that spike cameras could transform. Zhuoheng Gao and co-authors present SpikeStereoNet in this 2025 paper, the first system to read depth straight from raw spike streams. Spike cameras fire events at microsecond pace instead of taking frames, catching edges that normal cameras blur.

The network merges the two spike streams and refines depth through a recurrent spiking module, echoing brain circuits. The team built both a synthetic data set and a real stereo spike data set with dense depth labels to test it. The system beat frame-based rivals, above all on flat surfaces and harsh light. It also kept accuracy when training data was cut sharply.

Catalog record

CREATORZhuoheng Gao, Yihao Li, Jiyao Zhang, Rui Zhao, Tong Wu, Hao Tang, Zhaofei Yu, Hao Dong, Guozhang Chen, Tiejun Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19487

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SpikeStereoNet: A Brain-Inspired Framework for Stereo Depth Estimation from Spike Streams Zhuoheng Gao, Yihao Li, Jiyao Zhang, Rui Zhao, Tong Wu, Hao Tang, Zhaofei Yu, Hao Dong, Guozhang Chen, Tiejun Huang Abstract Conventional frame-based cameras often struggle with stereo depth estimation in rapidly changing scenes. In contrast, bio-inspired spike cameras emit asynchronous events at microsecond-level resolution, providing an alternative sensing modality. However, existing methods lack specialized stereo algorithms and benchmarks tailored to the spike data. To address this gap, we propose SpikeStereoNet, a brain-inspired framework and the first to estimate stereo depth directly from raw spike streams. The model fuses raw spike streams from two viewpoints and iteratively refines depth estimation through a recurrent spiking neural network (RSNN) update module. To benchmark our approach, we introduce a large-scale synthetic spike stream dataset and a real-world stereo spike dataset with dense depth annotations. SpikeStereoNet outperforms existing methods on both datasets by leveraging spike streams’ ability to capture subtle edges and intensity shifts in challenging regions such as textureless surfaces and extreme lighting conditions. Furthermore, our framework exhibits strong data efficiency, maintaining high accuracy even with substantially reduced training data. 1 Introduction Depth perception is fundamental for navigating and interacting with the 3D world, driving applications from robotic manipulation [ 1 , 2 , 3 , 4 ] to autonomous navigation [ 5 , 6 , 7 , 8 , 9 , 10 , 11 ] . Traditional stereo vision estimates depth from calibrated image pairs captured by frame-based cameras; however, it suffers from motion blur and latency in dynamic scenes. Biological vision systems efficiently process visual inputs using sparse, asynchronous spikes, achieving remarkable speed and energy efficiency [ 12 , 13 , 14 , 15 , 16 , 17 , 18 ] . Neuromorphic spike cameras [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ] implement this principle in hardware, delivering ultra-high temporal resolution (up to 40,000 Hz) and capturing rich luminance information through asynchronous binary spike streams. This makes them suitable for robust perception tasks, such as stereo depth estimation, particularly in highly dynamic scenarios where conventional methods are ineffective. Despite their potential, spike cameras present distinct challenges for stereo depth estimation. Their asynchronous, binary, high - throughput streams conflict with frame - based algorithms that expect synchronous, intensity - valued image pairs [ 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ] , and any conversions to frames introduce temporal quantization errors, motion blur, or significant computational overhead [ 23 , 25 , 43 ] , diminishing the sensor’s intrinsic advantages. Although event - based stereo methods have been developed for DVS cameras (e.g. [ 44 , 45 , 46 , 47 , 48 ] ), they rely on temporal contrast, whereas spike cameras emit integrated intensity streams that require tailored processing techniques. Critically, the field lacks specialized algorithms and benchmarks designed for stereo depth estimation directly from raw spike streams, hindering progress in this promising field. To bridge this gap, we introduce SpikeStereoNet (Fig. 1 ), an end-to-end, biologically inspired framework for generating stereo depth directly from raw spike streams. SpikeStereoNet integrates a recurrent spiking neural network (RSNN) into an iterative refinement loop [ 49 ] and models neuronal interactions [ 50 ] to capture the spatiotemporal dynamics inherent in spike data. The analysis of neuronal dynamics confirms the temporal stability, convergence, and expressive feature separation of the RSNN components. We introduce two novel benchmark datasets: a large-scale synthetic dataset with diverse scenes and ground-truth depth, and a real-world dataset containing synchronized stereo spike streams and corresponding depth sensor measurements. Experimental results demonstrate that SpikeStereoNet generalizes effectively, maintaining robust performance even when trained with substantially limited data. Our main contributions are as follows: • We present the first large-scale synthetic and real-world raw spike stream datasets for stereo depth estimation to offer an evaluation benchmark for this emerging field. • We propose a novel biologically inspired RSNN-based SpikeStereoNet architecture that refines asynchronous spike data through iterative updates. • We analyze the dynamics of RSNN iterations to demonstrate the stability and convergence properties of the model. • We demonstrate the data efficiency of the proposed framework, highlighting the strong generalization even with limited training samples. Figure 1: The overall pipeline is illustrated in the figure below. The upper path represents the training and evaluation process on synthetic dataset, while the lower path shows the transfer learning and testing procedure on real spike stream data by using the pre-trained model. 2 Related Work Spike Cameras and its Applications. Neuromorphic cameras, including event cameras [ 51 , 52 , 53 ] and spike cameras [ 54 , 55 ] , are inspired by the primate retina and operate asynchronously at the pixel level, providing ultra-high temporal resolution, wide dynamic range, low latency, and reduced energy consumption. Unlike event cameras, which follow a differential sampling model and capture only luminance changes in logarithmic space, spike cameras employ an integral sampling model, accumulating photons until a threshold is reached and then firing spikes. Although event cameras tend to generate sparser output, they might lose scene textures, especially in static regions due to their change-based design. In contrast, spike cameras can capture richer spatial details, making them well suited for tasks such as high-speed imaging [ 23 , 26 , 56 , 57 ] , 3D reconstruction [ 58 ] , motion deblurring [ 43 , 59 ] , optical flow [ 24 , 25 ] , object detection [ 60 ] , occlusion removal [ 61 ] and monocular depth estimation [ 21 ] . Frame-Based Stereo Depth Estimation. Deep learning has significantly advanced stereo matching, leading to various methods with enhanced accuracy and generalization. The early methods [ 62 , 63 , 64 , 65 ] constructed cost volumes for each disparity and applied 3D CNNs for refinement. In contrast, recurrent refinement-based models [ 31 , 32 , 33 , 34 , 36 ] inspired by RAFT-Stereo [ 29 , 49 ] iteratively update the disparity without building a full 4D volume. It restricts 2D flow to the 1D disparity dimension and uses multi-level convolutional Gated Recurrent Unit (ConvGRU) for broader receptive fields, demonstrating strong generalization. Successor models such as CREStereo [ 30 ] , IGEV-Stereo [ 33 ] and Selective-Stereo [ 36 ] improve in-domain performance, but RAFT-Stereo retains advantages in zero-shot settings [ 29 ] . Recent hybrid methods [ 66 ] combine cost filtering and iterative refinement for better trade-offs. Meanwhile, transformer-based solutions [ 37 ] utilize cross-attention or global self-attention. As deep stereo matching evolves, fully leveraging vision foundation models remains an open opportunity for real-world applications. Event-Based Stereo Depth Estimation. Event cameras capture pixel-level brightness changes asynchronously, offering significant advantages over conventional frame-based cameras. Recently, event-based stereo depth estimation has advanced rapidly, with notable methods exploiting camera velocity [ 45 ] or deriving depth without explicit event matching [ 67 ] . Deep learning approaches propose novel sequence embedding [ 68 ] and fuse frame and event data [ 69 , 70 ] to improve depth

Read the complete text