Perturbed State Space Feature Encoders for Optical Flow with Event Cameras

Gokul Raju Govinda Raju *, Nikola Zubić *, Marco Cannici, Davide Scaramuzza · 2025

Read Perturbed State Space Feature Encoders for Optical Flow with Event Cameras by Gokul Raju Govinda Raju *, Nikola Zubić *, Marco Cannici, Davide Scaramuzza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10669.txt.

Introduction

Gokul Raju Govinda Raju, Nikola Zubić, Marco Cannici, and Davide Scaramuzza published this paper in 2025. It improves optical flow estimation, the task of tracking motion between frames, for event cameras. These sensors fire only when brightness changes, which suits fast motion but strains standard deep learning models.

Their encoder processes features with a state space model, a design that sees far across space and time at modest cost. The key trick is a small deliberate wobble applied to the model's core dynamics, which steadies training and sharpens results. Folded into a system with two-way flows and memory across frames, the encoder cut error by about eight and twelve percent on two public test sets. The authors present it as the current best approach for event-based flow.

Catalog record

CREATORGokul Raju Govinda Raju *, Nikola Zubić *, Marco Cannici, Davide Scaramuzza
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-10669

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Perturbed State Space Feature Encoders for Optical Flow with Event Cameras Gokul Raju Govinda Raju *, Nikola Zubić *, Marco Cannici, Davide Scaramuzza Abstract With their motion-responsive nature, event-based cameras offer significant advantages over traditional cameras for optical flow estimation. While deep learning has improved upon traditional methods, current neural networks adopted for event-based optical flow still face temporal and spatial reasoning limitations. We propose Perturbed State Space Feature Encoders (P-SSE) for multi-frame optical flow with event cameras to address these challenges. P-SSE adaptively processes spatiotemporal features with a large receptive field akin to Transformer-based methods, while maintaining the linear computational complexity characteristic of SSMs. However, the key innovation that enables the state-of-the-art performance of our model lies in our perturbation technique applied to the state dynamics matrix governing the SSM system. This approach significantly improves the stability and performance of our model. We integrate P-SSE into a framework that leverages bi-directional flows and recurrent connections, expanding the temporal context of flow prediction. Evaluations on DSEC-Flow and MVSEC datasets showcase P-SSE’s superiority, with 8.48% and 11.86% improvements in EPE performance, respectively. 1 Introduction Optical flow estimation, a cornerstone of computer vision, plays a crucial role in diverse applications such as object tracking [ 55 , 26 ] , robotic navigation [ 41 , 35 ] , video interpolation [ 52 , 53 , 27 ] , and action recognition [ 40 , 48 ] . Recent deep learning approaches [ 50 , 25 , 43 ] have significantly outperformed traditional energy minimization techniques [ 24 ] , advancing the field substantially. However, optical flow estimation with traditional frame-based cameras faces challenges because images are captured at fixed time intervals, leading to temporal gaps, which could affect motion information. To overcome these limitations, researchers have turned to alternative imaging devices like event-based cameras [ 10 ] , which provide a continuous, asynchronous stream of data describing changes in relative pixel brightness with high temporal precision. Despite having access to rich and continuous motion information, existing learning-based optical flow estimation methods that leverage event-based data, [ 14 , 15 , 62 , 34 ] are still limited to pair-wise network designs, where only the local neighborhood of events surrounding two consecutive time instants is used to predict the corresponding optical flow. As a result, valuable temporal cues from preceding and successive instants are ignored. On the other hand, recent state-of-the-art methods in frame-based optical flow, such as VideoFlow [ 43 ] , have demonstrated that a multi-frame approach that optimizes bidirectional flows across multiple frames leads to superior performance, exploiting richer latent information within extended frame sequences. While VideoFlow [ 43 ] demonstrates improved temporal reasoning, it faces two significant limitations. First, the inherent temporal resolution constraints of traditional cameras hinder their ability to capture fine-grained motion details. Second, its reliance on the Twins Transformer [ 4 ] architecture for encoding introduces spatial limitations. The lack of a global effective receptive field in this encoder restricts the network’s capacity to capture extensive spatial dependencies. Consequently, these constraints potentially impair VideoFlow’s ability to effectively handle scenarios involving occluded objects or large-scale motions, highlighting areas where further advancements could yield substantial improvements in optical flow estimation. Drawing inspiration from recent advancements in frame-based optical flow techniques, we present a novel multi-event optical flow estimation approach. Our model departs significantly from conventional event-based methods through two key innovations. First, we implement bi-directional optical flow prediction and optimization across up to five consecutive time instants. This allows us to recursively propagate motion information between adjacent frames, substantially expanding the network’s temporal receptive field. To address the spatial limitations of existing methods, we introduce the Perturbed State Space Feature Encoders (P-SSE) to optical flow estimation—a first in this domain. This novel encoder is designed to capture more comprehensive spatial dependencies, improving the model’s ability to handle complex scenarios. These innovations combine to leverage extended temporal context and enriched spatial information, significantly improving optical flow estimation accuracy and robustness. The key innovation of P-SSE lies in the novel perturbation technique applied to the state dynamics matrix. While P-SSE builds upon the strengths of Transformers and Convolutional Neural Networks (CNNs), incorporating dynamic weights and global receptive fields characteristic of models like DeiT [ 51 ] , its distinguishing feature is the perturbation method that crucially improves its performance and stability. The perturbation technique is the key idea of P-SSE’s capabilities, making the state dynamics matrix of the State Space Model (SSM) more robust to noise and significantly stabilizing the entire system. By applying a carefully designed perturbation followed by diagonalization, we address the inherent instability issues often associated with SSMs, particularly when processing complex, asynchronous event data. P-SSE utilizes the scanning procedure of the VMamba model [ 33 ] . Still, our perturbation approach regularizes the state matrix and improves stability, allowing the model to capture the spatial context in event data more effectively. This improved spatial reasoning, combined with temporal insights from our VideoFlow-inspired [ 43 ] multi-event strategy, results in a comprehensive spatiotemporal approach to optical flow estimation. This innovative combination of perturbed spatial processing and multi-event temporal processing allows our model to capture complex motion patterns with strong accuracy and robustness, setting new standards in event-based optical flow estimation. We conducted a comprehensive evaluation of our model on the DSEC-Flow [ 13 ] and MVSEC [ 63 ] datasets, revealing several contributions: • Our experiments demonstrate the successful adaptation of multi-frame optical flow strategies to event-based vision, resulting in substantial performance improvements. • The global receptive field and perturbation technique provided by our proposed P-SSE, combined with the model’s capacity to leverage rich temporal motion information, significantly improves performance in challenging scenarios characterized by motion blur and occlusions. • On the DSEC-Flow benchmark [ 13 ] , our method achieves a top EPE score of 0.680, representing an 8.48% improvement over the previous best TMA model [ 32 ] , and a 9.33% improvement compared to the BFlow [ 15 ] model. On the MVSEC dataset [ 63 ] , we observe an 11.86% increase in EPE performance. 2 Related Work Event-based optical flow Event-based optical flow estimation has evolved through several distinct phases, each adapting to the unique characteristics of event data. Initial approaches focused on modifying classical computer vision techniques, including adaptations of the Lucas-Kanade algorithm [ 1 , 12 , 38 ] , local plane fitting strategies [ 2 , 36 ] , and methods recovering pixel trajectories via contrast maximization on the image plane [ 9 ] . The success of learning-based approaches in frame-based optical flow catalyzed a shift towards adapting these architectures for event processing. This transition necessitated the development of structured representations of raw event data [ 14 , 32 , 67 ] , capable of encapsulating rich motion details. Concurrently, unsupervised methods emerged, estimating sparse optical flow by minimizing photometric loss over warped images [ 62 ] or event-frames [ 59 ] , often incorporating additional smoothness constraints [ 61 ] or adversarial losses [ 28 ] . Recent advancements have prioritized dense predictions, aiming to provide reliable estimates even in event-free regions. E-RAFT [ 14 ] pioneered this approach by adapting the RAFT architecture [ 50 ]

Read the complete text