EV-LayerSegNet: Self-supervised Motion Segmentation using Event Cameras
Youssef Farah, Federico Paredes-Vallés, Guido C.H.E. De Croon, Muhammad Ahmed Humais, Hussain Sajwani, Yahya Zweiri · 2025
Read EV-LayerSegNet: Self-supervised Motion Segmentation using Event Cameras by Youssef Farah, Federico Paredes-Vallés, Guido C.H.E. De Croon, Muhammad Ahmed Humais, Hussain Sajwani, Yahya Zweiri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06596.txt.
Introduction
Youssef Farah and co-authors teach a camera system to split motion in a 2025 paper. Event cameras report brightness changes pixel by pixel with extreme timing precision, which suits motion tasks. Labels for training such systems are costly and rare, so the team built a self-teaching network instead.
Their network learns flow and motion masks separately, in a layered view of the scene. It then uses both to sharpen the raw event stream, and the sharpening quality becomes its own training signal. Trained on simulated data with simple motion, it reached solid overlap and detection scores. The authors present the method as a way to gain motion awareness without hand-labeled data.
Catalog record
| CREATOR | Youssef Farah, Federico Paredes-Vallés, Guido C.H.E. De Croon, Muhammad Ahmed Humais, Hussain Sajwani, Yahya Zweiri |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06596
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EV-LayerSegNet: Self-supervised Motion Segmentation using Event Cameras Youssef Farah, Federico Paredes-Vallés, Guido C.H.E. De Croon, Muhammad Ahmed Humais, Hussain Sajwani, Yahya Zweiri Abstract Event cameras are novel bio-inspired sensors that capture motion dynamics with much higher temporal resolution than traditional cameras, since pixels react asynchronously to brightness changes. They are therefore better suited for tasks involving motion such as motion segmentation. However, training event-based networks still represents a difficult challenge, as obtaining ground truth is very expensive, error-prone and limited in frequency. In this article, we introduce EV-LayerSegNet, a self-supervised CNN for event-based motion segmentation. Inspired by a layered representation of the scene dynamics, we show that it is possible to learn affine optical flow and segmentation masks separately, and use them to deblur the input events. The deblurring quality is then measured and used as self-supervised learning loss. We train and test the network on a simulated dataset with only affine motion, achieving IoU and detection rate up to 71% and 87% respectively 1 1 1 Project page: fulmen67.github.io/EV-LayerSegNet . 1 Introduction Event cameras, such as the Dynamic Vision Sensor (DVS), are novel bio-inspired sensors that perceive motion by detecting brightness changes instead of capturing intensity changes in images within a time interval Gallego et al. 2022 . In other words, if at time t a brightness change larger than the threshold C is detected by a pixel with x and y coordinates, an event is generated by recording the brightness increase or decrease, the pixel location and the time the event was generated. In contrast, traditional cameras record color intensities in all pixels at a specified frame rate. This fundamental shift in the acquisition of visual information brings several advantages. High temporal resolution and low latency (in the order of \mu s ) make event cameras well suited for recording very fast moving scenes without suffering from motion blur as frame-based cameras. The High Dynamic Range (HDR) allows the cameras to capture details in very dark or very bright scenes, making them appropriate to be used in light challenging environments such as the underwater ocean, night driving or fireworks display Zhang et al. 2025 . The combination of these advantages unlocks the possibility of performing tasks inaccessible until now for frame-based cameras such as low-latency optical flow estimation, high-speed control and tracking, and Simultaneous Localization and Mapping (SLAM) Gallego et al. 2022 . Nevertheless, a new generation of event-based vision algorithms needs to be developed to enable the event cameras to express their full potential, as the data structure of events is not compatible with the conventional algorithms based on images. Furthermore, an additional challenge is posed by the lack of ground truth from real-word datasets, at microsecond resolution and with HDR. In image-based datasets (COCO Lin et al. 2014 , BD100K Yu et al. 2018 ), the ground truth is often obtained by humans manually annotating each frame. This process is already expensive and laborious for image-based datasets, and becomes unfeasible for event data, which are sparse and have low latency. To avoid manual annotation, additional sensors and cameras were used to generate the ground truth in traditional event-based datasets such as DSEC Gehrig et al. 2021 and MVSEC Zhu et al. 2018a . Yet, sensors and cameras are limited by their natural Field Of View (FOV), spatial and temporal resolution. Therefore, it is of paramount importance to develop algorithms that are able to perform their tasks by relying on the nature of events, without the use of ground truth that are expensive to collect and that carry error measurements from additional sensors. In the context of motion estimation and its use in other tasks such as surveillance, tracking or obstacle avoidance, segmenting the scene into independently moving objects is fundamental and often referred as motion segmentation Stoffregen et al. 2019 . Even though great progress has been made using frame-based cameras, the latter are not ideally suited for tasks involving motion as they suffer from motion blur and keep track of all pixel values even when no motion occurs Stoffregen et al. 2019 . Under constant illumination, event cameras are well suited for this task, since events are sampled at exactly the same rate of the scene dynamics and the information acquired is only related to the motion of the camera and the objects in the scene. Event-based motion segmentation can be broadly categorized in model-based methods Stoffregen et al. 2019 ; Parameshwara et al. 2021b ; Zhou et al. 2021 and learning-based methods Mitrokhin et al. 2020 ; Sanket et al. 2020 ; Parameshwara et al. 2021a . Model-based methods exploit the nature of event data to cluster events generated by Independently Moving Objects (IMO) in an iterative fashion, thus they are unsupervised and do not use ground truth. Yet, the iterative approach requires long computational time and limits their real-word applicability. On the other hand, learning-based methods apply deep learning techniques while also relying on the event characteristics. In some instances, they perform motion segmentation in combination with other tasks such as tracking and optical flow, but they all require ground truth or pretrained weights during training. To the best of our knowledge, there is no existence of a learning-based method that is self-supervised, thus learning directly from the input events instead of relying on ground truth or pretrained weights from supervised methods. Looking outside of the event domain, significant progress has been made in self-supervised image segmentation. These methods segment the scene into the dominant moving object as foreground and the rest as background, and learning is driven by image or optic flow reconstruction. In most instances, they frequently make use of layered representation Wang and Adelson 1994 . The pixels are separated into layers based on motion similarity, and they are moved according to the associated motion and synthesized together to reconstruct the next image in the sequence. Recently, Shrestha et al . Shrestha et al. 2021 integrated this process into a end-to-end differentiable CNN pipeline which segments the dominant moving object in a video using two consecutive frames. Inspired by their work, we propose a self-supervised convolutional neural network for motion segmentation using events. Specifically, we transfer the CNN segmentation approach from Shrestha et al. 2021 to the event domain, and combine it with the encoder-decoder structure from Zhu et al. 2019 . We show that, under the assumption of affine motion and constant brightness, it is possible to segment independently moving objects using contrast maximization loss. We identify our main contributions as follows: 1. We propose a novel self-supervised network for learning event-based motion segmentation by introducing a novel optical flow module that enables self-supervised learning of affine optical flow and a segmentation module that learns separately the masks corresponding to the independently moving objects. 2. We contribute a new event-based dataset of several simulated backgrounds and objects moving according to affine motion. 3. We demonstrate through experiments that our method shows superior performance in comparison with the state-of-the-art in unsupervised motion segmentation. 2 Related Work In computer vision, motion segmentation is defined as the task of retrieving the shape of moving objects Zhang et al. 2018 . Image difference such as in Zappella et al. 2011 attempts to detect