Event-guided Low-light Video Semantic Segmentation

Zhen Yao, Mooi Choo Chuah · 2024

Read Event-guided Low-light Video Semantic Segmentation by Zhen Yao, Mooi Choo Chuah (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00639.txt.

Description

Abstract Recent video semantic segmentation (VSS) methods have demonstrated promising results in well-lit environments. However, their performance significantly drops in low-light scenarios due to limited visibility and reduced contextual details. In addition, unfavorable low-light conditions make it harder to incorporate temporal consistency across video frames and thus, lead to video flickering effects.

Catalog record

CREATORZhen Yao, Mooi Choo Chuah
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00639

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Event-guided Low-light Video Semantic Segmentation Zhen Yao, Mooi Choo Chuah Abstract Recent video semantic segmentation (VSS) methods have demonstrated promising results in well-lit environments. However, their performance significantly drops in low-light scenarios due to limited visibility and reduced contextual details. In addition, unfavorable low-light conditions make it harder to incorporate temporal consistency across video frames and thus, lead to video flickering effects. Compared with conventional cameras, event cameras can capture motion dynamics, filter out temporal-redundant information, and are robust to lighting conditions. To this end, we propose EVSNet, a lightweight framework that leverages event modality to guide the learning of a unified illumination-invariant representation. Specifically, we leverage a Motion Extraction Module to extract short-term and long-term temporal motions from event modality and a Motion Fusion Module to integrate image features and motion features adaptively. Furthermore, we use a Temporal Decoder to exploit video contexts and generate segmentation predictions. Such designs in EVSNet result in a lightweight architecture while achieving SOTA performance. Experimental results on 3 large-scale datasets demonstrate our proposed EVSNet outperforms SOTA methods with up to 11 × 11\times higher parameter efficiency. 1 Introduction Video semantic segmentation, a problem of assigning a category label to each pixel in the video frames, has become a hot research topic in recent years. It plays a fundamental role in a wide range of multimedia and computer vision applications including video parsing [ 29 , 35 ] , video processing [ 3 , 70 , 33 ] , and autonomous driving [ 68 , 75 , 74 , 23 ] . Figure 1 : Comparison among (a) normal light frame, (b) low-light frame, (c) events generated from normal light frames, and (d) events generated from low-light frames. Events demonstrate robustness against lighting changes and effectively capture temporal motion features in low-light environments. While video semantic segmentation of normal light scenes has made tremendous achievements [ 48 , 33 , 40 , 53 , 73 ] , low-light scenarios are still challenging due to limited visibility and degraded image quality. In low-light conditions, conventional frame-based cameras have difficulty capturing a wide range of brightness levels, resulting in low contrast and loss of textures. The reduced contrast hinders accurate discrimination of object boundaries, ultimately diminishing clarity and fidelity of the captured images. In addition to limited visibility, low-light semantic segmentation is also challenging because of luminance noise. Severe noises caused by constrained photon counts and imperfections in photodetectors manifest as random bright or dark pixels scattered throughout the whole image. Such noises often lead to inaccurate segmentation predictions. To resolve the above issues, researchers have explored event cameras as an alternate sensing modality. Event sensors asynchronously measure sparse data streams at high temporal resolution (10µs vs 3ms), higher dynamic range (140dB vs 60dB), and significantly lower energy (10mW vs 3W) compared to conventional cameras. In recent years, it has been increasingly utilized in the computer vision [ 16 , 34 , 5 , 22 , 47 , 25 ] and robotics [ 57 , 52 ] community. Researchers have explored event modality in many tasks such as 3D reconstruction [ 76 ] , pose estimation [ 5 ] , and simultaneous localization and mapping (SLAM) [ 17 ] . Figure 2 : Different from existing approach which fuse event features in the encoder, our model utilizes event features to learn short-term and long-term motions and leverage these motion features to gain video temporal contexts. Instead of capturing an image at a fixed interval, the event cameras, such as the Dynamic Vision Sensor (DVS), only record a single event based on the brightness changes at each pixel. This makes it suitable for edge-case scenarios and video-related tasks. For low-light video semantic segmentation task, event modality offers two advantages: (1) it facilitates the learning of temporal consistency; (2) it provides an alternative perspective through intensity changes. Figure. 1 explains the potential of event modality in low-light conditions. Although bringing a new paradigm shift, event modality presents two challenges: firstly, it only captures pixels in motion, leading to sparse information; secondly, it demonstrates distinct attributes compared with visual image modality, highlighting the significance of effectively modeling event features. To address the above challenges, we propose a lightweight framework, namely EVSNet, that exploits both image and event modalities for low-light video semantic segmentation. Figure. 2 illustrates the difference between existing works and our approach. The architecture of EVSNet is shown in Figure. 3 . It consists of three parts: an Image Encoder, a Motion Extraction Module, a Motion Fusion Module, and a Temporal Decoder. Specifically, we first adopt a lightweight backbone as the Image Encoder to extract image features. Inspired by Atkinson-Shiffrin memory model [ 2 ] which hypothesizes that the human memory consists of short memory and long memory, we then propose a Motion Extraction Module to extract short-term and long-term motion features and acquire the video contexts to guide semantic understanding through such motions. We apply Event Encoder to extract event features (can be seen as short-term motions between 2 consecutive frames) and leverage the Temporal Convolutional Block to learn long-term motions. Furthermore, we devise a Motion Fusion Module to integrate image and motion features adaptively. It contains a Channel Attention layer and a Spatial Attention Layer to blend cross-channel and spatial information. By leveraging both modalities, EVSNet extracts richer and complementary information, leading to more accurate segmentation compared to the single modality. Finally, we use a Temporal Decoder to exploit video contexts and generate final predictions. Extensive evaluations using three large-scale low-light datasets show EVSNet results in better semantic segmentation results. In summary, our contributions to this paper include: • We propose EVSNet, a lightweight framework that exploits image and event modality. To the best of our knowledge, we are the first to introduce event modality to video semantic segmentation task. Event information focuses on the motion changes and thus can be used to learn better short and long-term temporal consistent representations. • We propose a Motion Extraction Module (MEM) and a Motion Fusion Module (MFM) for learning temporal motion and adaptively learning the spatial and channel-wise relationship between image and motion features. Unlike existing extraction and fusion modules, our design alleviates misalignment while lowering computational cost. • We conduct experiments to evaluate our EVSNet model using three large-scale low-light video semantics segmentation datasets and demonstrate its effectiveness using standard segmentation metrics. Compared to SOTA models with similar parameter efficiency and inference cost, our EVSNet achieves superior performance on these 3 datasets. 2 Related Works 2.1 Event-Based Semantic Segmentation Event cameras have the potential for semantic segmentation and there have been several efforts exploring event modality. Some researchers use knowledge distillation [ 46 ] to solve domain shift problems. ESS [ 50 ] proposed to leverage unsupervised domain adaptation by aligning event features with image features. EvDistill [ 54 ] designed a student network on unlabeled event data to distill knowledge from a teacher network trained with labeled data. Some researchers use other ways to encode event features. Ev-SegNet [ 1 ] built an Xception-based CNN to train on event data. HALSIE [ 4 ] proposed to use hybrid spiking neural networks [ 18 ] and convolution neural networks to extract spatiotemporal features to combine events

Read the complete text