Flex Event : Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies

Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, Wei Tsang Ooi · 2024

Read Flex Event : Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies by Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, Wei Tsang Ooi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06708.txt.

Introduction

Event cameras report only the pixels that change, which makes them fast but hard to pair with normal video. Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, and Wei Tsang Ooi build a detector in 2024 that fuses both streams at whatever rate the job needs.

Most event detectors run at one fixed frequency and waste the sensor's speed. The framework, FlexEvent, has two parts: a fusion module that blends high-rate event data with rich color frames, and a tuning step that relabels training data for each target frequency. It stayed accurate from 20 up to 180 detections per second in tests, beating prior methods in both normal and high-speed settings. Fast and slow scenes are both handled well.

Catalog record

CREATORDongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, Wei Tsang Ooi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06708

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Flex Event : Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies Dongyue Lu, Lingdong Kong, Gim Hee Lee, Camille Simon Chane, Wei Tsang Ooi Abstract Event cameras offer unparalleled advantages for real-time perception in dynamic environments, thanks to the microsecond-level temporal resolution and asynchronous operation. Existing event detectors, however, are limited by fixed-frequency paradigms and fail to fully exploit the high-temporal resolution and adaptability of event data. To address these limitations, we propose Flex Event , a novel framework that enables detection at varying frequencies. Our approach consists of two key components: FlexFuse , an adaptive event-frame fusion module that integrates high-frequency event data with rich semantic information from RGB frames, and FlexTune , a frequency-adaptive fine-tuning mechanism that generates frequency-adjusted labels to enhance model generalization across varying operational frequencies. This combination allows our method to detect objects with high accuracy in both fast-moving and static scenarios, while adapting to dynamic environments. Extensive experiments on large-scale event camera datasets demonstrate that our approach surpasses state-of-the-art methods, achieving significant improvements in both standard and high-frequency settings. Notably, our method maintains robust performance when scaling from 20 20 Hz to 90 90 Hz and delivers accurate detection up to 180 180 Hz, proving its effectiveness in extreme conditions. Our framework sets a new benchmark for event-based object detection and paves the way for more adaptable, real-time vision systems. Figure 1: Illustrative examples of event camera object detection at varying frequencies . The detection performance of the classic RVT detector [ 18 ] tends to drop significantly at higher event operational frequencies. Motivated by this observation, we propose Flex Event , a robust and flexible event-frame detector that maintains high detection accuracy across a wide range of frequencies (low, middle, high), ensuring strong adaptability in real-world, dynamic sensing environments. 1 Introduction Event cameras have garnered significant attention for their ability to capture dynamic scenes with microsecond-level temporal resolution [ 12 ] . Unlike RGB cameras, which capture entire frames at fixed intervals, event cameras operate asynchronously, responding to changes in pixel intensity at each location [ 68 ] . This low-latency operation reduces motion blur and enables highly energy-efficient sensing, making event cameras ideal for real-time applications [ 45 , 30 , 6 ] . Despite their potential, existing event detectors often fail to fully leverage the high-frequency temporal information [ 8 , 15 , 23 ] . Most approaches align event data with the lower frequency of frames by adopting fixed time intervals between event streams and frame-based annotations [ 40 , 17 ] . While this simplifies data processing, it inevitably overlooks the rich temporal details embedded in high-frequency event streams. Given that human annotations are often synchronized with slower frame rates, current detectors miss valuable information from higher frequencies, resulting in suboptimal performance when rapid object detection is required in dynamic environments [ 36 , 43 ] . To address these limitations, we introduce Flex Event , a novel framework designed to tackle the challenging problem of object detection at varying operational frequencies. Our approach addresses the need for high-frequency detection in fast-changing environments, while adapting to different operational frequencies. We propose two key innovations: 1) an adaptive event-frame fusion module, and 2) a frequency-adaptive fine-tuning mechanism. The first component, FlexFuse , addresses the limitations of event data, which often lacks semantic and texture-rich information, especially at higher frequencies [ 66 ] , by integrating the rich spatial and semantic information from RGB frames with the high-temporal resolution of event streams. It enables high detection accuracy even in fast-moving environments. Furthermore, training on high-frequency event data is computationally expensive and impractical due to the significant human effort required to label such data. FlexFuse mitigates this by sampling event data at varying frequencies, aligning them with the normal frame rate during training, thus maintaining efficiency while preserving the high-frequency benefits at inference time. The second component, FlexTune , enhances the generalization capability of event camera detectors across varying operational frequencies, by generating frequency-adjusted labels for the unlabeled high-frequency data. These labels allow the model to learn from high-frequency event streams without manual annotations, and iterative refinement through self-training ensures that the model remains robust across different motion dynamics and frequency settings. Together, these two components allow for accurate real-time detection in rapid scene changes and adapt to a wide range of operational frequencies, by leveraging the temporal richness of event data and the semantic detail of RGB frames. Our extensive experiments validate the effectiveness of Flex Event on multiple large-scale event camera datasets. Our approach consistently outperforms recent detectors across both standard and high-frequency settings. In particular, we achieve mAP gains of \mathbf{15.5\%} , \mathbf{9.4\%} , and \mathbf{10.3\%} over the previous best-performing detectors on DSEC-Det [ 16 ] , DSEC-Detection [ 50 ] , and DSEC-MOD [ 66 ] , respectively. Our model also maintains \mathbf{96.2\%} of its performance when the operational frequency shifts from \mathbf{20} Hz to \mathbf{90} Hz , and delivers accurate detection at frequencies as high as \mathbf{180} Hz , proving its robustness under extreme conditions. In summary, our contributions are listed as follows: \bullet The proposed Flex Event framework is aimed at tackling the challenging problem of event camera object detection at varying frequencies, being one of the first works to address this challenging problem explicitly in real-world conditions. \bullet We propose FlexFuse, an adaptive event-frame fusion module that leverages the strengths of both event and frame modalities, enabling more efficient and accurate event-based object detection in dynamic environments. \bullet We introduce FlexTune, a frequency-adaptive fine-tuning mechanism that generates frequency-adjusted labels and improves generalization across various motion frequencies. \bullet We demonstrate that our approach achieves promising results across large-scale datasets, particularly in high-frequency scenarios, validating its effectiveness to handle real-world, safety-critical problems. 2 Related Work Event Camera Object Detection. Event-based detectors can be broadly split into two approaches: GNNs/SNNs and dense feed-forward models. GNNs build dynamic spatio-temporal graphs by sub-sampling events [ 15 , 47 , 36 , 43 ] , but they face challenges in propagating information over large spatio-temporal regions, especially for slow-moving objects. SNNs offer efficient sparse information transmission but are often hindered by their non-differentiable nature, complicating optimization processes [ 9 , 8 , 59 ] . Dense, feed-forward models represent the second approach. Initial methods using fixed temporal windows [ 5 , 22 , 25 , 19 ] struggle with slow-moving or stationary objects due to their limited capability to capture long-term temporal data. Subsequent work incorporates RNNs and transformers to enhance temporal modeling [ 40 , 69 , 34 , 18 , 39 ] , but these models still lack semantic richness and face difficulties in adapting to variable frequencies and highly dynamic scenarios. Event-Frame Multimodal Learning. Combining events with frame-based data has proven effective in tasks such as deblurring [ 48 , 61 ] , depth estimation [ 14 , 51 ] , and tracking [ 62 , 13 ] . Early fusion methods perform post-processing combinations [ 33 , 4 ] , but lack feature-level interaction. More recent works propose deeper feature fusion techniques [ 50 , 1 , 2 ] , introducing pixel-level attention or temporal transformers for asynchronous processing [ 66 , 32 , 16 ,

Read the complete text