GERD: Geometric event response data generation
Jens Egholm Pedersen, Dimitris Korakovounis, Jörg Conradt · 2024
Read GERD: Geometric event response data generation by Jens Egholm Pedersen, Dimitris Korakovounis, Jörg Conradt (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03259.txt.
Introduction
Event cameras, which report only changes in a scene, are fast and frugal but hard to study, because real recordings rarely say exactly how the view moved. Jens Egholm Pedersen, Dimitris Korakovounis, and Jörg Conradt built a simulator, GERD, to fill that gap in a 2024 paper. It renders objects under precisely controlled shifts, stretches, and timing changes, and records the true motion at every step.
The tool adds three kinds of sensor noise and allows motion finer than one pixel. The authors show it can train and test models that claim geometric guarantees, and they release it openly as a companion to real sensor data.
Catalog record
| CREATOR | Jens Egholm Pedersen, Dimitris Korakovounis, Jörg Conradt |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03259
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GERD: Geometric event response data generation Thanks: EC Horizon 2020 Framework Programme under Grant Agreements 785907 and 945539 (J.E.P., D.K., J.C.), the Pioneer Centre for AI, under the Danish National Research Foundation grant number P1 (J.E.P, D.K). Jens Egholm Pedersen, Dimitris Korakovounis, Jörg Conradt Abstract Event-based vision sensors are appealing because of their time resolution, higher dynamic range, and low-power consumption. They also provide data that is fundamentally different from conventional frame-based cameras: events are sparse, discrete, and require integration in time. Unlike conventional models grounded in established geometric and physical principles, event-based models lack comparable foundations. We introduce a method to generate event-based data under controlled transformations. Specifically, we subject a prototypical object to transformations that change over time to produce carefully curated event videos. We hope this work simplifies studies for geometric approaches in event-based vision. GERD is available at https://github.com/ncskth/gerd I Introduction Event-based vision sensors capture sparse, asynchronous data, providing significant advantages in temporal resolution, dynamic range, and power efficiency. But the events are discrete and distributed in time and space, unlike traditional RGB-frames, which have been the subject of much work in computer vision. To compete with conventional frame-based models, there is a need to improve our theoretical understanding of the underlying principles that govern the spatio-temporal “point clouds” from event cameras. Similar to how recent advances in geometric [ 3 ] and deep learning [ 13 ] revolutionized the processing of images, we expect a more fundamental understanding can significantly improve the performance of event-based computer vision models and eventually compete with conventional frame-based models. This is in line with contemporary machine learning research which have invested significant efforts into pre-processing and data augmentation, claiming that generalization in the data distribution is the most important factor towards good performance [ 18 ] . We contribute a simulator that generates event-based recordings of objects subject to carefully controlled transformations. The resulting data is well-suited to test the robustness and generalization of event-based computer vision models. We designed our simulator to strongly resemble data from an event-camera, but they should not be considered equivalent. Instead, we focus on studying the spatio-temporal structure of event data streams under controlled conditions as a prerequisite for the processing of real-world data. I-A Related work Event-based datasets are still in their infancy and are vastly outnumbered by frame-based computer vision datasets. The existing event-based datasets can be divided into two classes: (1) recorded using physical sensors or (2) generated in simulation. I-A 1 Recorded event-based datasets The N-MNIST dataset [ 20 ] is an early attempt to transfer the existing MNIST handwritten digits dataset [ 14 ] from classical machine learning to event-based vision. The dataset records digits on an LCD monitor with an event-based vision sensor. While this dataset consists of event bases recordings, the effect of the low frequency frame update of the LCD monitor used does not generate realistic event streams. Other datasets of continuous signals recorded using event sensors have been developed, such as the DVS-Gesture [ 1 ] , and SHD and SSC [ 4 ] that consist of classification tasks for visual and auditory event base signals respectively. Other datasets only contain event streams with annotations [ 26 , 24 ] or various modalities such as recording from frame-based vision sensors [ 7 ] , IMU and other sensors [ 8 ] , but no information about the geometrical structure and transformation of the objects has been extracted and provided. I-A 2 Event-based generators A different method to obtain event-based data is to artificially generate them. Approaches mostly differ on whether the generation comes as a transformation of existing frame sequences or generating new scenes. Converting sequences of frames to events is usually done by subtracting consecutive frames and applying a threshold. If the difference exceeds the threshold, an event is generated [ 11 ] . In [ 2 ] , Bi et al. also account for the contrast around the pixel. However, these approaches do not account for the timing in between the frames. To address this, Mueggler et al. [ 19 ] propose interpolating linearly between the frames and obtaining the timing of the event at the exact time that the log luminance surpasses the threshold. Other methods, such as using rate encoding on the threshold, have been proposed as well [ 2 , 5 ] . To produce more realistic event streams, [ 16 ] and [ 9 ] account for characteristics of the event sensors, such as leak currents and their effect on event generation and hot pixel generation. Another family of works uses rendered images to produce realistic event data [ 12 , 19 , 25 , 15 ] . This permits controlling the frame rate, to optionally provide very high frame rates [ 15 ] , or dynamical sampling in cases where the luminance changes fast or significant pixel displacement is noted [ 25 ] . In [ 6 ] , Gehrig et al. upsample a video in the temporal domain to obtain finer temporal resolution before using ESIM [ 25 ] for event generation. Deep learning approaches have also been proposed, such as in [ 27 ] , where a UNet is trained to predict the event streams from image sequences and in [ 28 ] , where a GAN network is trained to produce realistic event data. II Event generation method Fig. 1: The three built-in shape templates in the dataset are used to generate sparse signals when moved. (a) The prototypical shapes: square, circle, and triangle. (b) The shapes translated to the right. (c) The difference between two frames generates a sparse frame with positive changes in green and negative changes in red. We generate events by (1) rendering a shape at a given position (see Figure 1 a), (2) subjecting it to a transformation (e.g., translation in Figure 1 b), and then (3) taking the difference between the two frames (as in Figure 1 c). The output is a sparse tensor in pixel coordinates with two channels: positive and negative changes (green and red in Figure 1 c, respectively). By repeating the procedure frame-by-frame, we can generate arbitrarily long videos. Fig. 2: By controlling the amount of transformation, we can control the amount of signal per time step. Here, a circle is shrinking by three different amounts for a single timestep, producing increasingly sparse frames. The red color indicates a negative polarity change. Changing the magnitude of the transformations produces arbitrarily sparse recordings, since our transformations determine the amount of change per frame (see Figure 2 ). In real life, subjecting an object to tiny transformations over time produces highly sparse frames, which means that the object is either moving very slowly or that the timesteps between the frames are minimal. Since the generated frames are not bound by physical time, these interpretations are equivalent. Therefore, the dataset is suitable for settings with arbitrarily small timesteps or transformational speeds. The simulator is built using PyTorch [ 21 ] and can be accelerated with hardware accelerators supported by PyTorch, such as graphical or tensor processing units. We represent the frames as sparse tensors using an address-event representation, which can easily be converted to dense frames if