V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation

Hanyue Lou Jinxiu Liang Minggui Teng Yi Wang Boxin Shi · 2025

Read V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation by Hanyue Lou Jinxiu Liang Minggui Teng Yi Wang Boxin Shi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16797.txt.

Introduction

Hanyue Lou, Jinxiu Liang, and colleagues turn ordinary video into training data for event cameras in a 2025 computer vision study. Event cameras report pixel changes with microsecond timing and huge dynamic range, but learning algorithms for them starve. Real event data are scarce, and existing simulators produce files too bulky to move or store. Their video-to-voxel pipeline simulates event streams efficiently enough to scale.

Compact simulated data let event vision models train at the scale their accuracy demands. Better event models serve fast robotics and low-power sensing, where ordinary cameras blur or saturate.

Catalog record

CREATORHanyue Lou Jinxiu Liang Minggui Teng Yi Wang Boxin Shi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16797

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation Hanyue Lou Jinxiu Liang Minggui Teng Yi Wang Boxin Shi Abstract Event-based cameras offer unique advantages such as high temporal resolution, high dynamic range, and low power consumption. However, the massive storage requirements and I/O burdens of existing synthetic data generation pipelines and the scarcity of real data prevent event-based training datasets from scaling up, limiting the development and generalization capabilities of event vision models. To address this challenge, we introduce Video-to-Voxel (V2V), an approach that directly converts conventional video frames into event-based voxel grid representations, bypassing the storage-intensive event stream generation entirely. V2V enables a 150× reduction in storage requirements while supporting on-the-fly parameter randomization for enhanced model robustness. Leveraging this efficiency, we train several video reconstruction and optical flow estimation model architectures on 10,000 diverse videos totaling 52 hours—an order of magnitude larger than existing event datasets, yielding substantial improvements. † † footnotetext: ∗ Corresponding author. † † footnotetext: 1 Code available: https://github.com/HYLZ-2019/V2V 1 Introduction Event-based cameras [ 8 ] are bio-inspired visual sensors that asynchronously record per-pixel intensity changes, offering high temporal resolution and high dynamic range with low power consumption, compared to traditional frame-based cameras. These properties make event cameras particularly promising for applications involving high-speed motion, challenging lighting conditions, and resource-constrained environments such as robotics, autonomous driving, and augmented reality. Despite their theoretical advantages, event cameras have yet to match the practical performance of conventional cameras. This performance gap can be attributed to several factors, with the scarcity of large-scale training data constituting a critical bottleneck. While deep learning approaches have revolutionized frame-based computer vision through decades of dataset collection and massive repositories like ImageNet [ 4 ] , COCO [ 19 ] , and WebVid [ 2 ] , event camera data remains scarce and limited in diversity. The restricted commercial deployment of event cameras creates a fundamental chicken-and-egg problem: widespread adoption requires robust algorithms, which in turn depend on diverse training data that can only come from widespread deployment. To address this data limitation, researchers have developed simulation pipelines that convert conventional visuals into synthetic event ones. However, significant challenges remain in both approximating the high temporal dynamics of event cameras and utilizing these simulations efficiently. Standard video datasets typically have frame rates of around 30 per second (FPS), providing temporal resolution orders of magnitude lower than event cameras. This fundamental mismatch creates a significant hurdle for data simulation. The field has consequently developed two primary simulation approaches, each with distinct limitations: Video-based simulators like V2E [ 13 ] attempt to bridge this temporal gap through frame interpolation techniques; however, they inevitably introduce artifacts and interpolation errors since they attempt to reconstruct information that was never captured in the original footage. Model-based simulators like those in E2VID+ [ 32 ] render high frame rate videos from 3D scene models, providing precise control of the temporal resolution but suffering from significant realism limitations. In practice, these methods often resort to simplistic approximations, such as randomly flying 2D images against static backgrounds – far removed from real-world scene complexity, as illustrated in topleft of Figure . Beyond realism concerns, both approaches encounter severe storage and computational inefficiency. Unlike conventional video data, which benefits from decades of codec optimization, event data compression remains relatively underdeveloped [ 30 , 14 ] . Consequently, storing and processing synthetic event data becomes exceedingly resource-intensive. For context, a single 10-second sequence from the ESIM-280 dataset occupies approximately 1GB. If we were to scale to 10,000 diverse scenes, substantial I/O bottlenecks during training will emerge. Moreover, since each event stream corresponds to a specific camera configuration, exploring multiple parameter variations (thresholds, noise levels, etc . ) multiplies storage requirements by the number of configurations. These technical challenges create a significant barrier to training on large-scale, diverse datasets, leading us to question: Is explicit event stream synthesis truly necessary for model training? Our analysis reveals a surprising answer: No . We observe that most event-based deep learning pipelines convert asynchronous events into dense representations (particularly voxel grids) before neural network processing. This creates an opportunity to directly target these representations instead of generating intermediate event streams. In this paper, we introduce Video-to-Voxel (V2V), a principled approximation that efficiently generates discrete voxel representations directly from video frames. This approach bypasses the computationally expensive event generation step while preserving the essential spatial-temporal information needed for learning. The key advantages of our approach are illustrated in Figure : • Efficiency. Our approach eliminates the need to store intermediate event streams, drastically saving storage resources (up to 150 times) and enabling the use of substantially larger datasets. • Flexibility. Our approach allows on-the-fly parameter randomization during training, enabling models to learn more robust representations from diverse virtual camera configurations without multiplicative storage requirements. • Scalability. This efficiency enables direct utilization of massive internet-scale video repositories like WebVid [ 2 ] , creating a training dataset containing 10,000 videos with a total duration of 52 hours—an order of magnitude larger than existing event camera datasets. We validate the effectiveness of the V2V module on two fundamental event-based visual dense prediction tasks: event-based video reconstruction and optical flow estimation. In particular, our method enables the training of existing models on a diverse dataset with 35 times more scene variations than the previous practice, with performance scaling with dataset size. As shown in the right of Figure , this leads to significant improvements in reconstruction quality. These results demonstrate that our V2V pipeline effectively addresses the data scarcity bottleneck in event-based vision, opening new possibilities for developing robust event-based algorithms. 2 Related Works Synthetic event data. Video-based and model-based event simulators have been proposed to generate synthetic event data. Video-based simulators [ 13 , 18 , 39 , 42 , 44 ] synthesize events from videos, using frame interpolation algorithms [ 13 ] or learning-based methods [ 42 , 39 ] to compensate the lack of temporal information due to limited video frame rate. This requries reconstructing intermediate intensity changes that were never captured—an ill-posed problem. Model-based event simulators such as ESIM [ 27 ] and PECS [ 12 ] render events from synthetic 3D scenes, providing fully precise temporal information. However, the diversity and realism of the scenes are limited. Random scenes with flying 2D [ 32 ] or 3D [ 33 ] objects and elaborate models of city scenes [ 22 ] have been used, but complex real-world phenomena, such as with non-rigid movement, fluid dynamics, and complex lighting interactions, remain difficult to model. Training data of event-based video reconstruction. The development of video reconstruction from event streams (E2VID) has been predominantly constrained by available training data. Existing learning-based E2VID methods use model-based simulated events or real events for training. E2VID [ 28 ] proposed to use the ESIM simulator for training data generation, and Stoffregen et al . [ 32 ] improved model performance by using diversified thresholds in ESIM. FireNet [ 29 ] , SPADE-E2VID [ 3 ] , ETNet [ 34 ] , Event-Diffusion [ 17 ] , HyperE2VID [ 7 ] , EVSNN [ 43 ] and TFCSNN [ 36 ] all used ESIM data for training. Mostafavi et al . [ 24 ] used

Read the complete text