EventBench: Towards Comprehensive Benchmarking of Event-based MLLMs
Shaoyu Liu, Jianing Li, Guanghui Zhao, Yunjian Zhang, Xiangyang Ji · 2025
Read EventBench: Towards Comprehensive Benchmarking of Event-based MLLMs by Shaoyu Liu, Jianing Li, Guanghui Zhao, Yunjian Zhang, Xiangyang Ji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18448.txt.
Introduction
Shaoyu Liu, Jianing Li, Guanghui Zhao, Yunjian Zhang, and Xiangyang Ji build EventBench, a test suite for models that read event camera streams. Event cameras report pixel change as it comes, not frames, which suits fast motion and harsh light. Multimodal models have grown strong here, but no shared test yet spans their full skill range.
EventBench stands out in four ways. All raw streams and task prompts are open. Tasks span reading, naming, and space reasoning across eight scores. It adds 3D space tasks, a first for this field. A training set tops a million event text pairs. Scores across models show where event based reading lags and where it leads.
Catalog record
| CREATOR | Shaoyu Liu, Jianing Li, Guanghui Zhao, Yunjian Zhang, Xiangyang Ji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18448
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EventBench: Towards Comprehensive Benchmarking of Event-based MLLMs Shaoyu Liu, Jianing Li, Guanghui Zhao, Yunjian Zhang, Xiangyang Ji Abstract Multimodal large language models (MLLMs) have made significant advancements in event-based vision. However, the comprehensive evaluation of these models’ capabilities within a unified benchmark remains a crucial yet largely unexplored aspect. In this paper, we introduce EventBench , which presents an evaluation benchmark covering 8 diverse task metrics and a large-scale event stream dataset. Our work distinguishes existing event-based benchmarks in four key aspects: i) Openness in accessibility , releasing all raw event streams and task instructions across 8 evaluation metrics; ii) Diversity in task coverage , covering diverse understanding, recognition, and spatial reasoning tasks, enabling comprehensive evaluation of model capability; iii) Integration in spatial dimensions , pioneering the design of 3D spatial reasoning task for event-based MLLMs; iv) Scale in data volume , with an accompanying training set of over one million event–text pairs supporting large-scale training and evaluation. With EventBench, we evaluate state-of-the-art closed-source models such as GPT-5 and Gemini-2.5 Pro, leading open-source models including Qwen2.5-VL and InternVL3, as well as event-based MLLMs like EventGPT that directly process raw event inputs. Extensive evaluation reveals that current event-based MLLMs perform well in event stream understanding, yet they still struggle with fine-grained recognition and spatial reasoning. 1 Introduction Figure 1 : Our EventBench is a publicly accessible and comprehensive evaluation benchmark for event-based MLLMs. It offers diverse task metrics across multiple dimensions (i.e., understanding, recognition, and spatial reasoning) and will accelerate research on event-based MLLMs in challenging scenarios. Table 1 : Comparison of event-based MLLM datasets and benchmarks. We compare datasets in terms of scale, modalities, evaluation metrics, and four key attributes: \mathbf{\Delta d} = Downstream task support, \mathbf{\Delta s} = Spatial task support, \mathbf{\Delta t} = Training data release, \mathbf{\Delta b} = Benchmark release. Time ranges are categorized as short (50 ms), medium (50 ms–10 s), and long ( > 10 s). ( RGB, Event Stream, and Text) Dataset Venue Size Modalities Metric Types Data Collection Source Temporal Span Attributes \mathbf{\Delta d} \mathbf{\Delta s} \mathbf{\Delta t} \mathbf{\Delta b} N-ImageNet 28 ICCV’21 1M – Synthesis Short ✗ ✗ ✓ ✓ Talk2Event 30 NeurIPS’25 30.7K – Real World Short ✓ ✗ ✗ ✗ EventChat 37 CVPR’25 1.1M 3 Synthesis & Real World Short ✗ ✗ ✓ ✗ EventVL-Bench 34 arXiv’25 1.4M 4 Synthesis & Real World Short & Medium ✓ ✗ ✗ ✗ LLaFEA-Bench 72 ICCV’25 – 3 Synthesis & Real World Short & Medium ✓ ✗ ✗ ✗ EVQA-Bench 12 arXiv’25 1.0M 4 Synthesis & Real World Short & Medium & Long ✗ ✗ ✗ ✗ EventBench Ours* 1.4M 8 Synthesis & Real World Short & Medium & Long ✓ ✓ ✓ ✓ Event cameras, offering high temporal resolution and high dynamic range, have been widely applied in challenging scenarios 20 ; 59 ; 69 ; 60 ; 74 ; 64 ; 65 ; 35 ; 29 ; 11 ; 55 ; 44 ; 68 ; 54 ; 23 ; 53 ; 9 ; 21 ; 47 . With the rapid development of multimodal large language models (MLLMs) 3 ; 70 ; 51 ; 32 ; 43 in recent years, event-based MLLMs 37 ; 34 ; 72 ; 12 ; 46 ; 71 ; 31 ; 38 ; 67 ; 25 have increasingly demonstrated strong capabilities and generality in understanding asynchronous event streams. While video-based MLLMs have been extensively benchmarked across diverse tasks (e.g., understanding, recognition, and spatial reasoning) 19 ; 33 ; 57 ; 75 ; 50 ; 24 , comprehensive evaluation benchmarks for event-based MLLMs remain largely unexplored. Most existing event-based MLLMs achieve strong performance on their respective evaluation tasks, lacking a publicly available and unified benchmark to comprehensively assess their general capabilities across diverse dimensions. Pioneering works such as EventGPT 37 and EventVL 34 present instructional datasets for scene understanding and visual question answering. Subsequent efforts, such as LLaFEA 72 and LET-US 12 , expand the scope to incorporate spatiotemporal localization and long-term event stream understanding. While these event-based MLLMs primarily focus on evaluating specific aspects of model capability, their assessment dimensions remain relatively limited compared with the mainstream benchmarks for video-based MLLMs (e.g., understanding, recognition, and spatial reasoning). In other words, this gap highlights the need for a unified benchmark to comprehensively evaluate event-based MLLMs across diverse dimensions. Moreover, the limited public accessibility of existing event-based benchmarks 30 ; 74 ; 63 ; 76 ; 16 ; 45 ; 42 ; 26 further restricts fair comparison, making it challenging to accurately gauge progress in event-based MLLMs. In this work, we introduce EventBench, a comprehensive evaluation benchmark covering 8 diverse task metrics and a large-scale event stream dataset. To achieve our objective, two main challenges require to be addressed: (i) How can we design a systematic and comprehensive evaluation benchmark that measures capabilities across multiple dimensions, such as understanding, recognition, and spatial reasoning? How can we employ various MLLMs to validate the proposed benchmark’s effectiveness across different tasks, ensuring it serves to foster progress in this topic? To address these challenges, we introduce a comprehensive benchmark that systematically evaluates event-based MLLMs across diverse dimensions (see Fig. 1 ). It comprises 8 representative tasks covering understanding, recognition, and spatial reasoning. It integrates over 20 heterogeneous data sources drawn from both synthetic and real-world event streams to ensure rich scene diversity. Importantly, it pioneers the introduction of spatial reasoning evaluation for event-based MLLMs, filling a gap in assessing models’ 3D understanding capabilities. Furthermore, we present a fully open and publicly accessible evaluation pipeline that enables different models to be evaluated under an unified benchmark, providing the opportunity to explore this evolving research topic. In addition, we construct a large-scale and carefully curated instruction dataset (i.e., EQA-1.4M) to enhance the baseline performance of open-source event-based MLLMs. We also establish objective metrics (e.g., multiple-choice accuracy, text matching, and numerical evaluation) to ensure fairness and reproducibility. To evaluate the effectiveness of EventBench across different tasks, we assess both state-of-the-art closed-source models (i.e., GPT-5 41 , GPT-4o 40 , Gemini 2.5-Pro 14 , and Gemini 2.5-Flash 14 ) as well as leading open-source MLLMs (i.e., Qwen2.5-VL 3 , InternVL 3.5 58 , MiMo-VL 51 , and MiniCPM4-V 52 ). We further benchmark open-source event-based MLLMs (i.e., EventGPT 37 ). The results demonstrate that EventGPT+, trained with our large-scale instruction dataset EQA-1.4M, achieves the best overall performance. Moreover, fine-tuning open-source MLLMs on EQA-1.4M leads to substantial performance gains, highlighting the dataset’s effectiveness in enhancing event-based model performance. Our main contributions can be summarized as follows: • We present EventBench , a publicly accessible evaluation benchmark with a large-scale event stream dataset, which will accelerate research on event-based MLLMs. • We design a comprehensive benchmark covering 8 diverse task metrics for understanding, recognition, and spatial reasoning, which significantly expands the range of evaluation tasks compared to existing benchmarks. • We build EQA-1.4M, a large-scale dataset of over one million synthetic and real-world event streams for event-based MLLMs, considering scene diversity across 20 data sources and various lengths of event streams. • We conduct extensive experiments to compare state-of-the-art closed-source MLLMs, open-source MLLMs, and event-based MLLMs. The experimental results provide a guiding benchmark for event-based MLLMs. Figure 2