β β π― β π± β - β B β e β n β c β h \mathcal{RTV}\text{-}Bench : Benchmarking MLLM Continuous Perception, Understanding and Reasoning through β \mathcal{R} eal- π― \mathcal{T} ime π± \mathcal{V} ideo
Shuhang Xun, Sicheng Tao, Jungang Li, Yibo Shi, Zhixin Lin, Zhanhui Zhu, Yibo Yan Hanqian Li, Linghao Zhang, Shikang Wang, Yixin Liu, Hanbo Zhang, Ying Ma, Xuming Hu Β· 2025
Read β β π― β π± β - β B β e β n β c β h \mathcal{RTV}\text{-}Bench : Benchmarking MLLM Continuous Perception, Understanding and Reasoning through β \mathcal{R} eal- π― \mathcal{T} ime π± \mathcal{V} ideo by Shuhang Xun, Sicheng Tao, Jungang Li, Yibo Shi, Zhixin Lin, Zhanhui Zhu, Yibo Yan Hanqian Li, Linghao Zhang, Shikang Wang, Yixin Liu, Hanbo Zhang, Ying Ma, Xuming Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02064.txt.
Introduction
Shuhang Xun, Sicheng Tao, Jungang Li, and a team of co-authors build a benchmark for testing how well large multimodal models follow real-time video. Most tests use still frames or short clips, yet real streams keep changing, and a model must hold a coherent grasp of a scene as events unfold.
Their benchmark, RTV-Bench, pairs hundreds of varied videos with thousands of questions asked at several moments in each stream. The questions step from plain seeing up to reasoning, and scoring covers perception, understanding, and reasoning over time. When the team ran many leading models through it, real-time models beat offline ones but still trailed the top closed systems, and simply feeding more frames did not reliably help.
Catalog record
| CREATOR | Shuhang Xun, Sicheng Tao, Jungang Li, Yibo Shi, Zhixin Lin, Zhanhui Zhu, Yibo Yan Hanqian Li, Linghao Zhang, Shikang Wang, Yixin Liu, Hanbo Zhang, Ying Ma, Xuming Hu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-02064
Browse the full library Β· About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
β β π― β π± β - β B β e β n β c β h \mathcal{RTV}\text{-}Bench : Benchmarking MLLM Continuous Perception, Understanding and Reasoning through β \mathcal{R} eal- π― \mathcal{T} ime π± \mathcal{V} ideo Shuhang Xun, Sicheng Tao, Jungang Li, Yibo Shi, Zhixin Lin, Zhanhui Zhu, Yibo Yan Hanqian Li, Linghao Zhang, Shikang Wang, Yixin Liu, Hanbo Zhang, Ying Ma, Xuming Hu Abstract Multimodal Large Language Models (MLLMs) increasingly excel at perception, understanding, and reasoning. However, current benchmarks inadequately evaluate their ability to perform these tasks continuously in dynamic, real-world environments. To bridge this gap, we introduce β β π― β π± β - β B β e β n β c β h \mathcal{RTV}\text{-}Bench , a fine-grained benchmark for MLLM real-time video analysis. β β π― β π± β - β B β e β n β c β h \mathcal{RTV}\text{-}Bench includes three key principles: (1) Multi-Timestamp Question Answering (MTQA) , where answers evolve with scene changes; (2) Hierarchical Question Structure , combining basic and advanced queries; and (3) Multi-dimensional Evaluation , assessing the ability of continuous perception, understanding, and reasoning. β β π― β π± β - β B β e β n β c β h \mathcal{RTV}\text{-}Bench contains 552 diverse videos (167.2 hours) and 4,631 high-quality QA pairs. We evaluated leading MLLMs, including proprietary (GPT-4o, Gemini 2.0), open-source offline (Qwen2.5-VL, VideoLLaMA3), and open-source real-time (VITA-1.5, InternLM-XComposer2.5-OmniLive) models. Experiment results show open-source real-time models largely outperform offline ones but still trail top proprietary models. Our analysis also reveals that larger model size or higher frame sampling rates do not significantly boost β β π― β π± β - β B β e β n β c β h \mathcal{RTV}\text{-}Bench performance, sometimes causing slight decreases. This underscores the need for better model architectures optimized for video stream processing and long sequences to advance real-time video analysis with MLLMs. Our benchmark toolkit is available at Project Page . * * footnotetext: Both authors made equal core contributions. $\dagger$ $\dagger$ footnotetext: Corresponding author. Figure 1: Specific examples of different tasks in RTV-Bench. and represent the fundamental questions in the question group, and the answers are underlined. is a dynamically answered question, and the correct answer is determined by the time of the query. The correct answer to the query will change at different times, and we have marked the correct answers that should be responded to at different query times. 1 Introduction The ability to comprehend and respond to complex real-world scenarios in real-time represents a crucial challenge in the pursuit of general artificial intelligence Dang et al. 2024 ; Zhang et al. 2025b ; Hu et al. 2025 . Drawing inspiration from the remarkable achievements of large language models (LLMs) in a wide range of tasks Liu et al. 2024 ; Huang et al. 2025 , the potential of multimodal large language models (MLLMs) is increasingly being realized in the domain of visual scene analysis. The research trajectory of Video-LLMs Lin et al. 2023 ; Maaz et al. 2024 ; Cheng et al. 2024 ; Wang et al. 2024a is progressively advancing from the understanding of short, vision-centric videos Li et al. 2023a ; Jin et al. 2024 ; Lin et al. 2024a ; Li et al. 2024d to the integration of omni-modal informationβincluding video, audio, and subtitles Chen et al. 2023 ; Duan et al. 2023 ; Li et al. 2023b ; Tan et al. 2024 ; Li et al. 2024c βfor comprehensive long video analysis. Recently, VStream Zhang et al. 2024a was the first to attempt to test this capability, with a focus primarily on extending the duration of videos. In addition, both StreamingBench Xiong et al. 2025 and OVOBench Li et al. 2025 have made varying degrees of improvements in the types and standards of assessment. However, their evaluation of real-time responsiveness is often inadequate, overlooking the capacity to capture transitions and fleeting details from visual input that arrive sequentially, not instantaneously. This limitation highlights the need for a more focused assessment of continuous analysis abilities. Based on the above considerations, we introduce \mathcal{RTV}\text{-}Bench , featuring three core innovations designed to benchmark the continuous analysis capabilities βspecifically perception, understanding, and reasoningβof MLLMs within real-time video contexts. First, the Multi-Timestamp Q&A mechanism challenges real-time tracking and state update by posing queries whose answers evolve within a video. Crucially, unlike benchmarks like OVO-Bench Li et al. 2025 that typically introduce different questions at different timestamps, RTV-Bench revisits the same conceptual query, where only the correct answer shifts as the scene unfolds. This approach more rigorously tests the model capacities for continuous analysis in real-time scenarios, surpassing single-query-single-answer and static evaluations. Second, the Hierarchical Question Structure enforces reliable, sequential reasoning by employing a basic-to-advanced design where higher-order questions logically depend on grasping foundational perceptions and understanding, thus mitigating cognitive shortcuts. Finally, Multidimensional Evaluation moves beyond single scores to provide granular diagnostics, assessing performance across eight fine-grained dimensions critical to continuous analysis in dynamic scenarios, offering a more insightful perspective on model capabilities and bottlenecks in handling real-time video information, detailed cases are shown in Figure 1 . Through RTV-Benchβs novel design and the detailed evaluations presented herein, we provide comprehensive insights into the current state of MLLMs for continuous video analysis: Results for all tested models on RTV-Bench reveal significant bottlenecks for existing MLLMs in real-time video analysis, as (1) most models achieve accuracy below 50%. Furthermore, (2) model performance shows minimal correlation with the number of input frames or model size. Notably, (3) models specifically designed for real-time video significantly outperform traditional offline video models; even the lowest-performing real-time model tested (VITA-1.5 Fu et al. 2025 ) surpasses a traditional offline counterpart (VideoLLaMA2 Cheng et al. 2024 ). Building upon these findings, we further discuss promising research trajectories to enhance MLLM online video processing. We expect that our benchmark, analysis, and outlined future directions will contribute significantly to the development of more robust and capable online video MLLMs. 2 RTV-Bench Figure 2: Video Categories and Distribution of Question Difficulty and Query Characteristics. (Left) RTV-Bench overs 3 key domains and 16 sub-class video types. (Center) Distribution of question difficulty levels across eight representative task types, measured by percentage-based performance ranges. (Right) Distribution of question queries by video length, categorized into Shallow, Moderate, and Deep levels. The bar heights indicate counts, while the line chart overlays query proportions for each duration bucket. In this section, we discuss a common challenge in both long video comprehension and real-time video analysisβthe continuous analysis capability of MLLMs. To tackle this, we have meticulously curated a benchmark to assess model performance in real-time and continuous scenarios, and developed new metrics to characterize the performance of multimodal large models in this bench. 2.1 Challenge for Video Analysis Existing benchmarks for long video analysis Fu et al. 2024 ; Wu et al. 2025 attempt to gauge the true capacity of MLLMs by extending video duration, enhancing the difficulty of QA tasks, and introducing new sub-tasks. Other works Lin et al. 2024b focus on real-time video scenarios, evaluating model responsiveness to queries in real-time contexts. However, current models face the risk of memory loss and attention shift, both in long and real-time video contexts. We categorize