VideoReasonBench : Can MLLMs Perform Vision-Centric Complex Video Reasoning?

Yuanxin LiuKun OuyangHaoning WuYi Liu Lin SuiXinhao LiYan ZhongY. Charles, Xinyu ZhouXu Sun State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University Moonshot AI Nanjing University School of Mathematical Sciences, Peking University · 2025

Read VideoReasonBench : Can MLLMs Perform Vision-Centric Complex Video Reasoning? by Yuanxin LiuKun OuyangHaoning WuYi Liu Lin SuiXinhao LiYan ZhongY. Charles, Xinyu ZhouXu Sun State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University Moonshot AI Nanjing University School of Mathematical Sciences, Peking University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23359.txt.

Introduction

Yuanxin Liu and co-authors test whether video models can truly reason from what they see in this 2025 paper. Long step-by-step reasoning has boosted text models on hard tasks. For video, most tests lean on stored knowledge rather than the pictures, so the gain was unproven.

Their benchmark, VideoReasonBench, packs videos that are rich in visual detail and demand deep chains of thought. Each task forces the model to use the footage itself. Scores show how far multimodal models still sit from human-like video reasoning. The set gives builders a harder, fairer yardstick for progress.

Catalog record

CREATORYuanxin LiuKun OuyangHaoning WuYi Liu Lin SuiXinhao LiYan ZhongY. Charles, Xinyu ZhouXu Sun State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University Moonshot AI Nanjing University School of Mathematical Sciences, Peking University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23359

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VideoReasonBench : Can MLLMs Perform Vision-Centric Complex Video Reasoning? Yuanxin LiuKun OuyangHaoning WuYi Liu Lin SuiXinhao LiYan ZhongY. Charles, Xinyu ZhouXu Sun State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University Moonshot AI Nanjing University School of Mathematical Sciences, Peking University Abstract Recent studies have shown that long chain-of-thought (CoT) reasoning can significantly enhance the performance of large language models (LLMs) on complex tasks. However, this benefit is yet to be demonstrated in the domain of video understanding, since most existing benchmarks lack the reasoning depth required to demonstrate the advantages of extended CoT chains. While recent efforts have proposed benchmarks aimed at video reasoning, the tasks are often knowledge-driven and do not rely heavily on visual content. To bridge this gap, we introduce VideoReasonBench , a benchmark designed to evaluate vision-centric, complex video reasoning . To ensure visual richness and high reasoning complexity, each video in VideoReasonBench depicts a sequence of fine-grained operations on a latent state that is only visible in part of the video. The questions evaluate three escalating levels of video reasoning skills: recalling observed visual information, inferring the content of latent states, and predicting information beyond the video. Under such task setting, models have to precisely recall multiple operations in the video, and perform step-by-step reasoning to get correct final answers for these questions. Using VideoReasonBench , we comprehensively evaluate 18 state-of-the-art multimodal LLMs (MLLMs), finding that most perform poorly on complex video reasoning—e.g., GPT-4o achieves only 6.9% accuracy—while the thinking-enhanced Gemini-2.5-Pro significantly outperforms others with 56.0% accuracy. Our investigations on “test-time scaling” further reveal that extended thinking budget, while offering none or minimal benefits on existing video benchmarks, is essential for improving the performance on VideoReasonBench . Data: huggingface.co/datasets/lyx97/reasoning_videos Code: github.com/llyx97/video_reason_bench Project: https://llyx97.github.io/video_reason_bench/ Figure 1: Examples from VideoReasonBench and three existing VideoQA benchmarks. Responses are generated by Gemini-2.5-Flash in both “Thinking” and “No Thinking” modes. The text highlighted in green / red indicate correct/incorrect responses. While questions from existing benchmarks can be answered correctly without “Thinking” using only a few tokens, VideoReasonBench requires “Thinking” for accurate reasoning and consumes substantially more tokens (See Figure 5 for quantitative results). It also demands finer-grained visual perception during reasoning. 1 Introduction Recent advances in long chain-of-thought (CoT) reasoning ( DeepSeek-AI et al., 2025 ; Jaech et al., 2024 ; Team et al., 2025a ) have remarkably enhanced the problem-solving capabilities of large language models (LLMs). By scaling up the test-time compute with extended CoT reasoning chains, substantial performance gains have been observed in complex tasks such as mathematics ( MAA, 2024 ; Lightman et al., 2024 ; Lu et al., 2024 ) , coding ( Jain et al., 2024 ; Jimenez et al., 2024 ) , and scientific reasoning ( Rein et al., 2023 ) . However, the benefits of long CoT reasoning have not been fully demonstrated in the domain of video understanding. This gap is largely due to limitations in existing benchmarks ( Mangalam et al., 2023 ; Fu et al., 2024 ; Liu et al., 2024 ; Li et al., 2024b ; Wu et al., 2024 ; Zhou et al., 2024 ; Li et al., 2024c ; Shangguan et al., 2025 ) , which often lack the reasoning depth necessary to showcase the advantages of extended CoT chains. As shown in Figure 1 , the advanced multimodal LLM (MLLM) Gemini-2.5-Flash can correctly answer the questions from two popular benchmarks, Video-MME ( Fu et al., 2024 ) and TempCompass ( Liu et al., 2024 ) using only a few response tokens without activating the thinking mode. To address this gap, several benchmarks have been proposed recently to better emphasize CoT reasoning in video understanding. Video-MMMU ( Hu et al., 2025 ) and MMVU ( Zhao et al., 2025 ) integrate video understanding with domain-specific knowledge, thus introducing a need for reasoning. However, the required reasoning process is primarily knowledge-driven, lacking a strong reliance on the visual content. Two concurrent studies, VCR-Bench ( Qi et al., 2025 ) and MINERVA ( Nagrani et al., 2025 ) , evaluate the correctness of video reasoning process in addition to the final answer. Nonetheless, the videos and questions in these benchmarks often resemble those in general video understanding benchmarks, emphasizing short-horizon skills such as temporal grounding, action counting, and temporal order comprehension, while fall short in demanding deeper video reasoning. Motivated by these limitations, this work introduces the VideoReasonBench to evaluate the capabilities of MLLMs in performing vision-centric, complex video reasoning . We define three levels of video reasoning, each requiring progressively more sophisticated reasoning: The first level is to precisely recall the sequential visual observations from the video. The second level is to infer latent information that is not directly observable from the video. The third level is to predict new information beyond the video. For instance, as shown in Figure 1 , a video from VideoReasonBench presents a “sliding number puzzle”, in which numbered tiles are initially visible but become masked as sliding movements occur. To accurate answer the question, a model must first recall the initial tile arrangement and all subsequent movements (Level 1), then infer the final arrangement of tiles (Level 2), and finally apply this inferred information to predict future tile positions (Level 3). VideoReasonBench is constructed based on the aforementioned core ideas. Each video illustrates a sequence of operations (e.g., sliding movements) performed to a latent state (e.g., tile arrangement). The richness of visual information can be flexibly controlled by adjusting the size of the latent state and the number of operations. In addition to the ”sliding number puzzle”, our benchmark includes six types of video demonstrations spanning various scenes, featuring both synthetic and real-world videos. To evaluate reasoning across all three levels, we design six corresponding reasoning skills, with two for each level (see Figure 2 ). Based on VideoReasonBench , we comprehensively evaluate 18 state-of-the-art MLLMs. Our results reveal that most MLLMs struggle with vision-centric complex video reasoning, achieving accuracies below 10%. In contrast, the thinking-augmented Gemini-2.5-Pro significantly outperforms all other models, reaching an accuracy of 56%. Further analysis shows that while extended chain-of-thought (CoT) reasoning offers minimal performance improvements on existing benchmarks, it is crucial to VideoReasonBench . Additionally, we observe that removing visual information from VideoReasonBench leads to a substantially larger drop in performance compared to other benchmarks, highlighting its strong reliance on visual content. The main contributions of this work are summarized as follows: 1. We introduce the VideoReasonBench for evaluating vision-centric, complex video reasoning. It poses a necessity for models to correctly perceive multiple actions in a sequential order and perform step-by-step reasoning to finally answer the questions, therefore by-principle featuring higher demand for reasoning depth and stronger reliance on the visual content. 2. We reveal the concerning deficiency of most SOTA MLLMs in our benchmark: Several latest thinking models, such o4-mini and Seed1.5-VL, only gets around 10% accuracy; non-thinking SOTA MLLMs (e.g. GPT-4o and Qwen2.5VL-72B) scores lower than 10%; all efficient MLLMs ( < 10B) cannot reach even 2%. 3. Our experimental investigation confirms that the accuracy of Gemini-2.5-Flash drastically degrade while dropping 50% input video or disabling thinking-mode , while existing video benchmarks do not show similar properties. This result underscores

Read the complete text