V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models

Yang Luo, Xuanlei Zhao, Baijiong Lin, Lingting Zhu, Liyao Tang, Yuqi Liu, Ying-Cong Chen, Shengju Qian, Xin Wang, Yang You · 2025

Read V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models by Yang Luo, Xuanlei Zhao, Baijiong Lin, Lingting Zhu, Liyao Tang, Yuqi Liu, Ying-Cong Chen, Shengju Qian, Xin Wang, Yang You (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16668.txt.

Introduction

Yang Luo and nine co-authors introduce V-ReasonBench, a benchmark suite for measuring reasoning in video generation models, whose zero-shot reasoning feats have outrun the tools for checking them. The suite tests four dimensions: structured problem-solving, spatial cognition, pattern-based inference, and physical dynamics.

Tasks are built from synthetic and real image sets and are designed to be answer-verifiable, reproducible, scalable, and unambiguous. Six leading video models show clear dimension-by-dimension differences rather than a single pecking order. The authors also compare video models with strong image models, catalogue hallucination habits, and examine how clip duration affects chain-of-frames reasoning. The benchmark aims to push video models toward reasoning that is reliable and aligned with human judgment.

Catalog record

CREATORYang Luo, Xuanlei Zhao, Baijiong Lin, Lingting Zhu, Liyao Tang, Yuqi Liu, Ying-Cong Chen, Shengju Qian, Xin Wang, Yang You
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-16668

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models Yang Luo, Xuanlei Zhao, Baijiong Lin, Lingting Zhu, Liyao Tang, Yuqi Liu, Ying-Cong Chen, Shengju Qian, Xin Wang, Yang You Abstract Recent progress in generative video models, such as Veo-3, has shown surprising zero-shot reasoning abilities, creating a growing need for systematic and reliable evaluation. We introduce V-ReasonBench, a benchmark designed to assess video reasoning across four key dimensions: structured problem-solving, spatial cognition, pattern-based inference, and physical dynamics. The benchmark is built from both synthetic and real-world image sequences and provides a diverse set of answer-verifiable tasks that are reproducible, scalable, and unambiguous. Evaluations of six state-of-the-art video models reveal clear dimension-wise differences, with strong variation in structured, spatial, pattern-based, and physical reasoning. We further compare video models with strong image models, analyze common hallucination behaviors, and study how video duration affects Chain-of-Frames reasoning. Overall, V-ReasonBench offers a unified and reproducible framework for measuring video reasoning and aims to support the development of models with more reliable, human-aligned reasoning skills. * * footnotetext: Equal Contribution. $\dagger$ $\dagger$ footnotetext: Corresponding Author. 1 Introduction Recent advancements in generative video models ( 16 ; 44 ; 28 ; 2 ; 24 ) , such as Veo-3.1 ( 14 ) , Sora-2 ( 27 ) , and Kling-2.5 ( 19 ) , have demonstrated remarkable zero-shot reasoning capabilities across diverse visual tasks. These models exhibit emergent abilities to perceive, model, and reason over dynamic visual data, reflecting a growing trajectory toward general-purpose vision foundation models. Such progress suggests that video models are beginning to move beyond mere frame synthesis, displaying cognitive-like behaviors such as pattern inference, causal prediction, and strategic planning ( 41 ) . However, despite these promising signs, the capacity to quantitatively evaluate reasoning ability in these models remains underexplored and poorly standardized. Figure 1 : Evaluation of Video Generation Models on V-ReasonBench. The performance of six video generation models across the four core reasoning dimensions is illustrated. Detailed numerical results are provided in Tab. 2 . The “Chain-of-Frame” (CoF) ( 41 ; 12 ) paradigm treats video generation as a sequence of reasoning steps, in direct analogy to “Chain-of-Thought” in language models ( 39 ; 47 ; 38 ; 37 ; 23 ) . In this framework, a model receives an initial image and a prompt, then produces a series of frames where the intermediate frames embody its reasoning trajectory and the final frame represents its answer or outcome. Because the final frame encapsulates the model’s inferred solution, CoF enables a last-frame evaluation pipeline: we judge the model on its concluding frame rather than requiring annotation of all intermediate steps. This provides an efficient and scalable approach for unambiguous evaluation of video reasoning, because evaluating every intermediate step requires substantial annotation effort, incurs prohibitive computational cost, and often yields noisy or ambiguous supervisory signals ( 6 ; 18 ; 33 ) . While last-frame evaluation offers an efficient way to assess reasoning within the CoF paradigm, vision-language models (VLMs) are not always reliable as sole automatic judges. In particular, VLMs may face difficulty interpreting visually dense or grid-structured layouts that require precise recognition of small cells, thin boundaries, or subtle geometric relationships 20 ; 29 ; 18 . To complement these limitations and ensure stable evaluation across diverse task types, V-ReasonBench adopts a hybrid strategy: mask-based evaluation for tasks with well-defined object regions, grid-based evaluation for tasks requiring fine-grained structural accuracy, and lightweight VLM-based evaluation only for visually simple outputs where consistency can be maintained. Each method produces a numerical score that is converted into a pass or fail decision using task-specific thresholds, enabling scalable and reproducible pass@k evaluation while preserving strong alignment with human judgment. To support consistent evaluation under the Chain-of-Frame framework and our last-frame scoring strategy, V-ReasonBench is partitioned into four complementary reasoning classes, each targeting a specific dimension of reasoning: 1) Structured problem-solving encompasses tasks involving numerical manipulation, strategic planning in dynamic game states, and procedural logic derived from visualized program traces. 2) Spatial cognition evaluates understanding of spatial relations, geometric transformations, and symmetry-based patterns. 3) Pattern-based inference probes sequence completion, analogical mapping, and abstract rule induction beyond surface-level visual cues. 4) Physical dynamics examines comprehension of motion, force interactions, temperature effects, and pressure-driven behavior. Together, these four classes provide a comprehensive assessment of cognitive capabilities in video understanding and reasoning. Built upon these methodological principles, V-ReasonBench provides a unified suite for evaluating reasoning in generative video models under the CoF framework. Each instance consists of an initial image, a task instruction, and a target final image, with both procedurally generated and curated scenarios enabling fine control over difficulty and perceptual variation. By applying deterministic, last-frame scoring across four complementary reasoning classes, the benchmark offers consistent measurements of a model’s ability to interpret structured inputs, follow task-specific rules, and produce coherent final outcomes. Evaluating six cutting-edge video generation models (Sora-2 ( 27 ) , Veo-3.1 ( 14 ) , Kling-2.5-Turbo-Pro ( 19 ) , Seedance-1.0-Lite ( 11 ) , Vidu-Q2 ( 32 ) , and Hailuo-02 ( 26 ) ) reveals clear strengths and limitations across multiple reasoning dimensions in Fig. 1 . Our analysis further examines each model’s characteristic reasoning patterns, the influence of video duration on CoF performance, and the contrast between video and image models, highlighting several recurring hallucination behaviors unique to video generation. As an early exploratory effort, V-ReasonBench provides an efficient and reproducible reasoning-centric framework that offers a unified foundation for benchmarking video reasoning and guiding future models toward more reliable, human-aligned visual reasoning. 2 Related Works Video Generation. Recent advances in video generation have been strongly driven by diffusion–transformer models, which provide scalable architectures for producing high-quality visual content ( 3 ; 25 ; 10 ; 46 ) . Commercial systems such as OpenAI’s Sora-2 ( 27 ) , Minimax’s Hailuo-02 ( 26 ) , Runway’s Gen-3 ( 31 ) , and Google DeepMind’s Veo-3.1 ( 14 ) achieve impressive visual quality and strong alignment with text prompts, but their training data, model designs, and evaluation procedures remain undisclosed. Alongside these developments, recent research efforts, including CogVideoX ( 42 ) , HunyuanVideo ( 35 ) , and the Wan series ( 36 ) , have focused on enhancing temporal consistency, motion fidelity, and semantic coherence, reflecting a broader trend toward more controllable and robust video generation systems. Video Reasoning. Recent video generation models have shown emergent visual reasoning abilities that go beyond their original training goals ( 41 ) . The Chain-of-Frame (CoF) paradigm ( 41 ) , analogous to chain-of-thought reasoning in language models ( 40 ) , frames video generation as a stepwise spatial and temporal reasoning process. This idea has been explored through approaches such as VChain ( 17 ) , which uses multimodal models to produce keyframes as intermediate reasoning signals, and visual chain-of-thought methods ( 4 ; 45 ) , which combine visual perception with iterative reasoning for knowledge-intensive tasks and control. However, despite growing interest in evaluating these emergent capabilities, there remains a lack of a scalable and comprehensive benchmark that systematically measures the breadth and depth of video reasoning across diverse task settings. Figure 2 : Overview of V-ReasonBench pipeline. The benchmark covers four

Read the complete text