How Far Are Video Models from True Multimodal Reasoning?

Xiaotian Zhang *,§, Jianhui Wei *,§, Yuan Wang *, Jie Tan, Yichen Li, Yan Zhang ‡, Ziyi Chen, Daoan Zhang, Dezhi Yu, Wei Xu, Songtao Jiang, Zuozhu Liu † · 2026

Read How Far Are Video Models from True Multimodal Reasoning? by Xiaotian Zhang *,§, Jianhui Wei *,§, Yuan Wang *, Jie Tan, Yichen Li, Yan Zhang ‡, Ziyi Chen, Daoan Zhang, Dezhi Yu, Wei Xu, Songtao Jiang, Zuozhu Liu † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19193.txt.

Introduction

Zhang and eleven colleagues ask how far video models are from true multimodal thought, introducing CLVG-Bench for zero-shot thought via Context Learning in Video Generation. The benchmark holds over 1,000 metadata in 6 categories and 47 subcategories. An Adaptive Video Evaluator matches expert judgment using few annotations and gives clear feedback. Top models like Seedance 2.0 handle some subtasks but fall short on grounded and interactive generation, with success under 25 percent and about 0 percent, showing multimodal thought and physical grounding as limits.

Catalog record

CREATORXiaotian Zhang *,§, Jianhui Wei *,§, Yuan Wang *, Jie Tan, Yichen Li, Yan Zhang ‡, Ziyi Chen, Daoan Zhang, Dezhi Yu, Wei Xu, Songtao Jiang, Zuozhu Liu †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38944
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19193

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Far Are Video Models from True Multimodal Reasoning? Xiaotian Zhang *,§, Jianhui Wei *,§, Yuan Wang *, Jie Tan, Yichen Li, Yan Zhang ‡, Ziyi Chen, Daoan Zhang, Dezhi Yu, Wei Xu, Songtao Jiang, Zuozhu Liu † Abstract Despite remarkable progress toward general-purpose video models, a critical question remains unanswered: how far are these models from achieving true multimodal reasoning? Existing benchmarks fail to address this question rigorously, as they remain constrained by straightforward task designs and fragmented evaluation metrics that neglect complex multimodal reasoning. To bridge this gap, we introduce CLVG-Bench, an evaluation framework designed to probe video models’ zero-shot reasoning capabilities via C ontext L earning in V ideo G eneration. CLVG-Bench comprises more than 1,000 high-quality, manually annotated metadata across 6 categories and 47 subcategories, covering complex scenarios including physical simulation, logical reasoning, and interactive contexts. To enable rigorous and scalable assessment, we further propose an A daptive V ideo E valuator (AVE) that aligns with human expert perception using minimal annotations, delivering interpretable textual feedback across diverse video context tasks. Extensive experiments reveal a striking answer to our central question: while state-of-the-art (SOTA) video models, such as Seedance 2.0, demonstrate competence on certain understanding and reasoning subtasks, they fall substantially short with logically grounded and interactive generation tasks (achieving success rates < 25 % <25\% and ∼ 0 % \sim 0\% , respectively), exposing multimodal reasoning and physical grounding as critical bottlenecks. By systematically quantifying these limitations, the proposed method provides actionable feedbacks and a clear roadmap toward truly robust, general-purpose video models. CLVG-Bench and code are released here . 1 Introduction Driven by large-scale training on web-scale data with generative objectives [ 6 , 74 , 31 ] , video models have demonstrated groundbreaking zero-shot capabilities, evolving from mere instruction following to complex understanding and reasoning [ 76 , 57 , 62 , 71 ] . Specifically, traditional reference-based video generation has transitioned from simple, single-reference conditioning [ 35 , 51 , 65 , 37 , 39 , 86 , 32 ] to accommodating multi-subject and multimodal references [ 26 , 72 ] . Concurrently, the primary focus of these tasks is shifting from prioritizing sheer visual fidelity to emphasizing broader reasoning capabilities. Despite this immense potential and the surging interest in video reasoning, the community still lacks a systematic formulation and comprehensive benchmarking for these emerging paradigms. To address this, we formalize the task of reasoning-driven video generation, conditioned on arbitrary combinations of modalities (text, image, audio, and video), as Context Learning in Video Generation . Figure 1 : Overview of CLVG-Bench . Our framework evaluates video models via Context Learning in Video Generation across arbitrary modalities (Text, Image, Audio, Video). It systematically probes reasoning capabilities across 6 categories and 47 subcategories, which provides a roadmap for general-purpose video models. Second, video quality evaluation has transitioned from fragmented scoring metrics [ 28 ] toward fine-grained and unified textual interpretations. Previous evaluation paradigms typically rely on task-specific metric combinations [ 77 , 12 , 93 , 27 , 45 , 46 , 30 , 58 ] or reward models trained on multi-dimensional preference datasets [ 23 , 81 , 22 , 4 , 3 ] . However, fragmented and coarse metrics often fail to provide actionable textual feedback, while reward model training is frequently bottlenecked by the scarcity of high-quality data and remains susceptible to reward hacking [ 13 ] . To circumvent these issues, recent agent-based frameworks [ 83 , 20 , 75 ] have been adapted for unified video evaluation, offering both quantitative scores and qualitative feedback. Nonetheless, their reliance on hand-crafted system prompts and static in-context examples severely constrains their generalization and reproducibility. They primarily focus on visual fidelity while overlooking intrinsic logical reasoning within videos. Consequently, efficiently aligning these agents to new video evaluation tasks with minimal expert supervision remains a critical yet underexplored problem. We introduce CLVG-Bench , designed to comprehensively evaluate video models’ ability to perform zero-shot reasoning on multimodal contexts. We systematically define 6 fundamental categories and 47 subcategories, and have carefully curated more than 1,000 high-quality metadata entries with video reasoning-oriented contexts, including text instruction, multi-shot and multi-subject videos, reference images and audio. To prevent the copyright issues and data contamination in evaluation, all metadata ( e.g . videos, images) are manually curated. We further propose an A daptive V ideo E valuator ( AVE ), leveraging an Automatic Prompt Optimization (APO) framework equipped with a novel semantic matching function for open-ended text evaluation. This mechanism empowers the evaluation agent to adapt to specific tasks using minimal expert annotations while simultaneously generating detailed textual feedback. Consequently, AVE serves as a interpretable metric for assessing the reasoning capabilities of video models, supporting scalable evaluation across diverse models and tasks, and facilitating fine-grained error attribution. Experimental results reveal significant performance variations among current video models across different reasoning tasks. In particular, these models exhibit considerable shortcomings in tasks related to physical reasoning, logical inference, and interactive generation. This highlights a critical gap in their capability to simulate and reason about real-world dynamics. Further analysis reveals that explicit context understanding provided by vision-language models aids video models in generating physically plausible content. This indicates that current video models lack the intrinsic ability to autonomously infer physical causality within instructions. Moving forward, future architectures should integrate comprehension and generation components more seamlessly to achieve robust, reliable reasoning capabilities in complex scenarios. Our major contributions are threefold: • We introduce CLVG-Bench, a novel evaluation framework that abstracts video generation tasks into the definition of context learning video generation, and systematically evaluates the capability of current video models in simulating and reasoning about real-world dynamics. • We propose the Adaptive Video Evaluator, a flexible evaluation framework designed for open-ended generation tasks. This evaluator dynamically adjusts based on a minimal set of human annotations, offering a versatile approach to assess tasks with varying contexts. • Our study uncovers the limitations of current video models in multimodal reasoning. We advocate for a tighter integration of understanding and generation to enhance model performance in complex video generation scenarios. 2 Related Works 2.1 Evolution of Video Models Video models have evolved from early diffusion frameworks [ 66 , 79 , 65 ] to advanced Diffusion Transformer (DiT) architectures [ 62 , 64 , 16 , 38 , 50 , 94 ] , achieving superior cinematic quality, physical simulation, and efficiency [ 19 ] . To bridge understanding and generation, unified models [ 61 , 70 ] now integrate cross-modal decoding within single architectures. Recent works advance this bidirectional paradigm through joint audio-visual DiTs [ 56 ] and integrating LLMs with 3D visual tokenizers [ 10 , 42 , 72 ] , driving holistic multimodal intelligence. 2.2 Video Benchmark Video benchmarks have shifted from discriminative understanding to generative reasoning. Early efforts [ 82 , 7 , 34 ] focused on temporal and diagnostic reasoning, a trajectory recently extended by VBVR [ 68 ] to probe complex logical chains. However, these web-scraped datasets face copyright and contamination risks while prioritizing perception over synthesis. Consequently, generation and editing benchmarks [ 29 , 55 , 60 , 21 ] established aesthetic and instruction-following frameworks. Although recent works [ 87 ,

Read the complete text