HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding
Mengqi Shi Haopeng Zhang · 2026
Read HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding by Mengqi Shi Haopeng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19223.txt.
Introduction
Mengqi Shi and Haopeng Zhang introduce HAVEN, a benchmark for unified video understanding built on layered alignment. Past summarization tests split supervision across keyframes, key shots, or disjoint text notes, missing the layered structure of cross-modal alignment. HAVEN pairs video and text at frame, shot, and whole-video levels, with continuous links between modes. Its test suite covers summarization, temporal reasoning, grounding across modes, and saliency ranking. Tests of leading models expose a gap between fluent text and grounded understanding, and the dataset, suite, and protocols are public.
Catalog record
| CREATOR | Mengqi Shi Haopeng Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44794 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19223
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding Mengqi Shi Haopeng Zhang Abstract While Multimodal Large Language Models (MLLMs) exhibit strong performance on standard video tasks, their ability to faithfully summarize and reason over complex narratives remains poorly evaluated. Existing summarization benchmarks fragment supervision across isolated granularities, such as keyframes, key shots, or disjointed text summaries, failing to capture the inherently hierarchical structure of cross-modal alignment. To address this critical gap, we introduce HAVEN, a hierarchically aligned multimodal benchmark for unified video understanding. HAVEN pioneers a fully granular (frame, shot, and video levels) and fully multimodal (video and text) dataset architecture, complete with explicit, continuous alignment between modalities. Built upon this unified annotation paradigm, we propose a comprehensive evaluation suite spanning summarization, temporal reasoning, multimodal grounding, and saliency ranking. Extensive benchmarking of state-of-the-art MLLMs exposes a persistent gap between surface-level textual fluency and grounded multimodal understanding. Ultimately, HAVEN advances the evaluation of multimodal systems beyond traditional QA formats, offering a rigorous, standardized testbed to drive future research in interpretable, hierarchical video understanding. We publicly release the dataset, benchmark suite, and evaluation protocols at https://github.com/alohalab-ai/align_vsum . 1 Introduction Multimodal large language models (MLLMs) have rapidly advanced video understanding, achieving strong performance on tasks such as summarization ( 19 ; 47 ; 10 ) , captioning ( 39 ; 34 ; 1 ) , and question answering ( 15 ; 7 ; 23 ) . However, these gains raise a fundamental question: do MLLMs genuinely understand video content, or do they produce plausible outputs by exploiting language priors? Resolving this ambiguity is challenging due to the lack of evaluation frameworks capable of rigorously assessing fine-grained, temporal alignment between visual evidence and generated language. Current video benchmarks suffer from two systemic limitations ( 15 ; 7 ; 2 ; 23 ; 22 ) . First, they provide fragmented supervision. Most datasets operate at a single temporal granularity (e.g., keyframes, clips, or video-level summaries), ignoring the inherently hierarchical structure of real-world videos, where frames build shots, and shots construct coherent narratives. Second, they lack precise cross-modal grounding. Text annotations are typically weakly aligned with visual content, enabling models to succeed without accurately localizing or reasoning over relevant evidence 41 . Together, these limitations create an evaluation blind spot: models can achieve high scores through fluent generation while exhibiting poor multimodal alignment. As a result, current evaluation protocols systematically overestimate the capabilities of modern MLLMs, particularly on tasks that do not require temporal compositionality or saliency-aware reasoning ( 23 ; 15 ; 7 ; 16 ; 32 ) . To address these limitations, we introduce HAVEN (Hierarchically Aligned multimodal benchmark for unified Video undErstandiNg), a comprehensive dataset and evaluation framework designed to systematically measure hierarchical cross-modal alignment . HAVEN is built upon two core principles: (1) Hierarchical completeness , providing unified annotations at the frame, shot, and video levels to enable evaluation across multiple temporal scales; and (2) Cross-modal continuity : dense, explicit alignment between visual content and textual descriptions, ensuring that language must be grounded in temporally localized evidence. Pioneering a fully granular and fully multimodal dataset architecture, HAVEN supports a suite of tasks spanning summarization, temporal reasoning, multimodal grounding, and saliency ranking, each targeting a distinct aspect of video understanding while sharing a common annotation backbone. Crucially, this enables controlled and fine-grained analysis of model behavior across levels of abstraction and modalities. We benchmark a range of state-of-the-art MLLMs on HAVEN (Fig. 1 ) and reveal a critical "illusion of capability." While models consistently produce fluent textual summaries, they frequently fail to ground their outputs in the correct visual evidence. Moreover, we observe that joint multimodal generation does not reliably improve alignment, highlighting the fragility of current approaches. These results expose a significant gap between apparent and actual understanding. These findings highlight the significant risk of overestimating MLLM capabilities when evaluations are limited to coarse or weakly grounded tasks. Our main contributions are three-fold: (1) We introduce HAVEN, a novel dataset that pioneers a fully granular (frame-, shot-, and video-level) and fully multimodal framework with explicit, continuous cross-modal alignment. (2) We establish a rigorous testing framework featuring distinct tasks across four core dimensions (summarization, multimodal grounding, temporal reasoning, and saliency ranking), enabling a multi-faceted assessment of model capabilities. (3) We provide an extensive empirical analysis that identifies key limitations in state-of-the-art MLLMs, exposing a severe discrepancy between surface-level textual fluency and true multimodal grounding. Figure 1 : Comparison between different MLLMs on HAVEN across different capabilities. We include text-proxy LLMs as baselines. 2 Related Work Video Understanding Benchmarks. Recent benchmarks have advanced the evaluation of MLLMs on video understanding, covering tasks such as event understanding, temporal reasoning, question answering, and multi-shot comprehension ( 15 ; 7 ; 23 ; 10 ; 1 ; 13 ) . Recent works have introduced benchmarks designed for long video understanding, such as Hourvideo ( 2 ) and other Q&A-based benchmarks ( 37 ; 38 ) . These benchmarks extend evaluation to more realistic long-video scenarios, but are typically organized around isolated task formats and focus on general understanding. As a result, they provide limited support for jointly evaluating model capabilities of diverse dimensions within a unified framework. Compared to existing benchmarks, we enhance the dataset with richer multimodal signals and fine-grained cross-modal alignment, which enables a more diverse set of tasks beyond traditional Q&A or coarse-grained formulations. This provides a new perspective for evaluating video understanding. Video Summarization Benchmarks. Video summarization research has been supported by datasets such as SumMe ( 9 ) , TVSum ( 40 ) , and subsequent resources ( 31 ; 19 ) , enabling studies on key segment selection, importance estimation, and summary generation ( 8 ; 44 ; 29 ) . Other large-scale video-text datasets ( 35 ; 3 ; 12 ; 43 ) support multiple tasks on video understanding evaluations. More recent work has explored multimodal understanding and evaluation, including cross-modal integration, factual consistency, and evaluation reliability ( 47 ; 21 ; 25 ; 33 ; 17 ) . In addition, prior studies highlight the importance of alignment and structured representations for understanding long or segmented content ( 6 ; 4 ) . However, existing datasets and evaluation protocols are typically designed for single granularity, and lack hierarchical annotations and explicit cross-modal alignments, making it difficult to evaluate whether models can connect textual outputs with their supporting visual evidence across tasks. 3 HAVEN Benchmark Table 1 : Source datasets used to construct HAVEN Benchmark. . “Alignment“ denotes aligned annotations between visual and texual data, “ K_{f} ” = key frames, “ D_{f} ” = frame descriptions, “ K_{s} ” = key shots, and “ D_{s} ” = shot descriptions. “Avg Frames” and “Avg Shots” denote the average sampled frames and shots per video. Dataset #Videos Modalities Alignment K_{f} D_{f} K_{s} D_{s} Avg Frames Avg Shots HAVEN 11055 V+T \rightarrow V+T \checkmark \checkmark \checkmark \checkmark \checkmark 164.9 16.2 \hookrightarrow SumMe ( 9 ) 25 V \rightarrow V \times \checkmark \times \checkmark \times 293.4 29.8 \hookrightarrow TVSum ( 40 ) 50 V \rightarrow V \times \checkmark \times \checkmark \times 470.2 47.5 \hookrightarrow OVP ( 5 ) 50 V \rightarrow V \times \checkmark \times \times \times 103.0 9.9 \hookrightarrow YouTube ( 5