ViMU: Benchmarking Video Metaphorical Understanding
Qi Li, Xinchao Wang, [email protected] · 2026
Read ViMU: Benchmarking Video Metaphorical Understanding by Qi Li, Xinchao Wang, [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14607.txt.
Introduction
Li and Wang argue that most video models grasp literal content but miss subtext such as metaphor, irony and social meaning. Their ViMU benchmark is the first designed to test whether frontier video models can read the implicit ideas and intentions creators convey through context, style and social experience. It probes understanding beyond objects, actions and temporal relations, where meanings can also shift across cultures and social groups.
Catalog record
| CREATOR | Qi Li, Xinchao Wang, [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61893 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14607
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ViMU: Benchmarking Video Metaphorical Understanding Qi Li, Xinchao Wang, [email protected] Abstract Any new medium, once it emerges, is used for more than the transmission of overt content alone. The information it carries typically operates on two levels: one is the content directly presented, while the other is the subtext beneath it—the implicit ideas and intentions the creator seeks to convey through the medium. Likewise, since video technologies became widely adopted, video has served not only as a powerful tool for recording and communicating visual information, but also as a vehicle for emotions, attitudes, and social meanings that are often difficult to articulate explicitly. Thus, the true meaning of many videos does not reside solely in what is shown on screen; it is often embedded in context, style of expression, and the viewer’s social experience. Some forms of such video subtext are humorous, while others carry irony, mockery, or criticism. These implicit meanings can also be interpreted very differently across cultural backgrounds and social groups. However, most existing video understanding models still focus primarily on literal visual comprehension, such as recognizing objects, actions, or temporal relations, and lack a systematic ability to understand the metaphorical, ironic, and social meanings embedded in videos. To bridge this gap, we introduce ViMU ( Vi deo M etaphorical U nderstanding), the first benchmark designed to systematically evaluate the subtext understanding capabilities of frontier models in videos. ViMU assesses whether video understanding models can go beyond literal perception to infer implicit meaning, rhetorical devices, social signals, target subjects, and culturally grounded subtext, while grounding their interpretations in multimodal evidence and answering both open-ended and multiple-choice questions. Importantly, all questions are designed to be hint-free, ensuring that no key evidence is disclosed to models before answering. Extensive experiments show that most frontier models, including closed-source ones, achieve below 50% overall performance. We further conduct fine-grained analyses to uncover distinctive model behaviors. Disclaimer: This paper contains potentially offensive and harmful content. Project Page GitHub Dataset “The most important thing in communication is hearing what isn’t said.” - Peter Drucker 1 Introduction Recent advances in large language models have enabled the integration of rich real-world information, including videos, into model representations 1 ; 8 ; 2 ; 34 ; 28 ; 18 ; 17 ; 30 ; 35 . Consequently, video understanding models have become effective for tasks such as visual grounding and causal reasoning 7 ; 37 ; 31 . Yet these forms of understanding remain largely confined to the surface-visible content. Put simply, directly observable content explains how an event unfolds, but not what it ultimately means, as such meaning often lies in the underlying social subtext 1 1 1 As Roland Barthes notes in his book Mythologies , ”myth is a second-order semiological system” 14 ; 21 , in which literal content serves as the basis for a secondary layer of cultural or ideological meaning. : the deeper layer that maps an event onto broader social meanings, values, and collective attitudes. Together, the visible content and its subtext constitute the full depth of video understanding 14 ; 9 ; 13 . Figure 1 : Examples illustrating the large gap between observable content and underlying subtext in videos. In the top example, the video appears to show a girl dancing on a reality show, while its implied meaning alludes to Nazi symbolism. In the bottom example, the video appears to show a child catching an apple above Newton and a strange flying scene, while its underlying joke is that the apple missing Newton led to a setback in the development of physics. As illustrated in Figure 1 , the gap between observable content and underlying subtext can be substantial. In such a case, understanding the video requires more than recognizing objects, actions, or temporal structure, which are typically emphasized in prior works 7 ; 37 ; 31 ; 32 ; 3 ; 16 . It demands integrating multimodal evidence, recovering culturally situated references, and inferring the creator’s communicative intent beyond what is explicitly shown. Existing evaluations left far behind for such subtext interpretation in videos. Most existing benchmarks fall short in three ways: (i) targeting implicit reasoning over hidden spatial, physical, or interactional relations rather than socially grounded meanings 27 ; 5 ; (ii) focusing only on narrower phenomena such as non-verbal humor 26 ; or (iii) relying on multiple-choice formats whose options may expose plausible subtext hypotheses 11 . These settings do not fully capture genuine hint-free inference over socially grounded video meaning. To fill this gap, we introduce ViMU , a benchmark specifically designed to evaluate whether models can move beyond observable content to recover the underlying subtext of videos. In particular, ViMU requires models to infer implicit meaning in a hint-free manner, without being told in advance which socio-cultural cues are relevant. To achieve this, we build ViMU through a meticulous curation process involving multiple rounds of annotation and filtering by advanced closed-source models and human experts. This procedure is designed not only to ensure task difficulty and a genuinely hint-free evaluation setting, but also to maintain broad coverage of diverse rhetorical mechanisms and social value signals. Finally, we obtain a high-quality dataset of 588 videos with 2,352 questions across four tasks, covering both open-ended and multiple-choice questions. We extensively investigate 16 popular MLLMs with ViMU, which brings in several critical insights. Firstly, video metaphorical understanding remains a technically challenging problem for the existing MLLMs . Even the most advanced closed-source models achieve below 50% average performance across the four tasks. Secondly, many models systematically over-predict generic or safer categories while under-predicting more implicit or socially coded ones , suggesting a shared tendency to favor more accessible interpretations over deeper subtextual inference. Thirdly, we observe a clear mismatch between general video understanding and metaphorical video understanding : models that excel on conventional video understanding task do not necessarily perform best on our tasks. In addition to the overall conclusion, individual tasks enable fine-grained analysis in each specialized aspects. Therefore, we anticipate the benchmark to assist in improving MLLMs’ video metaphorical understanding capabilities by providing insights into their current strengths and weaknesses. 2 Related Work Reasoning beyond explicit visual evidence. Some recent work has moved beyond explicit-evidence-centric VideoQA by requiring models to infer answers from indirect or partially unavailable cues. I-VQA 5 studies settings where explicit visual evidence is missing and answers must be inferred from context, building on related work in visual commonsense and context-based reasoning such as VisualCOMET 23 , Video2Commonsense 6 , and causal video reasoning methods like MECD 3 and MECD+ 4 . VRR-QA 27 further focuses on implicit relational reasoning across frames when key relations are not directly co-visible. While these benchmarks go beyond literal perception, they still focus on inferential VideoQA or inter-frame relation reasoning rather than broader subtext understanding in open online videos. Humor understanding, meme interpretation, and social meaning. A closely related line of work studies higher-level interpretation in humorous or socially contextualized media. v-HUB 26 focuses on multimodal video humor understanding, especially in non-verbal short videos, while AVMeme Exam 11 extends evaluation to contextual and cultural understanding of Internet audio-visual memes. Related audio benchmarks, including Dynamic-SUPERB 10 , AudioBench 29 , MMAU 24 , and MMAR