MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
Arushi Goel, Sreyan Ghosh, Vatsal Agarwal, Nishit Anand, Kaousheik Jayakumar, Lasha Koroshinadze, Yao Xu, Katie Lyons, James Case, Karan Sapra, Kevin J. Shih, Siddharth Gururani, Abhinav Shrivastava, Ramani Duraiswami, Dinesh Manocha, Andrew Tao, Bryan Catanzaro, Mohammad Shoeybi, Wei Ping · 2026
Read MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos by Arushi Goel, Sreyan Ghosh, Vatsal Agarwal, Nishit Anand, Kaousheik Jayakumar, Lasha Koroshinadze, Yao Xu, Katie Lyons, James Case, Karan Sapra, Kevin J. Shih, Siddharth Gururani, Abhinav Shrivastava, Ramani Duraiswami, Dinesh Manocha, Andrew Tao, Bryan Catanzaro, Mohammad Shoeybi, Wei Ping (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14145.txt.
Introduction
Goel and co-authors introduce MMOU, a benchmark for joint sight, sound, and text reasoning in long real-world videos. It holds 20,000 hand-checked questions over 11,877 web videos across many domains. Thirteen core skills are tested, all needing facts merged across modes and time. More than twenty top multimodal models are evaluated. The best closed model scores only about 64 percent, and the best open one about 47 percent, which shows how far such systems still have to go.
Catalog record
| CREATOR | Arushi Goel, Sreyan Ghosh, Vatsal Agarwal, Nishit Anand, Kaousheik Jayakumar, Lasha Koroshinadze, Yao Xu, Katie Lyons, James Case, Karan Sapra, Kevin J. Shih, Siddharth Gururani, Abhinav Shrivastava, Ramani Duraiswami, Dinesh Manocha, Andrew Tao, Bryan Catanzaro, Mohammad Shoeybi, Wei Ping |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59055 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14145
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos Arushi Goel, Sreyan Ghosh, Vatsal Agarwal, Nishit Anand, Kaousheik Jayakumar, Lasha Koroshinadze, Yao Xu, Katie Lyons, James Case, Karan Sapra, Kevin J. Shih, Siddharth Gururani, Abhinav Shrivastava, Ramani Duraiswami, Dinesh Manocha, Andrew Tao, Bryan Catanzaro, Mohammad Shoeybi, Wei Ping Abstract Multimodal Large Language Models (MLLMs) have shown strong performance in visual and audio understanding when evaluated in isolation. However, their ability to jointly reason over omni-modal (visual, audio, and textual) signals in long and complex videos remains largely unexplored. We introduce MMOU , a new benchmark designed to systematically evaluate multimodal understanding and reasoning under these challenging, real-world conditions. MMOU consists of 15,000 carefully curated questions paired with 9038 web-collected videos of varying length, spanning diverse domains and exhibiting rich, tightly coupled audio-visual content. The benchmark covers 13 fundamental skill categories, all of which require integrating evidence across modalities and time. All questions are manually annotated across multiple turns by professional annotators, ensuring high quality and reasoning fidelity. We evaluate 20+ state-of-the-art open-source and proprietary multimodal models on MMOU. The results expose substantial performance gaps: the best closed-source model achieves only 64.2% accuracy, while the strongest open-source model reaches just 46.8%. Our results highlight the challenges of long-form omni-modal understanding, revealing that current models frequently fail to apply even fundamental skills in long videos. Through detailed analysis, we further identify systematic failure modes and provide insights into where and why current models break. † † affiliationnotice: \icmlEqualLead Equal contribution \icmlEqualSignificant Project: https://huggingface.co/datasets/nvidia/MMOU 1 Introduction The pursuit of Artificial General Intelligence (AGI) has driven rapid progress in Large Language Models (LLMs), particularly through the emergence of Multimodal Large Language Models (MLLMs) that process information across multiple modalities such as text, images, audio, and video ( 62 ; 59 ; 27 ; 13 ; 7 ) . These models have enabled compelling applications, allowing LLMs to see through vision ( 14 ; 39 ; 37 ) and listen through audio ( 20 ; 18 ; 12 ; 54 ; 56 ) . Recent MLLMs demonstrate strong capabilities across audio tasks (e.g., automatic speech recognition, sound classification, and audio captioning) and visual tasks (e.g., OCR, visual question answering, and video grounding), often surpassing prior benchmarks by a large margin. Figure 1 : Overview of MMOU , a benchmark for evaluating omni-modal understanding in long, complex real-world videos, showing that both open and closed multimodal models struggle even with basic understanding. Figure 2 : Illustrative examples from MMOU , demonstrating the different skill types evaluated in the benchmark. Despite this progress, existing MLLMs exhibit notable limitations. Most models are optimized for single-modality reasoning ( 3 ; 20 ) , such as vision-only or audio-only understanding, and often fail to jointly perceive and reason across modalities in a manner analogous to human cognition. This limitation is partly due to the imbalance in available training data and benchmarks: single-modality datasets are more abundant, higher quality, and cover a wider range of tasks ( 36 ; 27 ; 22 ) than their multi-modal counterparts. As a result, current models rarely learn to integrate audio and visual cues in a unified and consistent manner. Benchmarking has long played a central role in advancing AI by providing structured, diagnostic evaluation frameworks ( 23 ; 49 ; 28 ; 17 ; 25 ) . While evaluation of LLMs has matured substantially, covering domains such as mathematics, code generation, reasoning, and instruction following, holistic evaluation of MLLMs remains underdeveloped. Although numerous image and video benchmarks have emerged in recent years, benchmarks that rigorously evaluate audio-visual reasoning are scarce. In particular, most video benchmarks either ignore audio entirely or treat it as auxiliary, and predominantly focus on short clips that fail to capture long-term temporal dependencies ( 33 ) . Consequently, existing evaluations do not adequately reflect the challenges posed by long and complex real-world videos, where meaningful understanding requires tightly coupled reasoning over audio and visual streams across extended time horizons. Figure 3 : Distribution of MMOU. (a) Video category distribution in MMOU, covering 10 major domains and 36 fine-grained subdomains. (b) Co-occurrence matrix of QA task types, illustrating how multiple reasoning skills are jointly required within individual questions. (c) Distribution of the relative temporal positions (average of start–end time-stamps) of answer evidence within videos, showing that answers are spread across the entire video timeline. (d) Distribution of QA instances across the 13 skill/task types in MMOU. (e) Video duration distribution, highlighting the prevalence of long and complex real-world videos. Main Contributions. We present MMOU, a M assive, M ulti-task O mni-modal U understanding and Reasoning. Our benchmark is designed to evaluate joint audio-visual understanding and reasoning on long and complex real-world videos under realistic conditions (see Fig. 1 ). Specifically, (i) each question requires simultaneous integration of audio and visual information, such that removing either modality leads to failure; (ii) the questions require models to demonstrate proficiency in 13 distinct and fundamental skills; (iii) the benchmark is large-scale, comprising 15,000 multiple-choice QA pairs sourced from 9038 long-form real-world videos spanning 10 domains and 36 fine-grained subcategories, with each video exhibiting strong temporal and semantic alignment between audio and visual streams; and (iv) all questions are annotated by a group 11 professionally trained human experts and each is optionally paired with 10 carefully constructed answer options that include hard distractors. To summarize, our main contributions are: • We introduce MMOU , a comprehensive benchmark for evaluating advanced omni-modal (audio-visual) perception and reasoning in MLLMs on long and complex real-world videos . MMOU spans 13 skill categories and includes 15,000 expertly annotated multiple-choice questions, covering both breadth and depth in multimodal understanding. • We evaluate 20+ open-source and proprietary MLLMs on MMOU and show that even the most advanced models struggle with tasks that humans find intuitive. The best closed-source model achieves only 64.2% accuracy, with open-source models performing substantially worse (46.8%), revealing significant gaps in current multimodal reasoning capabilities. • We conduct an in-depth analysis of model predictions, uncovering systematic failure modes. 2 Related Work Multimodal Large Language Models. Recent years have seen rapid progress in multimodal large language models (MLLMs), which extend the capabilities of text-only LLMs ( 27 ; 42 ; 60 ) to visual, audio, and audio–visual inputs ( 59 ; 20 ; 14 ; 5 ; 10 ; 58 ) . These models typically integrate modality-specific encoders ( 57 ; 46 ; 19 ; 47 ) with a shared language model backbone ( 11 ; 42 ; 27 ) , and are trained using large-scale multimodal instruction-tuning data ( 29 ; 65 ; 20 ; 59 ) . As a result, state-of-the-art models demonstrate strong performance on a wide range of established benchmarks, including image–text, video–text, and audio–text understanding tasks ( 16 ; 48 ; 64 ) . Despite these advances, existing evaluation protocols remain largely unimodal, with most benchmarks isolating a single modality or task. Such narrowly defined settings fail to capture the complexity of real-world multimodal reasoning. Consequently, strong results on individual benchmarks do not necessarily translate to robust omni-modal understanding, which requires joint reasoning across modalities, tasks, and temporal context ( 32 ) . A comprehensive benchmark is therefore