M 3 -Med \text{M}^{3}\text{-Med} : A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding
Shenxi Liu, Kan Li, Mingyang Zhao, Yuhang Tian, Bin Li, Shoujun Zhou, Hongliang Li, Fuxia Yang · 2025
Read M 3 -Med \text{M}^{3}\text{-Med} : A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding by Shenxi Liu, Kan Li, Mingyang Zhao, Yuhang Tian, Bin Li, Shoujun Zhou, Hongliang Li, Fuxia Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04289.txt.
Introduction
Shenxi Liu and co-authors present a benchmark for testing AI on medical teaching videos across languages, in a paper from 2025. Models that watch and understand video could aid medical training. Existing tests, though, are mostly in English and ask shallow questions whose answers sit on the surface.
Their benchmark pairs expert-checked medical questions with video clips. Its signature task demands multi-hop reasoning: find a key entity in the text, locate matching visual proof in the video, then combine both to answer. Two task forms ask a model to pin the answer's moment in one video or across videos. The set spans several languages, breaking the English-only habit of earlier benchmarks. The authors aim to expose how deep model understanding really goes.
Catalog record
| CREATOR | Shenxi Liu, Kan Li, Mingyang Zhao, Yuhang Tian, Bin Li, Shoujun Zhou, Hongliang Li, Fuxia Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-04289
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
M 3 -Med \text{M}^{3}\text{-Med} : A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding Shenxi Liu, Kan Li, Mingyang Zhao, Yuhang Tian, Bin Li, Shoujun Zhou, Hongliang Li, Fuxia Yang Abstract With the rapid progress of artificial intelligence (AI) in multi-modal understanding, there is increasing potential for video comprehension technologies to support professional domains such as medical education. However, existing benchmarks suffer from two primary limitations: (1) Linguistic Singularity: they are largely confined to English, neglecting the need for multilingual resources; and (2) Shallow Reasoning: their questions are often designed for surface-level information retrieval, failing to properly assess deep multi-modal integration. To address these limitations, we present M 3 -Med, the first benchmark for Multi-lingual, Multi-modal, and Multi-hop reasoning in Medical instructional video understanding. M 3 -Med consists of medical questions paired with corresponding video segments, annotated by a team of medical experts. A key innovation of M 3 -Med is its multi-hop reasoning task, which requires a model to first locate a key entity in the text, then find corresponding visual evidence in the video, and finally synthesize information across both modalities to derive the answer. This design moves beyond simple text matching and poses a substantial challenge to a model’s deep cross-modal understanding capabilities. We define two tasks: Temporal Answer Grounding in Single Video (TAGSV) and Temporal Answer Grounding in Video Corpus (TAGVC). We evaluated several state-of-the-art models and Large Language Models (LLMs) on M 3 -Med. The results reveal a significant performance gap between all models and human challengers, especially on the complex multi-hop questions where model performance drops sharply. M 3 -Med effectively highlights the current limitations of AI models in deep cross-modal reasoning within specialized domains and provides a new direction for future research. \ArticleType RESEARCH PAPER \Year 2025 \Month 1 \Vol \No \DOI \ArtNo \ReceiveDate \ReviseDate \AcceptDate \OnlineDate \AuthorMark Shenxi L [email protected] \AuthorCitation Shenxi Liu, Kan Li, Mingyang Zhao, Yuhang Tian, Bin Li, Shoujun Zhou, Hongliang Li, Fuxia Yang \text{M}^{3}\text{-Med} : A Benchmark for Multi-modal, Multi-lingual, and Multi-hop Reasoning in Medical Instructional Video Understanding keywords Artificial intelligence, Medical instructional video understanding, Multi-lingual, Multi-modal, Multi-Hop Reasoning, Knowledge Graph (KG), Benchmark 1 Introduction AI technology is profoundly reshaping our interaction with digital information, and a central challenge in this field is enabling machines to achieve a human-level understanding of complex video content [ 1 , 2 , 3 ] . In high-value domains such as medical education, clinical training, and patient communication, the intelligent analysis of video materials offers immense potential [ 4 , 5 , 6 ] . This technology can help medical students quickly locate key knowledge points, assist doctors in case reviews, and even intuitively explain complex medical procedures to patients [ 7 ] . Video Question Answering (VideoQA) and Temporal Video Grounding have become standard tasks for evaluating a model’s video comprehension [ 8 , 9 ] . In the open domain, benchmarks like ActivityNet [ 10 ] and QVHighlights [ 11 ] have driven significant progress. Subsequently, research has expanded into specialized domains, with datasets such as YouCook2 [ 12 , 13 ] for cooking and MovieNet [ 14 ] for films, demonstrating the value of domain-specific video understanding. Following this trend, several benchmarks have been introduced for the medical domain, including NurVid [ 15 ] , MedVid [ 16 ] and HealthVid [ 17 ] . However, despite these advances, existing benchmarks, particularly in medicine, suffer from critical limitations. These benchmarks suffer from two primary limitations. First, they are predominantly monolingual (English), failing to address the global need for multilingual medical information access. Second, they are susceptible to ’shortcut learning,’ where questions can often be answered by relying solely on subtitles through simple keyword matching, without requiring genuine comprehension of the visual content (see Fig. 1a ). This reliance on shortcuts prevents a genuine evaluation of a model’s ability to deeply integrate and reason over information from different modalities [ 18 , 19 , 20 , 21 , 22 ] . (a) (b) Figure 1: Performance disparity of models on simple v.s. complex Questions. This chart illustrates the performance gap of challenger models, highlighting their proficiency on straightforward, single-fact questions compared to their significantly reduced accuracy on complex, multi-hop reasoning tasks. To address these challenges, we propose M 3 -Med 1 1 1 The benchmark is publicly available at https://Med-M3-Dataset.github.io/ . , the first benchmark designed for M ulti-lingual, M ulti-modal, and M ulti-hop reasoning on Med ical instructional videos. Our dataset is constructed from instructional videos sourced from public platforms and annotated by a team of medical experts. A key innovation of M 3 -Med is its two-tier question design. “Simple questions” can be answered via direct information retrieval, similar to traditional benchmarks. In contrast, “complex questions” necessitate multi-hop reasoning: they require the model to first identify a key concept in the text, then locate corresponding visual evidence in the video, and finally synthesize these cross-modal cues to derive the correct answer. We benchmarked a range of state-of-the-art models, including Large Language Models (LLMs) and Multi-modal LLMs (MLLMs), against human challengers performance. Our experiments show that while human challengers achieve near-perfect accuracy, all models perform significantly worse, with their performance dropping sharply on complex, multi-hop questions. These results demonstrate that M 3 -Med effectively exposes the limitations of current models in deep, cross-modal reasoning. Our main contributions are threefold: (1) We construct and release M 3 -Med, a new high-quality dataset for evaluating multi-hop, cross-modal reasoning in the medical domain. (2) We propose a novel two-tier question design paradigm that moves beyond shallow text matching to assess a model’s true reasoning depth. (3) We provide extensive benchmarks with state-of-the-art models, revealing their current capabilities and outlining directions for future research. This work is a continuation of the NLPCC 2023 Shared Task 5: Chinese Medical Instructional Video Question Answering 2 2 2 Visit https://github.com/cmivqa/NLPCC-2023-Shared-Task-5 for details. [ 23 ] and the NLPCC 2024 Shared Task 7: Multilingual Medical Instructional Video Question Answering Challenge [ 24 ] 3 3 3 https://github.com/Lireanstar/NLPCC2024_MMIVQA . Furthermore, the dataset from our work served as the basis for the NLPCC 2025 Shared Task 4: Multi-modal, Multi-lingual, and Multi-hop Medical Instructional Video Question Answering Challenge [ 25 ] 4 4 4 https://github.com/cmivqa/NLPCC2025_M4IVQA . 2 Related Work The task of localizing specific video segments based on natural language queries, known as Video Temporal Grounding (VTG), has become a key area of research. Driven by the proliferation of online video and advances in Visual Language Models (VLMs), VTG has seen growing demand in applied domains like education and medicine, leading to the proposal of numerous benchmarks and methods. 2.1 Benchmarks for Medical Video Understanding Several benchmarks have been established for medical video understanding. For instance, NurVid [ 15 ] provides a dataset focused on nursing education, with tasks centered on classifying procedures and detecting specific nursing actions within videos. Similarly, the work by [ 16 ] introduced two datasets: MedVidCL, designed for classifying videos into medical-instructional, medical-non-instructional, and non-medical categories; and MedVidQA, which focuses on the temporal localization of answers to natural language questions in medical videos. Building on MedVidQA, [ 17 ] proposed a pipeline to automatically generate larger-scale datasets, resulting in HealthVidQA-CRF