MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios
Yang Shi Huanqian Wang Wulin Xie Huanyao Zhang Lijie Zhao Yi-Fan Zhang, Xinfeng Li Chaoyou Fu Zhuoer Wen Wenting Liu Zhuoran Zhang Xinlong Chen, Bohan Zeng Sihan Yang Yushuo Guan Zhang Zhang Liang Wang Haoxuan Li, Zhouchen Lin Yuanxing Zhang Pengfei Wan Haotian Wang Wenjing Yang, ⋄ Core Contributor · 2025
Read MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios by Yang Shi Huanqian Wang Wulin Xie Huanyao Zhang Lijie Zhao Yi-Fan Zhang, Xinfeng Li Chaoyou Fu Zhuoer Wen Wenting Liu Zhuoran Zhang Xinlong Chen, Bohan Zeng Sihan Yang Yushuo Guan Zhang Zhang Liang Wang Haoxuan Li, Zhouchen Lin Yuanxing Zhang Pengfei Wan Haotian Wang Wenjing Yang, ⋄ Core Contributor (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21333.txt.
Introduction
Yang Shi and co-authors built a benchmark to test how well multimodal language models read text inside videos. Models read still images well, but video adds motion blur, changing frames, and visual effects. Their benchmark, MME-VideoOCR, spans ten task families and twenty-five tasks across forty-four scenarios, using nearly fifteen hundred videos and two thousand hand-checked question-answer pairs.
Eighteen leading models were tested, and the best, Gemini-2.5 Pro, reached only seventy-three point seven percent accuracy. Models handled text sitting in one or a few frames well. They stumbled when an answer needed the whole video: linking frames over time, resisting language habits, and following spatial change. Sharp images and enough frames proved vital for reliable reading.
Catalog record
| CREATOR | Yang Shi Huanqian Wang Wulin Xie Huanyao Zhang Lijie Zhao Yi-Fan Zhang, Xinfeng Li Chaoyou Fu Zhuoer Wen Wenting Liu Zhuoran Zhang Xinlong Chen, Bohan Zeng Sihan Yang Yushuo Guan Zhang Zhang Liang Wang Haoxuan Li, Zhouchen Lin Yuanxing Zhang Pengfei Wan Haotian Wang Wenjing Yang, ⋄ Core Contributor |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21333
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios Yang Shi Huanqian Wang Wulin Xie Huanyao Zhang Lijie Zhao Yi-Fan Zhang, Xinfeng Li Chaoyou Fu Zhuoer Wen Wenting Liu Zhuoran Zhang Xinlong Chen, Bohan Zeng Sihan Yang Yushuo Guan Zhang Zhang Liang Wang Haoxuan Li, Zhouchen Lin Yuanxing Zhang Pengfei Wan Haotian Wang Wenjing Yang, ⋄ Core Contributor Project Lead ‡ Corresponding Author https://mme-videoocr.github.io/ Abstract Multimodal Large Language Models (MLLMs) have achieved considerable accuracy in Optical Character Recognition (OCR) from static images. However, their efficacy in video OCR is significantly diminished due to factors such as motion blur, temporal variations, and visual effects inherent in video content. To provide clearer guidance for training practical MLLMs, we introduce MME-VideoOCR benchmark, which encompasses a comprehensive range of video OCR application scenarios. MME-VideoOCR features 10 10 task categories comprising 25 25 individual tasks and spans 44 44 diverse scenarios. These tasks extend beyond text recognition to incorporate deeper comprehension and reasoning of textual content within videos. The benchmark consists of 1,464 1,464 videos with varying resolutions, aspect ratios, and durations, along with 2,000 {2,000} meticulously curated, manually annotated question-answer pairs. We evaluate 18 18 state-of-the-art MLLMs on MME-VideoOCR, revealing that even the best-performing model (Gemini-2.5 Pro) achieves only an accuracy of 73.7 % 73.7\% . Fine-grained analysis indicates that while existing MLLMs demonstrate strong performance on tasks where relevant texts are contained within a single or few frames, they exhibit limited capability in effectively handling tasks that demand holistic video comprehension. These limitations are especially evident in scenarios that require spatio-temporal reasoning, cross-frame information integration, or resistance to language prior bias. Our findings also highlight the importance of high-resolution visual input and sufficient temporal coverage for reliable OCR in dynamic video scenarios. 1 Introduction In recent years, the rapid advancement of Multimodal Large Language Models (MLLMs) [ 1 , 11 , 29 , 48 , 43 , 40 ] has garnered significant attention. These models, capable of processing and integrating information across various modalities (e.g., text, images, and video), have demonstrated considerable potential and significant value across a wide range of real-world applications [ 23 , 3 , 25 , 7 , 38 , 56 , 54 ] . Figure 1 : An example in MME-VideoOCR . The task requires the MLLM to first recognize the textual information distributed across multiple video frames, and then to perform semantic understanding and reasoning over the extracted text to accurately determine the correct answer. The correct information is marked in blue , while misleading information is marked in red . Optical Character Recognition (OCR) [ 34 ] , a fundamental technology in visual understanding, serves as a crucial link for enabling structured comprehension of image and video content. It transforms visual information into computationally analyzable semantic data. Within cross-modal learning, OCR provides critical feature support for text-visual alignment, directly impacting the performance of downstream tasks [ 49 , 28 ] . Previous OCR-based benchmarks [ 41 , 33 , 21 , 27 , 14 ] primarily focus on evaluating the OCR-based capabilities of MLLMs in static image scenarios. Several studies [ 35 , 8 ] have initiated preliminary investigations into video scenarios. However, they typically concentrate on perceiving textual content, often neglecting text-based understanding and reasoning. Considering the unique challenges of video understanding tasks, a comprehensive video OCR evaluation must address three key issues, as illustrated in Figure 1 . Firstly, textual information in videos can appear in various forms—such as foreground text, background scenery, on-screen annotations, watermarks, and floating overlays. This requires models to establish robust spatio-temporal visual-text associations and to effectively identify and extract relevant textual information from these diverse and often noisy sources across different shots. Secondly, critical textual information in videos is often distributed across multiple frames, rather than appearing in a single static image. Therefore, models must be capable of effectively recognizing, integrating, and understanding text content over time, leveraging temporal context to reconstruct and interpret fragmented or sequentially presented information. Thirdly, as task complexity increases, models must be able to reason over the recognized text. This reasoning ability is essential for deeper video understanding and remains a significant challenge for current MLLMs. Table 1 : Key differences between MME-VideoOCR and prior video-based OCR benchmarks . MME-VideoOCR features a larger number of task types and scenarios, employs fully manual annotations to ensure reliability, supports bilingual content for broader coverage, and enables comprehensive evaluation across perception, understanding, and reasoning. Benchmarks #Videos #QA #Tasks #Scenarios Annotation Bilingual Perception Understanding Reasoning OCR Benchmark [ 35 ] 25 1,477 1 20+ M FG Bench [ 8 ] 1,028 2,961 6 20+ A&M MME-VideoOCR 1464 2,000 25 44 M In this paper, we propose the MME-VideoOCR benchmark, which provides a comprehensive evaluation framework for OCR tasks in video scenarios. Recognizing the limitations of current OCR tasks in existing evaluation datasets, MME-VideoOCR encompasses 10 task categories and 25 specific tasks, incorporating a substantial number of actively collected or custom-created videos. As shown in Table 1 , MME-VideoOCR consists of 1,464 videos, paired with 2,000 diverse and accurately human-annotated question-answer (QA) pairs. The tasks require answers based on both localized key information and a holistic understanding of the entire video. The main contributions are summarized as follows: 1. MME-VideoOCR introduces a diverse set of video OCR tasks, utilizing manually quality-controlled videos and question-answer pairs. These tasks span multiple dimensions, such as perceptual accuracy, contextual comprehension, and cross-frame reasoning, which together enable a comprehensive evaluation of MLLMs’ OCR capabilities in video scenarios. 2. We evaluate 18 state-of-the-art MLLMs, including publicly available models ranging from 7B to 78B in size, as well as closed-source models like GPT-4o and Gemini-2.5 Pro. The results demonstrate strong discriminative power and the challenges posed by MME-VideoOCR. Regarding discriminative power, the worst-performing model, LLaVA-OneVision 7B, has an accuracy of 46.0\% , while the best-performing model achieves an accuracy of 73.7\% , showing a significant gap in performance. Regarding task difficulty, on several tasks we designed, such as Cross-Frame Text Understanding and Text-Based Video Understanding, most models score below 60\% . 3. The evaluation results further reveal significant deficiencies in current models on OCR tasks that require spatio-temporal reasoning and cross-frame information integration, thereby indicating a critical direction for MLLM optimization. Moreover, both high-resolution visual inputs and sufficient temporal coverage are essential for achieving reliable OCR performance in dynamic video settings. Notably, MLLMs exhibit a strong language prior bias during text recognition, frequently favoring semantically plausible outputs over visually accurate transcriptions. 2 Related Work 2.1 Multimodal Large Language Models Through the integration of a vision encoder into Large Language Models (LLMs) and pretraining on large-scale multimodal data, MLLMs exhibit strong OCR capabilities, making them well-suited for downstream tasks such as document understanding [ 16 , 32 ] , key information extraction [ 28 ] , and scene text recognition [ 62 , 47 ] . Building on this foundation, some recent MLLMs [ 59 , 40 , 2 , 63 , 10 , 55 ] have further extended their capabilities to handle video inputs, enabling them to process dynamic visual information. This advancement enables MLLMs not only to recognize text in static images, but also to extract text-related information from videos and leverage it for