KMMMU: Evaluation of M assive M ulti-discipline M ultimodal U nderstanding in K orean Language and Context
Nahyun Lee, Guijin Son, Hyunwoo Ko, Chanyoung Kim, Junyoung An, Kyubeen Han, Il-Youp Kwak, [0.5em] Chung-Ang University Seoul National University SK A.X OnelineAI HAE-RAE · 2026
Read KMMMU: Evaluation of M assive M ulti-discipline M ultimodal U nderstanding in K orean Language and Context by Nahyun Lee, Guijin Son, Hyunwoo Ko, Chanyoung Kim, Junyoung An, Kyubeen Han, Il-Youp Kwak, [0.5em] Chung-Ang University Seoul National University SK A.X OnelineAI HAE-RAE (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13058.txt.
Introduction
Lee and six colleagues introduce KMMMU, a Korean-native benchmark of 3,466 questions across nine disciplines and nine visual modality types, built from Korean examination materials. The strongest open-weight model reaches 42.05 percent, while the best proprietary model scores 52.42 percent on the harder KMMMU-Adv set. Korean-specific questions show gaps up to 13.43 percent, with failures tied to local conventions and knowledge recall more than to reasoning depth.
Catalog record
| CREATOR | Nahyun Lee, Guijin Son, Hyunwoo Ko, Chanyoung Kim, Junyoung An, Kyubeen Han, Il-Youp Kwak, [0.5em] Chung-Ang University Seoul National University SK A.X OnelineAI HAE-RAE |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74613 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.13058
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KMMMU: Evaluation of M assive M ulti-discipline M ultimodal U nderstanding in K orean Language and Context Nahyun Lee, Guijin Son, Hyunwoo Ko, Chanyoung Kim, Junyoung An, Kyubeen Han, Il-Youp Kwak, [0.5em] Chung-Ang University Seoul National University SK A.X OnelineAI HAE-RAE Abstract We introduce KMMMU , a native Korean benchmark for evaluating multimodal understanding in Korean cultural and institutional settings. KMMMU contains 3,466 questions from exams natively written in Korean, covering nine disciplines and nine visual modality categories, along with a 300-item Korean-specific subset and a hard subset of 627 questions. Unlike translated or English-centric benchmarks, KMMMU targets information-dense problems shaped by local conventions, official standards, and discipline-specific visual formats. Experiments show that the strongest open-source model reaches only 42.05% accuracy on the full set, while the best proprietary model achieves 52.42% on the hard subset. Performance varies across disciplines, with some disciplines emerging as bottlenecks, and Korean-specific questions showing gaps of up to 13.43%. Error analysis suggests that these failures stem less from insufficient reasoning depth than from weak convention-to-label mapping, few-shot symbolic induction, localized knowledge recall, and domain-specific standards understanding. KMMMU provides a testbed for multimodal evaluation beyond English-centric benchmarks and for developing more reliable systems for expert real-world tasks. 1 1 1 Dataset is available at https://huggingface.co/datasets/HAERAE-HUB/KMMMU 1 Introduction Multimodal Large Language Models (MLLMs) have shown strong performance on a range of vision–language tasks, including visual recognition, document understanding, and multimodal question answering ( 2 ; 22 ; 24 ; 38 ; 4 ) . However, existing benchmarks do not fully reflect the settings in which these models are increasingly deployed ( 37 ; 10 ; 14 ) . Past evaluations either are English-centric or derived from translated datasets ( 23 ; 44 ) , making them less suitable for assessing performance on tasks shaped by local institutional conventions, discipline-specific formats, and information-dense visual materials in non-English contexts. Figure 1: Comparison of English (MMMU, MMMU-Pro), Japanese (JMMMU, JMMMU-Pro), and Korean (others) multimodal benchmarks. Each point is positioned by benchmark size (x-axis, log scale) and difficulty proxy (100 - peak public score), with lighter colors indicating more recent releases. Shaded regions mark two common limitations: small size (left) and low headroom (bottom). To address this gap, we introduce KMMMU , a native Korean benchmark for expert-level multimodal understanding. KMMMU contains 3,466 questions drawn from Korean assessment sources, spanning nine disciplines, nine visual modality categories, and both multiple-choice and open-form question formats. Beyond broad evaluation, the benchmark is designed to diagnose localized knowledge, expert reasoning, and discipline- and modality-specific weaknesses. To support this analysis, we construct a hard subset of questions jointly missed by three baseline models, as well as a Korean-specific subset targeting domestic legal, administrative, and institutional knowledge. Figure 2: Examples of KMMMU questions. Examples include the original questions, associated images, English translations, and metadata such as visual modality, question format, and Korean-specific labels. Experiments on KMMMU reveal several consistent findings. Current models remain far from robust, with the strongest open-source model reaching 42.05% on the full set and the best proprietary model reaching 52.42% on the hard subset. Performance varies substantially across disciplines, and gains from model scale and explicit reasoning are uneven. Korean-specific questions remain particularly challenging, with accuracy gaps of up to 13.43% relative to non-Korean-specific items. These results show that strong general multimodal ability does not automatically transfer to Korean institutional and cultural contexts. 2 Related Work In recent years, a diverse range of Korean multimodal benchmarks has already been introduced, including KRETA for text-rich VQA ( 17 ) , KoNET for exam-based educational assessment ( 33 ) , and KorMedMCQA-V for medical reasoning ( 7 ) , alongside resources targeting free-form VQA (KOFFVQA) ( 20 ) , cultural understanding (K-Viscuit) ( 33 ) , under-specified user queries (HAERAE-Vision) ( 8 ) , translated benchmark variants (K-MMBench, K-SEED), and document-centric reasoning (K-DTCBench) ( 19 ) . However, despite this diversity, most existing benchmarks remain limited in coverage, and many are already saturated for current models. This calls for a bigger, and a stronger benchmark. Harvesting questions from existing examinations is a common strategy for benchmark construction. Benchmarks such as MMLU, MMMU, and M3Exam all draw on exam-style questions to evaluate broad knowledge and reasoning, and related efforts have extended this paradigm to local languages and cultural contexts, as in JMMMU for Japanese and CMMMU for Chinese ( 15 ; 44 ; 47 ; 30 ; 46 ) . This approach is valuable because exam questions offer scale, disciplinary breadth, and an interpretable link to human expertise, making them useful proxies for general capability even when the evaluation format is limited to multiple-choice or short-form responses ( 48 ) . So why another X-MMMU benchmark? The Korean case further highlights why localized benchmarks remain necessary. KMMLU, for instance, is constructed from original Korean exams rather than translations, thereby capturing linguistic and cultural factors that translated benchmarks often miss ( 35 ) . Similarly, KMMLU-Pro ( 16 ) shows that the gap between translated MMMLU ( 31 ) and locally authored Korean professional exams is relatively small in medicine but substantially larger in law-related domains, where country-specific knowledge is indispensable. Together, these findings underscore the need for localized MMMU-style benchmarks tailored to each linguistic and cultural context. As suggested by Figure 1 , the current landscape still reflects a trade-off between breadth, realism, and headroom. Translation-based benchmarks improve comparability with established English suites, but they largely inherit the structure and limitations of their source tasks ( 40 ) . More realistic or culturally grounded benchmarks capture important failure modes, including cultural reasoning, text-rich understanding, and under-specified real-world queries, yet they are often narrower in scope or smaller in scale. Moreover, most existing Korean benchmarks already lie in the low-headroom region, while HAERAE-Vision, although comparatively difficult, derives much of its challenge from deliberate under-specification rather than broad coverage of general capabilities ( 34 ; 42 ) . Accordingly, there remains a clear need for a large-scale Korean multimodal benchmark that is broad in coverage, grounded in local context, and sufficiently unsaturated to differentiate frontier models. 3 The KMMMU Benchmark 3.1 Data Collection and Annotation KMMMU is constructed from Korean-native official examinations and competitions. These sources include the civil service recruitment (PSAT), National Technical Qualifications (NTQ), National Competency Standards exam (NCS), and academic Olympiads (see Appendix A for details). We initially collect approximately 68k raw instances. We process the collected exam materials into structured multimodal instances using automated extraction, followed by manual verification. Technical qualification data are collected through web crawling, while other sources are digitized using the MinerU-2.5 OCR system ( 29 ) . To correct OCR artifacts and validate image cropping, we built a custom verification interface. Five Korean annotators use this system to review the dataset, refine LaTeX formulas, verify image references, and discard illegible questions (see Appendix B for details). Additionally, we expect this step to reduce contamination risk. As a big portion of the dataset is acquired from PDF documents, the benchmark is less susceptible to large-scale web crawled datasets. We provide additional ablation studies in Appendix I . 3.2 KMMMU Dataset Construction To ensure benchmark difficulty, we apply a multi-stage adversarial filtering pipeline ( 45 ; 21 ) removing instances solvable