EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer vision problems

JaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, Suan Lee · 2026

Read EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer vision problems by JaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, Suan Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27223.txt.

Introduction

Kim and co-authors present EuraGovExam, a benchmark built from real civil service exams in five Eurasian regions. The exams come from South Korea, Japan, Taiwan, India, and the European Union. Over 8,000 scanned multiple-choice questions span 17 academic and administrative domains. Each item sits inside one image, with choices and visuals embedded in the page. Models must reason across languages and layouts from that image alone. Even the best vision-language models reached only 86 percent accuracy. The set targets e-governance and public-sector document analysis too.

Catalog record

CREATORJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, Suan Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64965
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer vision problems JaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, Suan Lee Abstract . We present EuraGovExam , a multilingual and multimodal benchmark sourced from real-world civil service examinations across five representative Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union. Designed to reflect the authentic complexity of public-sector assessments, the dataset contains over 8,000 high-resolution scanned multiple-choice questions covering 17 diverse academic and administrative domains. Unlike existing benchmarks, EuraGovExam embeds all question content—including problem statements, answer choices, and visual elements—within a single image, providing only a minimal standardized instruction for answer formatting. This design demands that models perform layout-aware, cross-lingual reasoning directly from visual input. All items are drawn from real exam documents, preserving rich visual structures such as tables, multilingual typography, and form-like layouts. Evaluation results show that even state-of-the-art vision-language models (VLMs) achieve only 86% accuracy, underscoring the benchmark’s difficulty and its power to diagnose the limitations of current models. By emphasizing cultural realism, visual complexity, and linguistic diversity, EuraGovExam establishes a new standard for evaluating VLMs in high-stakes, multilingual, image-grounded settings. It also supports practical applications in e-governance, public-sector document analysis, and equitable exam preparation. Code | | Dataset | | Website \newfontfamily \cjkfont [Path=/usr/local/texlive/2025/texmf-dist/fonts/opentype/public/fandol/]FandolSong-Regular.otf Table 1 . Comparison of EuraGovExam with representative multimodal and exam-based benchmarks. Image-only : the model receives only an image with no separate text input. Orig. script : questions preserved in their source language without translation. N \times D : benchmark structure supports factorial analysis of nation/region and domain effects. \dagger subset of items are multimodal. Benchmark #Q #Lang. #Scripts #Domains Multi-modal Image-only Orig. script Real exam N \times D MMLU ( 13 ) 15,908 1 1 57 ✗ ✗ – ✓ ✗ MMMU ( 34 ) 11,550 1 1 30 ✓ ✗ – ✓ ✗ MATHVISTA ( 20 ) 6,141 1 1 5 ✓ ✗ – Partial ✗ AGIEval ( 38 ) 8,062 2 2 20 ✗ ✗ ✓ ✓ ✗ EXAMS ( 12 ) 24,000 16 2 24 ✗ ✗ ✓ ✓ ✗ M3Exam ( 37 ) 12,317 9 3 3 \dagger ✗ ✓ ✓ ✗ EXAMS-V ( 6 ) 20,932 11 4 20 ✓ ✓ ✓ ✓ ✗ EuraGovExam (Ours) 8,000 5+ 5 17 ✓ ✓ ✓ ✓ ✓ 1. Introduction Government examination documents are among the most visually demanding texts that people routinely encounter: a single page may combine dense tables, mathematical expressions, multilingual annotations, and region-specific layout conventions such as vertical Japanese typesetting or mixed-script Devanagari passages. Millions of civil servants worldwide are assessed through such documents every year, yet the vision-language models (VLMs) that increasingly assist in document analysis have never been systematically evaluated on them. Existing multimodal benchmarks either supply questions as clean digitized text ( 13 ; 38 ) , separate images from their textual context ( 37 ) , or rely on explicit task-specific prompts that bypass the visual complexity of authentic documents ( 20 ; 34 ) . As a result, it remains unclear how well current VLMs handle the full pipeline of real document understanding—from perceiving complex layouts and scripts to reasoning about the content within them. This gap is compounded by a pronounced bias toward English and Latin-script content ( 13 ; 6 ) , leaving open the question of whether script diversity poses a fundamentally different challenge from domain difficulty. We introduce EuraGovExam , a multilingual multimodal benchmark sourced from real civil-service examinations across five Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union. The dataset comprises over 8,000 high-resolution scanned multiple-choice questions spanning 17 academic and administrative domains. Two design principles distinguish EuraGovExam from prior work. First , all question content—problem statements, answer choices, tables, figures, and instructions—is embedded within a single image; models receive only a minimal, content-free formatting instruction as text input (§ 4 ). Second , questions are preserved in their original language, script, typography, and layout without any translation, OCR extraction, or reformatting, ensuring that the benchmark faithfully reflects the perceptual and linguistic demands of authentic documents. Our evaluation of 28 VLMs reveals findings that challenge prevailing assumptions about multimodal evaluation. (i) Region dominates domain : nation/script factors induce 2.52\times greater performance variance than domain factors, with within-model cross-regional gaps reaching 67.9 percentage points—even between regions sharing the same character family (e.g., Japan vs. Taiwan). (ii) Systematic geographic failure patterns : models achieving over 90% accuracy on Taiwanese and EU questions can simultaneously fall below 30% on Japanese questions of comparable format, suggesting that script-specific visual processing constitutes a critical bottleneck. (iii) Universal failure concentration : 50 questions (0.6%) defeat all 28 models, disproportionately concentrated in regions with complex non-Latin scripts and dense visual layouts. In summary, this paper makes the following contributions: • We present EuraGovExam , a high-fidelity benchmark of 8,000+ image-only exam questions from five Eurasian regions and 17 domains, with a leakage-aware, fully reproducible evaluation protocol. • We demonstrate that nation/script is the dominant source of VLM performance variance , inducing 2.52\times greater variance than domain factors—challenging the prevailing domain-centric view of multimodal evaluation. • We provide systematic cross-regional and universal failure analysis , revealing that geographic performance gaps persist across model families and scales, and characterizing the shared visual and linguistic properties of questions that defeat all evaluated models. 2. Related Work We position EuraGovExam at the intersection of multimodal question answering, exam-based benchmarks, and multilingual, domain-diverse evaluation datasets. This section reviews prior work across these axes. 2.1. Multimodal Question Answering Multimodal QA has received growing attention with the development of vision-language models. Early efforts such as VQA ( 3 ) and GQA ( 15 ) focused on everyday scenes, testing spatial and commonsense reasoning. More academic benchmarks like TQA ( 17 ) , ScienceQA ( 21 ) , and MMMU ( 34 ) extend this paradigm to educational domains, incorporating textbook diagrams and subject-specific visuals. MathVista ( 20 ) further challenges models with mathematical reasoning grounded in visual contexts. While these benchmarks cover structured content, they are largely English-centric and limited in subject diversity. In contrast, EuraGovExam introduces 17 domains—including law, administration, and philosophy—with fully image-based inputs drawn from real civil service exams across multilingual and regional sources. 2.2. Exam-Based and Domain-Specific Benchmarks Examination-based QA has emerged as a strong proxy for evaluating broad knowledge and reasoning. MMLU ( 14 ) set a standard with 57 subjects across disciplines, but remains text-only and English-based. Domain-specific datasets such as HEAD-QA ( 31 ) (Spanish medical exams) and MedMCQA ( 29 ) (Indian medical entrance tests) highlight the utility of expert-level assessments. AGIEval ( 38 ) includes diverse professional exams (e.g., LSAT, SAT, public service) to probe LLM capabilities, yet most remain unimodal. EuraGovExam builds on these works by combining the rigor of real-world exam questions with the complexity of image-based inputs, reflecting authentic visual and linguistic features. Figure 1 . EuraGovExam Dataset Construction Pipeline 2.3. Multilingual and Multimodal Exam QA Multilingual benchmarks such as EXAMS ( 12 ) , IndoMMLU ( 18 ) , and CMMLU ( 35

Read the complete text