Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan’s National Assessment of Academic Ability

Kyosuke Takami, Yuka Tateisi, Satoshi Sekine, Yusuke Miyao · 2026

Read Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan’s National Assessment of Academic Ability by Kyosuke Takami, Yuka Tateisi, Satoshi Sekine, Yusuke Miyao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11663.txt.

Introduction

Takami, Tateisi, Sekine and Miyao build a human-grounded benchmark from Japan's National Assessment of Academic Ability. Authentic school exams offer a strong test bed for multimodal language models, yet Japanese K-12 benchmarks remain scarce. Their dataset uses middle-school items in science, mathematics and Japanese, preserving real layouts, diagrams and text with nationwide student response distributions from about 900,000 pupils. It enables human-model comparison under one framework. Recent models show wide variation across subjects and sensitivity to visual reasoning, with human and LLM-as-judge scoring checks.

Catalog record

CREATORKyosuke Takami, Yuka Tateisi, Satoshi Sekine, Yusuke Miyao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39743
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11663

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan’s National Assessment of Academic Ability Kyosuke Takami, Yuka Tateisi, Satoshi Sekine, Yusuke Miyao Abstract Authentic school examinations provide a high-validity test bed for evaluating multimodal large language models (MLLMs), yet benchmarks grounded in Japanese K–12 assessments remain scarce. We present a multimodal dataset constructed from Japan’s National Assessment of Academic Ability, comprising officially released middle-school items in Science, Mathematics, and Japanese Language. Unlike existing benchmarks based on synthetic or curated data, our dataset preserves real exam layouts, diagrams, and Japanese educational text, together with nationwide aggregated student response distributions (N ≈ \approx 900,000). These features enable direct comparison between human and model performance under a unified evaluation framework. We benchmark recent multimodal LLMs using exact-match accuracy and character-level F1 for open-ended responses, observing substantial variation across subjects and strong sensitivity to visual reasoning demands. Human evaluation and LLM-as-judge analyses further assess the reliability of automatic scoring. Our dataset establishes a reproducible, human-grounded benchmark for multimodal educational reasoning and supports future research on evaluation, feedback generation, and explainable AI in authentic assessment contexts. Our dataset is available at: https://github.com/KyosukeTakami/gakucho-benchmark \setCJKmainfont [BoldFont=HaranoAjiMincho-Bold.otf]HaranoAjiMincho-Regular.otf \setCJKmonofont HaranoAjiGothic-Regular.otf 1 Introduction Multimodal large language models (MLLMs) are increasingly evaluated using exam-style benchmarks, yet resources grounded in authentic K–12 assessment contexts remain limited, particularly for Japanese educational settings. Existing benchmarks such as MMLU and C-Eval primarily focus on textual reasoning and only partially capture the multimodal characteristics of real school examinations. In Japanese middle-school assessments, questions often integrate dense visual elements—including diagrams, tables, and visually embedded text—within structured pedagogical layouts designed to assess curriculum-aligned competencies. These authentic design features introduce multimodal reasoning demands that are not fully represented in current evaluation resources, motivating the need for benchmarks derived from real educational assessments. Unlike existing benchmarks, our dataset includes nationwide aggregated student response distributions, enabling direct comparison between human and LLM performance under the same evaluation framework. 2 Contributions • We introduce a multimodal benchmark derived from Japan’s National Assessment of Academic Ability, preserving authentic exam layouts, diagrams, and Japanese educational text features. • The dataset includes nationwide aggregated response distributions (N \approx 900,000), enabling empirical estimation of task difficulty and reasoning patterns. • We enable direct comparison between human and LLM performance under a unified evaluation framework, supported by both human annotations and LLM-as-judge scoring. Motivation and challenge. Figure 1: An example of the original exam page. Existing benchmarks evaluate LLMs primarily using synthetic or curated datasets, which lack grounding in real-world educational data. In particular, they do not provide population-level response distributions, making it difficult to compare model performance with human learners in a meaningful way.However, constructing such datasets from real-world assessments is itself non-trivial. Raw exam materials are typically released as PDFs with complex multimodal layouts, requiring careful reconstruction of reading order, visual structure, and answer formats before they can be used for systematic evaluation. As illustrated in Figure 1 (an example Science item from Japan’s National Assessment of Academic Ability), real examination pages combine heterogeneous visual and textual primitives: dialog in speech balloons, numerical readouts on devices, maps with isobars and legends, and both horizontal and vertical Japanese text. Panel boundaries are irregular, reading order is implicit, and labels must be associated with graphical referents. While these multimodal layouts support student comprehension by providing redundancy and context, they introduce bottlenecks for machines—such as OCR errors on vertical text, difficulties in segmenting uneven panels, and challenges in linking callouts to figures. To address these issues, we propose a pipeline that transforms such human-friendly but machine-unfriendly exam materials into a structured and reproducible multimodal benchmark. Specifically, we segment each page into minimal visual units (e.g., figures, tables, text boxes), assign consistent identifiers and tags, and provide evidence-oriented descriptions that capture quantitative or referential information within each image. We further normalize reading order and cross-panel dependencies through a unified JSON schema that links subquestions to the panels they reference. Each item is then released in dual form—original image(s) and normalized text—to facilitate multimodal reasoning evaluation. The resulting dataset is derived from Japan’s National Assessment of Academic Ability and includes officially released middle-school questions in Science, Mathematics, and Japanese Language. Each question is represented in a structured format preserving original figures, mathematical expressions, and empirically reported answer distributions. In addition to automatic scoring, human evaluation was conducted for open-ended questions as a preliminary check of scoring validity. This dataset provides an initial step toward systematic benchmarking of multimodal comprehension in Japanese educational assessment contexts, bridging textual, visual, and cognitive reasoning while establishing reproducible standards for cross-model comparison. 3 Related Work 3.1 Exam-style Benchmarks for LLM Evaluation A growing body of work evaluates LLMs using exam-style benchmarks that probe academic knowledge and structured reasoning. Widely used datasets such as MMLU 9 , AGIEval 30 , and MMLU-Pro 31 provide large-scale collections of textual questions across multiple disciplines. Language-specific benchmarks including C-Eval 12 and CMMLU 15 , as well as domain-focused datasets such as GPQA 23 and mathematical benchmarks like GSM8K 4 and MATH 10 , further examine reasoning capabilities. Recent efforts have incorporated authentic educational materials, including RACE 14 , EXAMS 8 , GAOKAO-Bench 28 , and JEEBench 2 . However, benchmarks reflecting Japanese K–12 assessment practices remain limited, particularly those capturing multimodal reasoning in real exam settings. 3.2 Multimodal Benchmarks Multimodal benchmarks have rapidly expanded to evaluate vision–language understanding. Datasets such as MS-COCO 16 , VQA 1 , GQA 13 , and NLVR2 26 assess visual grounding and compositional reasoning, while DocVQA 19 and TextVQA 25 focus on structured visual-text understanding. Educational multimodal datasets such as ScienceQA 18 and MMMU 27 further introduce diagram-based and textbook-style problems. Nevertheless, benchmarks that combine curriculum-aligned tasks with authentic multimodal exam layouts remain scarce, especially in Japanese educational contexts. Our Position. This work bridges exam-style and multimodal evaluation by introducing a curriculum-grounded benchmark derived from Japan’s National Assessment of Academic Ability. The dataset captures authentic middle-school examination items combining textual, numerical, and diagrammatic reasoning, and preserves aggregated student response statistics to support reproducible evaluation and human–model comparison. 4 Dataset Construction Official Exam Materials PDF exam booklets; national assessment reports Manual Transcription Convert to Markdown; preserve reading order; insert image references; use LaTeX for mathematical expressions Image Extraction Extract diagrams and figures; link images to items; preserve visual layout Schema-based Conversion Automatic JSON/JSONL generation; item metadata structuring Response Metadata Answer distributions; correct / incorrect examples; scoring conditions Evaluation-ready Benchmark Multimodal QA tasks; open-ended scoring; LLM evaluation prompts Figure 2: Pipeline for constructing the benchmark from official exam PDFs and assessment reports. Japan’s National Assessment of Academic Ability. Japan’s Ministry of Education, Culture, Sports, Science and Technology (MEXT) has administered the National Assessment of Academic Ability annually since FY 2007 for all Grade 6 (elementary) and Grade 9 (middle school) students across the nation, making it a full-population (census-style) survey rather than a sampled one 20 ; 41 . The assessment combines subject tests with student and school questionnaires to support national PDCA-based educational policy and local classroom improvement. The core subjects—Japanese and mathematics—are assessed every year, while science and junior-high English are added on a roughly triennial rotation 41 . The National Institute for Educational Policy Research (NIER) publicly releases test items, answer

Read the complete text