CHECK-MAT: Checking Hand-Written Mathematical Answers for the Russian Unified State Exam
Khrulev Ruslan, Dataset · 2025
Read CHECK-MAT: Checking Hand-Written Mathematical Answers for the Russian Unified State Exam by Khrulev Ruslan, Dataset (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22958.txt.
Introduction
Khrulev introduces in twenty twenty-five a benchmark for judging how well vision-language models can grade handwritten maths work. Most tests check whether AI can solve problems. His test instead checks whether AI can read a student's written solution, spot mistakes, and grade it by fixed rules, as a human examiner would.
The set holds just over one hundred scanned solutions from Russia's national school exam, each with an official expert grade. He tested seven modern models from major labs in three modes of use. Results exposed real limits in maths reasoning and in matching human grading rules. The author offers the set as a spur for work on AI-assisted marking.
Catalog record
| CREATOR | Khrulev Ruslan, Dataset |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22958
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CHECK-MAT: Checking Hand-Written Mathematical Answers for the Russian Unified State Exam Khrulev Ruslan, Dataset Abstract This paper introduces a novel benchmark, EGE-Math Solutions Assessment Benchmark , for evaluating Vision–Language Models (VLMs) on their ability to assess hand-written mathematical solutions. Unlike existing benchmarks that focus on problem solving, our approach centres on understanding student solutions, identifying mistakes, and assigning grades according to fixed criteria. We compile 122 scanned solutions from the Russian Unified State Exam (EGE) together with official expert grades, and evaluate seven modern VLMs from Google, OpenAI, Arcee AI, and Alibaba Cloud in three inference modes. The results reveal current limitations in mathematical reasoning and human-rubric alignment, opening new research avenues in AI-assisted assessment. You can find code in https://github.com/Karifannaa/Auto-check-EGE-math Introduction The automated assessment of mathematical solutions, particularly those involving handwritten content and complex reasoning, represents a significant frontier for artificial intelligence. The evolution of this field has moved from deterministic, answer-focused systems to probabilistic models that attempt to evaluate a student’s entire problem-solving process. While numerous benchmarks exist to test a model’s ability to solve mathematical problems—such as MATH ( Hendrycks et al. 2021 ) or GSM8K ( Cobbe et al. 2021 ) — a critical gap remains in evaluating their capacity to understand, analyze, and grade human-generated solutions according to a predefined rubric. This paper addresses this gap by introducing a novel benchmark derived from the Russian Unified State Exam (EGE), a high-stakes graduation examination with a specialized mathematics track. The EGE mathematics exam comprises 19 tasks, with the second part requiring detailed, handwritten solutions that are manually graded by expert teachers. These experts assess scanned solutions based on clearly defined criteria that evaluate not just the final answer, but the correctness of intermediate steps and the validity of the underlying reasoning. Our benchmark leverages this structure by compiling a dataset of 122 problem solutions from the official EGE expert guide, including scanned handwritten work and the corresponding expert-assigned grades. This allows us to evaluate Vision-Language Models (VLMs) on their ability to emulate expert graders, a task that demands a shift from problem-solving to solution-assessment. We believe this approach offers a highly innovative perspective on model evaluation. Instead of merely testing a model’s problem-solving capabilities, our benchmark probes its capacity for nuanced understanding of human thought processes, error identification, and adherence to structured assessment rubrics. This is crucial for developing AI systems that can genuinely assist in educational assessment. Recent research has highlighted the challenges VLMs face, from error propagation due to inaccurate handwriting recognition to a lack of alignment with specific curricula ( Kasneci et al. 2023 ) . By focusing on a single, curriculum-constrained national exam, our work provides a practical case study in bridging the gap between the general capabilities of AI and the specific demands of real-world educational assessment. We evaluate seven state-of-the-art VLMs across three distinct evaluation modes, analyzing their performance on multiple metrics. Our findings shed light on the current strengths and weaknesses of these models in handling the complexities of handwritten mathematical solutions and the intricacies of expert grading criteria. This research contributes to the broader fields of educational technology and AI, paving the way for more sophisticated and human-centric AI assessment tools. Related Work Our work is situated at the intersection of automated mathematical assessment, multimodal reasoning, and diagnostic evaluation. The field has evolved significantly from its early reliance on deterministic systems to the current exploration of sophisticated AI models. From Answer Verification to Process Analysis The earliest forms of automated mathematical assessment were built on Computer Algebra Systems (CAS), with platforms like STACK ( Sangwin 2014 ) excelling at verifying the symbolic equivalence of a final answer. While effective for summative assessments, these systems are fundamentally "correctness-focused" and cannot evaluate the student’s reasoning process. This limitation spurred the adoption of Machine Learning and NLP techniques to analyze the textual content of student explanations, often by comparing a student’s response to a single "golden" solution. However, this still falls short of diagnosing errors in novel or unexpected solution paths. This long-standing tension between evaluating outcomes versus processes motivates the current research frontier of process-focused assessment, where our work is firmly situated. The Multimodal Challenge of Handwritten Solutions Assessing handwritten mathematics is an inherently multimodal task. A specialized field, Handwritten Mathematical Expression Recognition (HMER), has focused on the modular task of transcribing visual notation into a structured format like LaTeX ( Deng et al. 2017 ) . This is a non-trivial problem due to the two-dimensional structure of mathematical expressions and visual ambiguity between symbols. In parallel, the rise of end-to-end Vision-Language Models (VLMs) like GPT-4o ( OpenAI 2024 ) has introduced a more integrated paradigm. However, studies applying general-purpose VLMs to grade handwritten assignments have consistently highlighted the problem of error propagation : inaccurate Optical Character Recognition (OCR) of handwriting leads to faulty input for the reasoning module, resulting in incorrect grades ( Kasneci et al. 2023 ) . This suggests that a VLM’s generalist vision encoder may be less robust for the specific domain of mathematical notation than a specialized HMER model, making this a critical area for benchmarking. The Paradigm Shift to Diagnostic Assessment The most advanced research in this domain has shifted from simply assigning a score to performing diagnostic assessment—identifying the specific nature of a student’s error. This requires a model to parse a multi-step solution, compare it to valid reasoning pathways, and classify deviations into a meaningful taxonomy of error types. This paradigm shift is embodied by the recent development of specialized benchmarks. The Fermat benchmark ( Yuan et al. 2024 ) was explicitly designed to evaluate a VLM’s ability to perform error detection, localization, and correction on handwritten solutions containing synthetically generated, human-verified errors. Similarly, the MathCCS benchmark ( Wu et al. 2024 ) uses real-world student data to focus on systematic error analysis as a foundation for generating pedagogically useful feedback. The creation of these rigorous benchmarks signifies a maturation of the field, moving the central question from "Can the model get the score right?" to "Can the model identify and explain the error?" Our work aligns with this trajectory by requiring models to assess solutions against a multi-point rubric that implicitly requires error diagnosis. Evaluating the Evaluators: Reasoning Benchmarks The performance of any assessment system is capped by the reasoning power of its underlying models. While text-based benchmarks like MATH ( Hendrycks et al. 2021 ) and GSM8K ( Cobbe et al. 2021 ) drove initial progress, they suffer from issues like data contamination and an inability to penalize flawed reasoning that leads to a correct answer. The need to evaluate reasoning in visual contexts has led to more robust multimodal benchmarks. MathVista ( Lu et al. 2023 ) , for example, provides a comprehensive suite of problems requiring the interpretation of charts, diagrams, and figures. The significant performance gap between state-of-the-art models and humans on such benchmarks demonstrates that visually-grounded mathematical reasoning remains a formidable challenge. This provides essential context for our work, as it establishes a realistic upper bound on the expected performance of VLMs on the even more complex task of grading, which requires not only solving a problem but diagnosing errors in another agent’s solution. Benchmark Design