DiagramIR: An Automatic Pipeline for Educational Math Diagram Evaluation

Vishal Kumar, Shubhra Mishra, Rebecca Hao, Rizwaan Malik, David Broman, Dorottya Demszky, Stanford University, KTH Royal Institute of Technology · 2025

Read DiagramIR: An Automatic Pipeline for Educational Math Diagram Evaluation by Vishal Kumar, Shubhra Mishra, Rebecca Hao, Rizwaan Malik, David Broman, Dorottya Demszky, Stanford University, KTH Royal Institute of Technology (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08283.txt.

Introduction

Vishal Kumar, Shubhra Mishra, Rebecca Hao, and Rizwaan Malik present DiagramIR, an automatic pipeline for judging educational math diagrams generated by language models. Models can now write code that compiles into geometry figures for teaching, but checking those figures at scale is a bottleneck. Their method converts LaTeX TikZ code into intermediate representations that capture the intended structure of a diagram.

Scoring through those representations agrees with human raters better than asking a language model to judge directly. The pipeline also lets a smaller model, GPT-4.1 Mini, match a much larger one at about a tenth of the inference cost. Cheaper, trustworthy diagram evaluation supports wider use in accessible education tools.

Catalog record

CREATORVishal Kumar, Shubhra Mishra, Rebecca Hao, Rizwaan Malik, David Broman, Dorottya Demszky, Stanford University, KTH Royal Institute of Technology
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08283

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiagramIR: An Automatic Pipeline for Educational Math Diagram Evaluation Vishal Kumar, Shubhra Mishra, Rebecca Hao, Rizwaan Malik, David Broman, Dorottya Demszky, Stanford University, KTH Royal Institute of Technology Abstract Large Language Models (LLMs) are increasingly being adopted as tools for learning; however, most tools remain text-only, limiting their usefulness for domains where visualizations are essential, such as mathematics. Recent work shows that LLMs are capable of generating code that compiles to educational figures, but a major bottleneck remains: scalable evaluation of these diagrams. We address this by proposing DiagramIR : an automatic and scalable evaluation pipeline for geometric figures. Our method relies on intermediate representations (IRs) of LaTeX TikZ code. We compare our pipeline to other evaluation baselines such as LLM-as-a-Judge, showing that our approach has higher agreement with human raters. This evaluation approach also enables smaller models like GPT-4.1-Mini to perform comparably to larger models such as GPT-5 at a 10x lower inference cost, which is important for deploying accessible and scalable education technologies. \workshoptitle MATH-AI 1 Introduction Large Language Models (LLMs) have garnered significant attention for their potential to enhance education [ 7 , 8 ] , and recent studies have shown that students are increasingly adopting AI as part of their learning process [ 17 , 11 ] . Despite this, the current wave of LLM-assisted learning tools remains narrow: most rely on chatbot-style interfaces, where text is the sole input and output. While this paradigm has been useful for non-visual subjects, it can be limiting for highly visual subjects like mathematics, where diagrams and spatial reasoning play a key role in students’ understanding of problems and concepts [ 6 ] . Recent work has examined how LLMs can be used to generate pedagogically-useful mathematical diagrams [ 13 , 10 ] , but they face a critical challenge: evaluation. Evaluating whether a mathematical figure is useful is challenging: it involves (1) evaluating whether the diagram aligns with the prompt, (2) whether the diagram is mathematically and visually sound, and (3) whether the diagram is pedagogically meaningful given the context. The first and third aspects are difficult given their subjective nature and the complexity of pedagogical evaluation, requiring further study. We focus in this paper on the second aspect: scalable evaluation of whether a diagram is mathematically and visually sound. Current techniques largely rely on human evaluation [ 13 ] , which provide high quality insights for static projects, but cannot be deployed scalably in live chatbot-like application settings. One promising direction in these settings has been to apply LLM-as-a-Judge methods, which have proven effective in text-based evaluation tasks [ 16 ] . However, extending this paradigm to diagrams introduces unique obstacles. Unlike text, diagrams require reasoning over geometry, proportions, and spatial layout, which are all features that LLMs cannot directly process without translation into symbolic or textual representations. Multimodal Language Models (MLMs) [ 2 ] are a natural alternative, but benchmarks like MathVista [ 12 ] and MathVerse [ 15 ] demonstrate that current MLMs remain unreliable at solving problems about, or even interpreting mathematical visuals. As a result, diagram judgments made by LLMs or MLMs can be inconsistent, mathematically incorrect, or misaligned with pedagogical intent. And notably, closed-source and larger frontier models have been shown to be more effective at LLM-as-a-Judge techniques [ 16 ] . However, their high costs and limited accessibility make them impractical for real-world educational deployments, especially in resource-constrained settings. For education technology to be truly equitable, evaluation methods must work reliably with open models and lightweight pipelines that can be deployed at scale at low costs. This gap highlights the need for task-specific, automated pipelines for evaluating mathematical diagrams. In our work, we present DiagramIR: an automatic pipeline that evaluates TikZ-generated diagrams using back-translation [ 14 , 4 , 5 ] and grammar-based mathematical and spatial checks. Specifically, our contributions are as follows: 1. We introduce a back-translation pipeline to automatically and scalably evaluate mathematical and spatial criteria of diagrams (Section 2.3 ). This novel method outperforms LLM-as-a-Judge on agreement with human evaluation, with stronger performance across all models, which is essential for educational deployments (Section 3 ). 2. We utilize a 398-item evaluation dataset of real-world mathematical diagram generations from teachers (Section 2.1 ), demonstrating an approach for the continued development and improvement of automatic evaluation pipelines for mathematical diagrams. Figure 1: Different evaluation approaches for TikZ-generated code. The left shows TikZ code as the common input. Top: We asked human evaluators to rate the TikZ compiled images based on the rubric discussed in Section 2.2 . Middle: LLM-as-a-Judge uses either the TikZ code, the rendered image, or both to make judgments. Bottom: In our back-translation method, an LLM translates the TikZ code into an intermediate representation (IR) with multiple fields, after which automatic checks (e.g., whether the diagram is fully in canvas or whether outlines are closed) are run. A diagram is considered valid if all check pass. 2 Methods 2.1 Evaluation Dataset - Geometric Diagrams Given the downstream educational application of our pipeline, we ground our evaluation dataset in real-world data. Specifically, we use conversational data from Coteach 1 1 1 https://coteach.ai/ , an AI assistant for mathematics educators that uses the Illustrative Mathematics K–12 Math v.360 curriculum. For the dataset to reflect the types of diagrams teachers most commonly generate, we pulled 6,000 random conversations between teachers and the tool. The most frequent type of diagram request category were geometric constructions, which is why we choose to focus on 2D and 3D shapes in this work. We sampled 398 instances of geometric constructions (i.e. pairs of (teacher’s request, generated code)), resulting in 208 2D diagrams (triangles, circles, and rectangles) and 190 3D shape diagrams (prisms and cubes). We utilize 12 diagrams for human-eval calibration and pipeline development, leaving 386 for our test set. Since our entire dataset is derived from natural teacher-LLM interactions, we can evaluate the quality of diagrams used “in the wild”. The diagrams in the dataset represent a variety of error types (see Appendix A2.1 , which informed the design of our evaluation rubric. 2.2 Rubric Construction We developed a rubric that aims to capture criteria that (i) correspond to the most frequent failure modes observed in generated diagrams, (ii) are unambiguous, observable, and (iii) compatible with both human and LLM judging. The rubric targets two axes: mathematical correctness and spatial correctness . We designed the rubric by first reviewing a small sample of Coteach diagrams and open-coding recurrent failure modes (off-frame content, mislabeled angles, etc.). Then, we iteratively refined the criteria through calibration rounds where we scored parts of the Coteach dataset using our rubric to identify gaps. Our final rubric is shown in Table 1 . Criterion Allowed values Mathematical correctness Labeled angles match drawn angles Yes | No | N/A Labeled lengths/areas match proportions Yes | No | N/A Spatial correctness Diagram is fully in frame Yes | No Elements are scaled to be readable Yes | No Labels are associated with correct elements Yes | No | N/A Elements do not problematically overlap Yes | No Table 1: The rubric created for assessing the mathematical and spatial correctness of math diagrams. The rubric was used for human annotation, and was

Read the complete text