MathDoc: Benchmarking Structured Extraction and Active Refusal on Noisy Mathematics Exam Papers Chenyue Zhou, Jiayi Tuo, Shitong Qin, Wei Dai, Mingxuan Wang, Ziwei Zhao Duoyang Li, Shiyang Su, Yanxi Lu, Yanbiao Ma Abstract The automated extraction of structured questions from paper-based mathematics exams is fundamental to intelligent education, yet remains challenging in real-world settings due to severe visual noise. Existing benchmarks mainly focus on clean documents or generic layout analysis, overlooking both the structural integrity of mathematical problems and the ability of models to actively reject incomplete inputs. We introduce MathDoc, the first benchmark for document-level information extraction from authentic high school mathematics exam papers. MathDoc contains 3,609 carefully curated questions with real-world artifacts and explicitly includes unrecognizable samples to evaluate active refusal behavior. We propose a multi-dimensional evaluation framework covering stem accuracy, visual similarity, and refusal capability. Experiments on SOTA MLLMs, including Qwen3-VL and Gemini-2.5-Pro, show that although end-to-end models achieve strong extraction performance, they consistently fail to refuse illegible inputs, instead producing confident but invalid outputs. These results highlight a critical gap in current MLLMs and establish MathDoc as a benchmark for assessing model reliability under degraded document conditions. Our project repository is available at GitHub repository 1 Introduction In modern educational ( 13 ) scenarios, the complete and accurate extraction of questions from paper-based mathematics examinations into structured databases is a prerequisite for intelligent test assembly and personalized instruction ( 35 ) . Furthermore, constructing high-quality structured question banks provides high-value corpus support for training the mathematical reasoning ( 24 ; 8 ; 7 ; 23 ) capabilities of Large Language Models (LLMs). However, information extraction from examination documents in real-world scenarios faces several critical challenges: • Noise Interference: Handwritten solutions and grading marks frequently overlap with printed text, complicating the extraction of question stems. • Figure Extraction: Flexible layouts complicate text-diagram association, requiring simultaneous localization and contextual alignment. • Invalid Information Rejection: Real-world artifacts ( 33 ) like creases or occlusions frequently truncate question content. A key challenge lies in enabling models to actively identify and reject such incomplete information. Figure 1: Performance comparison of six Multimodal Large Language Models (MLLMs) across five metrics (Stem Accuracy, Visual Similarity, Refusal F1/Precision/Recall) in choice questions. While the rapid evolution of MLLMs ( 42 ; 32 ; 20 ; 29 ; 3 ; 16 ) and the establishment of benchmarks like OCRBench v2 ( 12 ) , OmniDocbench ( 31 ) have jointly advanced general document information extraction, current evaluations remain inadequate for authentic mathematics documents. Data-wise, existing benchmarks prioritize clean documents with standardized layouts, overlooking real-world noise. Task-wise, they operate on the assumption of input completeness, evaluating only extraction correctness while neglecting the critical capability to actively reject incomplete inputs. To bridge this gap, we introduce MathDoc, a large-scale benchmark of 3,609 high school mathematics questions captured in real educational settings. As is shown in Figure 1 , the radar chart illustrates the performance of selected models across different evaluation metrics in choice questions. We conducted comprehensive experiments on both open-source models ( Qwen3-VL ( 2 ) , InternS1 ( 1 ) , GLM-4.6V, DeepSeek-OCR ( 34 ) , MinerU2.5 ( 30 ) , PaddleOCR-VL ( 10 ) ) and closed-source models ( Gemini-2.5-Pro 9 , GPT-4o ( 17 ) , Doubao-Seed-1.6-vision ). Our experiments reveal that while SOTA models demonstrate impressive accuracy in extracting content from recognizable samples, they universally struggle with active refusal when facing incomplete or occluded inputs. Instead of acknowledging uncertainty, these models predominantly exhibit a tendency towards speculative completion and forced transcription, highlighting a critical gap in the reliability of current MLLMs for real-world applications. In summary, the contributions of this paper are summarized as follows: • We introduce MathDoc , the first benchmark for document-level information extraction from real-world high school mathematics exam papers. The benchmark consists of 3,609 high-quality math problems and establishes a novel multi-dimensional evaluation protocol to comprehensively assess a model’s ability in problem statement identification under noisy document conditions, figure extraction, and answer refusal. • We conduct a systematic evaluation of SOTA MLLMs , covering both open-source and close-source models, providing a comprehensive comparison of their performance. • We conduct a comprehensive analysis to determine the decision boundaries of active refusal in current MLLMs, quantifying the threshold of information loss to trigger refusal. Furthermore, we propose a simple yet effective method to enhance the refusal capability. 2 Related Works 2.1 General Document Information Extraction Benchmarks Existing benchmarks ( 36 ; 14 ; 25 ) for document information extraction primarily focus on layout analysis ( 40 ; 21 ; 39 ) and visual question answering ( 41 ; 27 ; 28 ; 19 ) , treating documents as collections of generic layout elements ( 18 ) or OCR tokens. However, they fail to assess unit-level structural integrity, which is critical for mathematical parsing. Specifically, current metrics overlook the cohesive extraction of “question units" (integrated stems, options, and figures), leaving a significant gap in evaluating question-level structural parsing ( 5 ) capabilities. 2.2 Mathematics Specific Document Information Extraction Benchmarks Exploration within the specific domain of mathematics document information extraction remains relatively scarce. Existing research falls into two primary categories: one focuses on formula extraction ( 15 ; 4 ; 22 ) ; the other ( 26 ; 38 ; 11 ; 37 ) , comprising the majority of recent studies, focuses on mathematical problem-solving capabilities under idealized visual inputs, rather than low-level document restoration capabilities. Crucially, both categories overlook complex document inputs from authentic educational scenarios, resulting in a lack of effective evaluation regarding model extraction performance under noise interference. Figure 2: (a) The process of Data Construction. (b) The distribution of all types of questions. (c) The average length of questions and their average formula ratio. 3 Dataset Construction 3.1 Data Acquisition We collected approximately 30,000 raw exam images from real-world high school mathematics scenarios, capturing diverse capture devices and physical conditions. To construct a benchmark that rigorously evaluates model robustness, as is illustrated in Figure 2 (a) we curated 460 highly challenging images via a two-stage pipeline: Layout Pre-screening: We explicitly filter out clean, simple samples to focus on complex geometries. We retain only those featuring multi-column layouts, dense text-figure interleaving to simulate authentic analysis challenges. Expert Selection: Human experts further identify samples with severe OCR noise, specifically targeting: (1) heavy handwriting occlusions and corrections in critical text regions; (2) physical obstructions (e.g., stains, wrinkles); and (3) compromised document integrity, such as incomplete questions or folded pages. 3.2 Data Annotation To balance high-quality ground truth with annotation efficiency, as is illustrated in Figure 2 (a) we established a standardized three-stage pipeline: Model-Assisted Pre-annotation: Leveraging the SOTA model Qwen3-VL-Plus, we generate initial structural Markdown sequences and automatically crop associated geometric figures. This provides a high-quality baseline, significantly reducing the manual workload of starting from scratch. Answer Refinement: Domain experts perform character-level verification under two strict protocols: (1) Content Correction , which involves rectifying textual errors and ensuring precise figure cropping; and (2) Refusal Labeling , where samples rendered illegible by noise are strictly assigned type-specific placeholders ( [Unrecognizable Choice] ). This establishes a basis for refusal benchmarking. Cross-Validation: To eliminate subjective bias, particularly in defining the boundary between “recognizable" and “unrecognizable," we implement a two-tier quality control mechanism: peer cross-checks (focusing on L a T e X syntax and refusal consistency), and a final adjudication by senior experts for ambiguous borderline samples. 3.3 Dataset Properties As is illustrated in Figure 2 (b), the charts present the proportion of Choice, Fill, and Solve questions for overall benchmark and its specific subsets. The first chart represents the overall distribution, which consists of 2,169 Choice, 758 Fill, and 682 Solve questions. The second chart shows the distribution within the subset of questions containing figures, comprising 162 Choice, 66 Fill, and 146 Solve.The third chart depicts the distribution within the “Unrecognizable” subset, which includes 381 Choice, 134 Fill, and 158 Solve. Regarding content details in Figure 2 (c), the average character length per question ranges from 110 to 195 , with Solve questions being the longest. Furthermore, mathematical formulas constitute a substantial portion of the text (ranging from 44% to 50%), requiring models to possess robust capabilities in complex formulation recognition. 4 Benchmark Construction 4.1 Task Definition We define this task as a document structured extraction problem. Given an input image of a mathematics exam paper I , the goal is to parse it and generate a structured set of problems: \mathcal{S}=\{q_{1},q_{2},\dots,q_{N}\}. (1) This task requires the model to simultaneously perform three sub-tasks: structured text extraction, figure extraction, and intelligent rejection. Structured Text Extraction: The extracted problem text T_{i} must not only match the content in the document exactly, but also preserve the reading order consistent with human perception (top-to-bottom, left-to-right). Figure Extraction: For problems q_{i} containing geometric figures, the model is required to predict the bounding box coordinates B_{i} of the figure, where the corresponding cropped image region must exhibit high semantic similarity with the ground-truth figure. Intelligent Rejection : Only problems with complete visual information should be extracted. We define a ground-truth-based completeness function \Phi(I,q_{i})\in\{0,1\}. (2) The expected output of the model O_{i} is: O_{i}=\begin{cases}T_{i},&\text{if }\Phi(I,q_{i})=1\\ \mathtt{\langle REJECT\rangle},&\text{if }\Phi(I,q_{i})=0\end{cases} (3) Here, \Phi=0 (ground-truth rejection) is defined by the following two cases: • Critical Occlusion : The core region of the problem is completely covered. • Physical Truncation : The problem is located at the edge of image, causing semantic interruption. 4.2 Evaluation Pipeline Construction Figure 3: The pipeline of evaluation process To provide a fair and comprehensive evaluation of different models, we propose an end-to-end evaluation pipeline, arranged sequentially in the following steps: pre-processing, text matching, rejection matching, and metric computation (see Figure 3 ). 4.2.1 Pre-Processing To ensure robust downstream matching, we define a text normalization function \phi(\cdot) that standardizes input sequences through the following operations: Filtering Non-Semantic Noise: We eliminate irrelevant metadata (exam instructions, headers) using rule-based regex to reduce interference. Canonicalizing Formats: We reduce ambiguity by unifying L a T e X mathematical syntax and converting relative image paths to absolute ones. Decoupling Reference Answers: We explicitly separate answers from problem statements to accurately confine the downstream matching scope. 4.2.2 Text Matching This process establishes a fine-grained semantic mapping between valid questions in the Ground Truth (GT) and the model predictions (Pred) by precisely locating question boundaries within the continuous, unstructured prediction stream. Problem Definition: Given a GT question set \mathcal{Q}=\{q_{i}\}_{i=1}^{M} and a Pred text stream \mathcal{T} of length N , the goal is to determine a sequence of boundary indices for precise text matching: \hat{\Omega}=\{[\hat{s}_{i},\hat{e}_{i}]\}_{i=1}^{M} (4) Here, \hat{s}_{i} and \hat{e}_{i} represent the integer character indices in \mathcal{T} ,corresponding to the start and end of the i -th question. These indices must satisfy the strict sequential constraint: 1\leq\hat{s}_{1}<\hat{e}_{1}\leq\hat{s}_{2}<\dots<\hat{e}_{M}\leq N (5) Boundary Indices Localization: To achieve robust alignment, we propose a hierarchical strategy that synergizes semantic understanding with deterministic search. The process operates in two sequential phases: (1) Substring Extraction, extracting text segments based on semantic correspondence; and (2) Index Localization, determining their precise integer indices. (1) Substring Extraction: This phase utilizes the LLM to identify the specific textual segments that demark the boundaries of each question q_{i} within the noisy prediction stream \mathcal{T} . (\text{str}^{\text{start}}_{i},\text{str}^{\text{end}}_{i})=\operatorname{LLM}(q_{i}\mid\mathcal{T},\mathcal{Q}) (6) By jointly processing the ground-truth question set \mathcal{Q} and the predicted text stream \mathcal{T} , the model locates the start ( \mathrm{str}_{i}^{\text{start}} ) and end boundary ( \mathrm{str}_{i}^{\text{end}} ) of each question q_{i} within the noisy stream \mathcal{T} . (2) Index Localization: Let k denote a candidate start index in the predicted text stream \mathcal{T} , used to locate either start anchor \hat{s}_{i} or end anchor \hat{e}_{i} of question q_{i} , with the optimal index determined via the following hierarchical search: \hat{k}=\begin{cases}\mathrm{find}(\text{str}\subset\mathcal{T}),&\text{L1},\\[1.0pt] \displaystyle\arg\max_{k}\;\mathrm{sim}(\text{str},\mathcal{T}_{k}),&\text{L2},\\[1.0pt] k_{\mathrm{fall}},&\text{L3}\end{cases} (7) Localization Logic: The search logic follows three distinct levels: • Level 1 (Exact Match) Scans the text stream to locate the exact literal occurrence of the anchor substring in pred text streams. • Level 2 (Fuzzy Match) If exact match fails, a sliding window (8–15 characters) traverses the text. The position with the highest normalized Levenshtein similarity is selected, provided the score exceeds the threshold \tau=0.8 . • Level 3 (Index Fallback) If both matches fail, the algorithm defaults to the last verified position to maintain continuity. Specifically, it assigns the previous question’s end index to a missing start anchor ( \hat{s}_{i}\leftarrow\hat{e}_{i-1} ) or collapses a missing end anchor to the current start index ( \hat{e}_{i}\leftarrow\hat{s}_{i} ). 4.2.3 Refusal Alignment For ground truth entries marked as noise (“[Unrecognizable]"), text matching is inapplicable. Instead, we employ a refusal alignment strategy: • Implicit Refusal: The model yields an empty output ( \hat{y}_{i}=\emptyset ) for a certain question. • Explicit Refusal: The model generates predefined rejection tokens (e.g., “[Unrecognizable]") or reasons for refusal to answer, achieving a direct label match with the ground truth. Figure 4: Model performance on Solve, sorted in descending order according to the Final metric (left to right). 4.2.4 Evaluation Metrics We evaluate the model’s performance across three dimensions using the following metrics. Stem Accuracy: We utilize the Normalized Levenshtein Ratio to quantify the text extraction similarity across all M questions: S_{\text{text}}=\frac{1}{M}\sum_{i=1}^{M}\left(1-\frac{\operatorname{Lev}(\phi(T^{\text{Pred}}_{i}),\phi(Q^{\text{GT}}_{i}))}{\max(|\phi(T^{\text{Pred}}_{i})|,|\phi(Q^{\text{GT}}_{i})|)}\right) (8) Visual Similarity: We employ a MLLM (Qwen3VL-Plus) as a Semantic Judge ( 6 ) \mathcal{J} to assess the visual similarity between the extracted image I^{\text{Pred}}_{i} and the ground truth I^{\text{GT}}_{i} : S_{\text{img}}=\frac{1}{M}\sum_{i=1}^{M}\mathcal{J}(I^{\text{Pred}}_{i},I^{\text{GT}}_{i}) (9) where the function \mathcal{J} outputs a semantic similarity score in the range [0,1] Refusal Capabilities: We formulate the identification of incomplete questions as a binary classification task, treating Refusal as the positive class. The model’s performance is evaluated using standard Precision, Recall, and F1-Score. Final Performance Score: The final performance score aggregates text, image, and refusal metrics. Specifically, the refusal dimension is composed of Precision, Recall, and F1-score, each contributing equally to the dimension’s total weight: S_{\text{final}}=\frac{1}{3}S_{\text{text}}+\frac{1}{3}S_{\text{img}}+\frac{1}{9}\left(P_{\text{ref}}+R_{\text{ref}}+F1_{\text{ref}}\right) (10) 5 Experiments In this section, we benchmark SOTA MLLMs on the MathDoc dataset, presenting a comprehensive quantitative evaluation followed by an in-depth analysis. The results are shown in Figure 4 , 5 and 6 . Furthermore, we design two targeted experiments to investigate the underlying mechanisms of the models’ refusal capabilities. The results are shown in Figure 7 and 8 . 5.1 Main Result Analysis Analysis 1: As shown in Figure 4 , Qwen3-VL-30B achieves state-of-the-art extraction performance ( S_{\text{text}}=0.89 , S_{\text{img}}=0.63 ). However, refusal evaluation reveals a consistent “Low Recall, High Precision” pattern across most models (excluding Gemini). For example, despite its strong extraction accuracy, Qwen3-VL-30B attains a refusal recall of only 0.14 (Figure 5 ). This imbalance indicates that while models can produce correct refusals when triggered, they rarely initiate refusal proactively. Instead, they tend to forcibly transcribe fragmented content or speculate missing details. We attribute this behavior to the lack of explicit negative supervision for document incompleteness during pretraining and instruction tuning, resulting in under-trained decision boundaries for completeness-aware rejection. Analysis 2: Most models perform better on Problem-Solving tasks than on Choice tasks, primarily due to structural differences. Choices require precise localization of densely packed options, where minor spatial misalignment can cause extraction errors. However, Solve tasks provide richer context, enabling models to leverage broader linguistic priors to mitigate perceptual ambiguities. Analysis 3: End-to-End MLLMs generally outperform multi-stage pipeline frameworks (MinerU, PaddleOCR-VL). Pipeline approaches typically suffer from error cascading, where failures in the initial layout analysis phase irreversibly corrupt subsequent text recognition. Figure 5: Model performance on Choice, sorted in descending order according to the Final metric (left to right). Figure 6: Model performance on Fill, sorted in descending order according to the Final metric (left to right). 5.2 Investigation of Refusal Mechanisms The pervasive low Refusal F1 scores shown in the table necessitate a deeper examination of the underlying refusal behaviors. Consequently, we design targeted experiments to answer two core questions: • Q1: What degree of information loss triggers a model’s refusal response? • Q2: What strategies can enhance the model’s refusal performance? 5.2.1 Sensitivity to Information Loss Experimental Setup: To investigate the specific boundary conditions under which models tend to refuse answering due to missing information, We constructed a evaluation set consisting of 50 independent questions cropped from the original full-page images. To simulate varying degrees of visual incompleteness, we applied a generative erasure technique to all of the images. Specifically, we quantify the extent of information loss based on the textual density within the visual region. We introduce the Information Loss Rate, denoted as \eta : \eta=\frac{N_{erased}}{N_{total}}\times 100 (11) Consequently, we define the Refusal Rate R_{ref}(\eta) at a specific occlusion level \eta as: R_{ref}(\eta)=\frac{N_{refused}^{(\eta)}}{N_{set}}\times 100\% (12) where N_{refused}^{(\eta)} denotes the count of explicitly refused answers under the occlusion rate \eta , and N_{set} represents the total number of questions. Result Analysis: As illustrated in Figure 7 , we observe a distinct correlation between model scale and sensitivity to visual occlusion. Taking the 50% refusal rate as a critical threshold, the larger Qwen3VL-30B exhibits the highest sensitivity, identifying and refusing compromised inputs at a low information loss rate of \eta=20\% , followed by GLM at \eta=30\% . In contrast, lightweight models like Interns1-8B and Qwen3VL-8B demonstrate lower awareness of information completeness, attempting to answer even until the occlusion reaches \eta=40\% . This indicates that larger models possess a superior perception of visual integrity , effectively detecting missing contexts to ensure response reliability , whereas smaller models are less sensitive to information loss, tending to process the incomplete visual inputs indiscriminately . 5.2.2 Exploration on Strategies for Refusal Enhancement In this subsection, we investigate methods to facilitate the model’s ability to refuse answering when appropriate. Our assumption is that image cropping enhances the model’s visual perception of the specific question. By isolating the question, the model can more easily identify flaws or incompleteness in the visual input, triggering refusal explicitly. Figure 7: The result of refusal performance. Experimental Setup: We designed a comparative experiment with two input modes: • In Full-Page Mode, the complete and original test paper image serves as the input. Under this setting, the model exhibits three distinct behaviors: (1) generate an answer, (2) offer no response , or (3) provide an explicit refusal. • In Single-Question Mode, the specific unrecognizable questions are cropped from the page and input individually. In this scenario, model’s behavior is categorized into two types: (1) generate an answer, or (2) provide an explicit refusal. To evaluate the effectiveness, we categorize the results based on the joint behavior of the model in both modes. Let N=110 denote the total number of unrecognized questions. We define n_{i}^{j} as the number of questions where the model exhibits behavior i in Full-Page , j in Single-Question, where i represents Full-Page behaviors (1: Answer, 2: Silence, 3: Refusal) and j denotes Single-Question behaviors (1: Answer, 2: Refusal). Based on this notation, we propose three metrics to evaluate the performance: • Refusal Failure Rate ( R_{fail} ): This metric measures the proportion of questions where the model persists in generating an answer: R_{fail}=\frac{\sum_{i=1}^{3}n_{i}^{1}}{N} (13) • Mitigation Rate ( R_{mitigation} ): This metric evaluates the effectiveness of the cropping strategy: R_{mitigation}=\frac{n_{1}^{2}+n_{2}^{2}}{N} (14) • Active Refusal Rate ( R_{active} ): This metric reflects the consistency of the model’s refusal mechanism in both Modes. R_{active}=\frac{n_{3}^{2}}{N} (15) Figure 8: Refusal performance comparison under Full-Page and Single-Question input modes. Result Analysis: As illustrated in Figure 8 , in the full-page mode, models exhibit a significant deficiency in active refusal ( R_{active} ) (only 1.82% for Qwen3vl-8b) alongside a high failure rate ( R_{fail} ), indicating a propensity to hallucinate responses rather than decline query. Conversely, upon applying the cropping strategy, we observe a substantial improvement in the models’ ability to refuse incomplete inputs, as evidenced by the mitigation metrics ( R_{mitigation} ). These empirical results strictly validate our assumption: highlighting visual incompleteness through cropping effectively triggers the model’s uncertainty, thereby enabling robust active refusal against incomplete visual contexts. 6 Conclusion This paper addresses the lack of realistic benchmarks for noisy educational documents by introducing MathDoc, a dataset featuring authentic high school mathematics exams with diverse real-world artifacts and “unrecognizable" samples, along with a comprehensive evaluation framework. MathDoc enables systematic assessments of both information extraction and active refusal capabilities, providing crucial insights for advancing model reliability. Its specific focus on refusal boundaries facilitates targeted model optimization, promoting more faithful and robust Large Language Models. Limitations Though we initially explored visual perception enhancement strategies, we have not yet delved into model architecture design or pre-training objective optimization to fundamentally suppress the model’s tendencies for speculative completion and forced transcription in incomplete contexts. Current evaluations serve more as a diagnostic tool rather than a generalizable algorithmic solution. Empowering models with the introspective capability to “know what they do not know” remains a critical challenge for building robust document processing systems. Appendix A Appendix A.1 The example of dataset In this section, we present representative examples from the MathDoc dataset, illustrating unrecognizable questions in Figure 9 . (a) (b) (c) Figure 9: Examples of three categories of unrecognizable cases defined in our work: (i) heavy handwriting occlusions and corrections in critical text regions; (ii) physical obstructions (e.g., stains and wrinkles); and (iii) compromised document integrity, such as incomplete questions or folded pages. Unrecognizable question regions are highlighted with red bounding boxes, as these degradations remove critical semantic information and violate the legibility constraints, thereby marked as [unrecognizable]. A.2 The fairness of LLM as judge We note that our evaluation relies on Large Language Model as automatic judge. Rather than claiming absolute objectivity, we aim to demonstrate that under our constrained evaluation protocol, the judge exhibits consistent and interpretable behavior. First, for image-based verification, the judge is restricted to assessing semantic consistency under a fixed instruction, rather than subjective visual quality. We therefore avoid pixel-level metrics such as IoU, which are highly sensitive to common layout variations in real-world documents and may not reliably reflect semantic validity. As shown in Figure 10 , two images with moderate IoU can still be semantically equivalent and are correctly assigned a high score by the LLM-based judge. In contrast, Figure 11 presents a visually similar prediction that omits critical semantic elements and is accordingly penalized. Second, at the text level, the judge is designed to tolerate common OCR-induced noise while preserving strict semantic alignment between the predicted and ground-truth questions. Figure 12 provides a representative matching example. Figure 10: An example of LLM-based image semantic similarity evaluation Figure 11: A failure case correctly identified by the LLM-based semantic judge A.3 Prompt Details In this section, we detail the prompts used in our evaluation pipeline. For readability, we present the English version of the prompt. The experiments are conducted using the original Chinese prompt, which is semantically equivalent. Specifically, we describe: (i) the prompt used to guide a language model in identifying the start and end indices of the predicted text spans within the generated output (Figure 14 ); (ii) the prompt employed in an MLLM-as-a-judge setting to assess semantic similarity between ground-truth figures and model-predicted cropped images (Figure 13 ); and (iii) the prompt provided to MLLMs for document-level information extraction (Figure 15 ). Figure 12: Illustration of dual-anchor semantic alignment between ground-truth and predicted text. The left hand-side two questions are all from GT, the right hand-side two are all from Pred text stream. Role. You are a Visual Semantic Completeness Evaluator with expertise in the structured assessment of mathematical documents. Your objective is to evaluate the semantic completeness of an Extracted Image (Image 2) relative to its corresponding Ground Truth Image (Image 1) . Task. Given a pair of images, assess whether Image 2 faithfully preserves all semantic information present in Image 1. This includes textual content, mathematical expressions, and graphical elements. 1. Evaluation Procedure • Completeness Check (Critical). Carefully compare Image 2 against Image 1 and determine whether any critical semantic information is missing. This includes, but is not limited to: truncated text, missing vertices or edges in geometric figures, and blurred or omitted superscripts/subscripts in formulas. • Redundancy Check (Secondary). Examine whether Image 2 contains excessive irrelevant background or whitespace. Additional whitespace is acceptable as long as it does not interfere with content readability or interpretation. 2. Scoring Criteria • 0.9 – 1.0 (Perfect / Excellent). Image 2 fully preserves all semantic information from Image 1. • 0.6 – 0.8 (Pass / Minor Flaws). The semantic content is fundamentally complete with negligible defects. • 0.0 – 0.59 (Unusable). Any semantic or structural loss that prevents full reconstruction of the original problem meaning. Figure 13: Prompt used for evaluating the visual similarity of extracted mathematical document images. Role. You are an OCR Text Alignment Expert . Your objective is to map structured questions from the Ground Truth (GT) to their physical boundaries within the noisy Predicted Text (Pred) . Task. Establish a correspondence between clean GT questions and noisy Pred content. For each GT question, identify the exact pred_start_snippet and pred_end_snippet in Pred. 1. Core Principles • Monotonicity (Strict Order). Question order in Pred must strictly follow GT. The start of Question N must appear after the end of Question N-1 . No backtracking or overlap is allowed. • Verbatim Extraction. pred_start_snippet and pred_end_snippet must be copied exactly from Pred, preserving all OCR errors, garbled characters, and missing punctuation. Do not normalize or correct the text. • Physical Truncation Strategy. Do not struggle to search for a semantic ending of Question N . Instead, if the end can not be found, try to locate the start of Question N{+}1 . The end of Question N is defined as the text segment immediately preceding it. • Robustness to Failure. If Question N cannot be matched due to severe noise or missing content, return empty strings and immediately proceed to align Question N{+}1 . Maximize the number of aligned questions. 2. Alignment Strategy For each GT question: • Locate Start. Identify the question beginning in Pred using semantic similarity . Prioritize topic-level matching (e.g., Complex Numbers , Vectors ) even when mathematical expressions are distorted (e.g., z=1+i vs. z=2/(1+i) ). • Ignore Status Tags. Skip metadata or preprocessing tags (e.g., [Format Normal] ) and begin extraction from the actual question content or index. • Locate End. Identify the start of Question N{+}1 . The end of Question N is the text segment strictly before it. Garbled text is acceptable as long as it lies within the physical boundary. 3. Output Schema Return a pure JSON list containing objects with the following fields: {question_id, gt_start_snippet, gt_end_snippet, pred_start_snippet, pred_end_snippet, pred_answer} Extract answers as concise values (e.g., A , 3 , [Answer: X] ). Return empty strings if no answer is found. Do not extract full question text. 4. Few-Shot • Distorted Mathematics. GT: z=2/(1+i) ; Pred: z=1+i . Match based on the shared semantic topic. • Missing Index. GT: 18. Function ... ; Pred: (17pts) Function ... . Match using semantic context despite index loss. • Unrecognizable Content. If the GT marks a question as [Unrecognizable] , inspect the corresponding section in Pred . Consider the question **successfully aligned** if the Pred content is either: 1) Empty (null string), or 2) Explicitly contains unrecognizability indicators (e.g., “unrecognizable", “illegible", or failure tokens). In these cases, extract the empty string or the specific indicator as the snippet and proceed to the next question. Figure 14: Full prompt used for OCR question-to-text boundary alignment under severe noise. Role. You are a Strict Document Structure Analysis Specialist . Your objective is to accurately transcribe the printed structure of exam papers without performing mathematical problem-solving or speculative completion. Global Constraints: (1) No Subjective Solving: Do not solve mathematical problems or reason about correctness. (2) No Hallucination: If printed text is occluded or missing, do not guess or complete it. (3) Strict Mode Switching: Apply transcription rules strictly according to the question type defined below. Workflow: Execute the following logic pipeline sequentially for each detected question region. Step 1: Legibility & Integrity Check. Condition: If any critical printed information (question index, stem content, or specific options) is blurred, occluded, or missing due to folding, stains, or damage. Action: Immediately output: [ID]. [Unrecognizable] and terminate processing for this question. Do not attempt to repair or infer the text. Step 2: Question Mode Classification & Transcription Classify the question into one of the following modes and apply the corresponding strict rules. Mode A: Subjective Questions (Calculation / Proof). Characteristics: High point value (e.g., > 10 points), sub-questions such as (1)(2), or directives like “Prove” or “Solve”. Rules: Transcribe the printed question stem only. Strictly ignore all handwritten content. Mode B: Objective Questions (Multiple Choice / Fill-in-the-Blank). Characteristics: Presence of options (A/B/C/D) or underline placeholders. Rules: • Transcribe all printed options (A, B, C, D) completely. • Answer Extraction: If a handwritten mark (e.g., a tick or letter) is explicitly visible, append [Answer: X] at the end. If the answer area is blank, output printed text only. • Prohibition: Do not infer answers based on internal knowledge. If no handwriting is present, do not generate an answer tag. Step 3: Figure Extraction Target: Computer-generated diagrams only (e.g., geometric figures or function plots). Action: Append an image marker after the question stem using normalized coordinates: where coordinates are normalized to the range [0,1000] . Few-Shot Demonstrations (Illustrative) Example 1: Objective (Answered). Input: Question 5 with handwritten “C” marked in the answer bracket. Output: Let M=\{1,2\},N=\{2,3\} , then M\cup N= ( ) [Answer: C] A. \{1\} B. \{2\} C. \{1,2,3\} D. \{3\} Example 2: Objective (Unanswered). Input: Question 6 with an empty answer bracket. Output: Which of the following functions is an even function? ( ) A. y=x B. y=x^{2} C. y=\sin x D. y=e^{x} ( No answer tag is generated. ) Example 3: Subjective (Handwriting Ignored). Input: Question 17 (10 points) with handwritten formulas covering whitespace. Output: (10 points) As shown in the figure… (1) Prove that… Example 4: Illegible Input (Refusal). Input: Question 8 partially covered by a page fold. Output: [Unrecognizable] Figure 15: Prompt specifying strict rules for document structure analysis and transcription of exam papers.