OCR-Quality: A Human-Annotated Dataset for OCR Quality Assessment
Yulong Zhang · 2025
Read OCR-Quality: A Human-Annotated Dataset for OCR Quality Assessment by Yulong Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21774.txt.
Introduction
Yulong Zhang introduces OCR-Quality, a dataset built by hand to judge how well computers read text from scanned pages. One thousand PDF pages were turned into images and drawn from academic papers, textbooks, e-books, and documents in several languages, so the material reflects real reading tasks rather than clean test samples.
Each page was processed by current vision-language models and then scored by people on a four-level scale from excellent to poor. The release also gives source details, scoring guidelines, and example cases at different difficulty levels. Because judging OCR quality has lacked a shared benchmark, the dataset gives builders of reading and checking systems a common ground for training and comparison.
Catalog record
| CREATOR | Yulong Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21774
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OCR-Quality: A Human-Annotated Dataset for OCR Quality Assessment Yulong Zhang Abstract We present OCR-Quality , a comprehensive human-annotated dataset designed for evaluating and developing OCR quality assessment methods. The dataset consists of 1,000 PDF pages converted to PNG images at 300 DPI, sampled from diverse real-world scenarios, including academic papers, textbooks, e-books, and multilingual documents. Each document has been processed using state-of-the-art Vision-Language Models (VLMs) and manually annotated with quality scores using a 4-level scoring system (1: Excellent, 2: Good, 3: Fair, 4: Poor). The dataset includes detailed source information, annotation guidelines, and representative cases across various difficulty levels. OCR-Quality addresses the critical need for reliable OCR quality assessment in real-world applications and provides a valuable benchmark for training and evaluating OCR verification systems. The dataset is publicly available at https://huggingface.co/datasets/Aslan-mingye/OCR-Quality . 1 Introduction Optical Character Recognition (OCR) has become a fundamental component in document understanding pipelines, powering applications from digital archiving to knowledge extraction [ 10 , 11 , 5 ] . Despite significant advances in OCR technology through Vision-Language Models (VLMs) [ 1 , 12 , 9 ] , the quality assessment of OCR outputs remains a challenging problem, particularly in real-world scenarios with diverse document types, languages, and formatting complexities. Existing OCR benchmarks [ 8 , 3 , 14 ] primarily focus on measuring average accuracy across standardized test sets, providing limited insight into the reliability of individual predictions. This gap is particularly problematic for downstream applications where low-quality OCR outputs can propagate errors and reduce system reliability. Recent work on Consensus Entropy [ 15 ] demonstrated the importance of uncertainty quantification for OCR quality assessment, highlighting the need for human-annotated ground truth data. OCR-Quality fills this gap by providing: • 1,000 diverse document pages from real-world sources • Human quality annotations with a 4-level scoring system • Multiple languages : Chinese, English, and multilingual content • Various document types : academic papers, textbooks, e-books, educational materials • Detailed metadata : source information, document characteristics • Representative cases : examples across all quality levels This dataset enables researchers to develop and evaluate methods for automatic OCR quality assessment, model selection, and reliability estimation. 2 Dataset Construction 2.1 Data Collection We collected 1,000 PDF pages from diverse real-world sources to ensure broad coverage of document characteristics. All PDF pages were converted to PNG images at 300 DPI resolution to maintain high-quality visual information for OCR processing. Table 1 summarizes the data sources organized by language and document type. Table 1: Sources of the OCR-Quality Dataset E-book Paper Textbook Other ZH zhishilei, zhongwen-zaixian, gift, thomas - by, kps, kmath, zhonggaokao, gaojiaoshe, gaodengjiaoyu, k12 edu platform, jiaoan, zju icicles - EN theeye, physics-andmathstutor, planetebook escholarship, biorxiv, springer, sagepub, scholarworks, psyarxiv, chemrxiv, iopscience, royal-society-publishing, csiro kps, bookboon, california 14sets, scholarworks dev books repository ML renhang, banshujiang - openstax, math - Other - - - coursehero, studypool Language Distribution: • Chinese (ZH) : Educational materials, textbooks, and e-books from various Chinese sources • English (EN) : Academic papers, textbooks, and literature from diverse repositories • Multilingual (ML) : Documents containing mixed languages, mathematical notations • Other : Specialized educational content with diverse formatting 2.2 OCR Processing All documents were processed using Qwen2.5-VL-72B [ 1 ] , a state-of-the-art Vision-Language Model, with the following specialized OCR prompt: OCR Processing Prompt You are a very professional expert at OCR tasks. Please analyze the following image and extract all text content. 1. Ensure that the extracted text matches the original in the image and maintains the original structure. 2. If the image contains two columns, you should extract all text in the left column before you move on to the right. 3. Ignore the headers, but keep the footnotes, title. 4. You could either use LaTeX, KaTeX or Markdown format for math formulas, physics equations, chemical expressions etc. 5. Do not add or modify the original content! This prompt was designed to: • Preserve document structure and layout • Handle multi-column documents appropriately • Support mathematical and scientific notation • Minimize hallucinations and modifications 2.3 Annotation Process 2.3.1 Annotation Interface We developed a custom annotation tool using Gradio to facilitate efficient human evaluation. The interface provides: • Side-by-side display of original image and OCR output • Quality score selection from a 4-level rating system • Progress tracking and navigation controls • Direct jump to specific samples • Automatic saving to ensure data persistence 2.3.2 Scoring Criteria Annotators were instructed to assign scores based on the following 4-level system (lower is better): 1. Score 1 (Excellent) : Near-perfect OCR with minimal or no errors. The prediction matches the image text exactly with no errors or omissions. 2. Score 2 (Good) : Minor errors that do not affect understanding. The prediction is very close to the image text, with only small mistakes (e.g., punctuation, spacing). 3. Score 3 (Fair) : Some noticeable errors but content is still usable. The prediction contains noticeable errors or captures only part of the text, reducing clarity. May include: • Missing paragraphs or sections • Significant character recognition errors • Incorrect mathematical expressions • Layout structure issues 4. Score 4 (Poor) : Significant errors affecting content quality. The prediction is largely incorrect or unrelated to the text in the image. May include: • Severe misrecognition across entire document • Wrong language or gibberish output • Completely missed content • Structural collapse This scoring system balances granularity and ease of annotation while capturing the practical utility of OCR outputs for downstream applications. 3 Dataset Statistics 3.1 Quality Score Distribution Table 2 presents the distribution of documents across quality levels: Table 2: Score Distribution in OCR-Quality Quality Level Score Count Percentage Excellent 1 507 50.7% Good 2 305 30.5% Fair 3 84 8.4% Poor 4 104 10.4% This distribution ensures adequate representation of both high-quality and challenging cases, enabling robust evaluation of quality assessment methods across the difficulty spectrum. 3.2 Document Characteristics Content Types: • Academic papers with complex layouts • Textbooks with mixed text and equations • E-books with varied formatting • Educational materials with diagrams and tables • Multilingual documents Formatting Complexity: • Single and multi-column layouts • Mathematical expressions and formulas • Tables and structured data • Figures with captions • Mixed language content • Various font sizes and styles 4 Representative Cases We present five representative cases from the dataset spanning different quality levels and document types. These cases illustrate the range of challenges in OCR quality assessment. 4.1 Case 1: High-Quality Chinese Text Human Score: 1 (Excellent) This case demonstrates near-perfect OCR performance on well-formatted Chinese academic text. The model correctly captures: • Complex Chinese characters • Document structure • Punctuation and formatting Reference: See Figure 1 in full dataset documentation 4.2 Case 2: Low-Quality English Text Human Score: 4 (Poor) This case shows severe OCR failure on a challenging English document, with: • Extensive character misrecognition • Structural collapse • Missing content Reference: See Figure 2 in full dataset documentation 4.3 Case 3: Academic Chinese Text Human Score: 1 (Excellent) High-quality performance on complex academic Chinese text containing: • Domain-specific terminology • Mixed formatting styles • Proper paragraph structure Reference: See Figure 3 in full dataset documentation 4.4 Case 4: Structured Tabular Content Human Score: 1 (Excellent) Excellent