PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Language

Ijazul Haq, Yingjie Zhang, Irfan Ali Khan · 2025

Read PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Language by Ijazul Haq, Yingjie Zhang, Irfan Ali Khan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10055.txt.

Introduction

Ijazul Haq, Yingjie Zhang, and Irfan Ali Khan test in a 2025 paper how well large AI models read Pashto text from images. Pashto uses a flowing script and lacks the large neat data sets that rich languages enjoy. Reading it by machine is thus a hard test of current models.

The team built PsOCR, a made up data set of one million labeled images. It spans 1,000 font families plus varied colors, sizes, and layouts, with marks at word, line, and page level. A 10,000 image bench set graded seven open models and four closed ones. Gemini scored best of all, and Qwen-7B led the open group. The set should aid work on kindred scripts such as Arabic, Persian, and Urdu.

Catalog record

CREATORIjazul Haq, Yingjie Zhang, Irfan Ali Khan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10055

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Language Ijazul Haq, Yingjie Zhang, Irfan Ali Khan Abstract This paper evaluates the performance of Large Multimodal Models (LMMs) on Optical Character Recognition (OCR) in the low-resource Pashto language. Natural Language Processing (NLP) in Pashto faces several challenges due to the cursive nature of its script and a scarcity of structured datasets. To address this, we developed a synthetic Pashto OCR dataset, PsOCR, consisting of one million images annotated with bounding boxes at word, line, and document levels, suitable for training and evaluating models based on different architectures, including Convolutional Neural Networks (CNNs) and Transformers. PsOCR covers variations across 1,000 unique font families, colors, image sizes, and layouts. A benchmark subset of 10K images was selected to evaluate the performance of several LMMs, including seven open-source models: DeepSeek’s Janus, InternVL, MiniCPM, Florence, and Qwen (3B and 7B), and four closed-source models: GPT-4o, Gemini, Claude, and Grok. Experimental results demonstrate that Gemini achieves the best performance among all models, whereas among open-source models, Qwen-7B stands out. This work provides an insightful assessment of the capabilities and limitations of current LMMs for OCR tasks in Pashto and establishes a foundation for further research not only in Pashto OCR but also for other similar scripts such as Arabic, Persian, and Urdu. PsOCR is available at https://github.com/zirak-ai/PashtoOCR . K eywords OCR \cdot LMMs \cdot VLMs \cdot NLP \cdot Datasets \cdot Benchmarks \cdot Computer Vision Figure 1: Performance Comparison of the top-6 LMMs on Pashto OCR 1 Introduction Optical Character Recognition (OCR) is essential for converting scanned and image-based documents into machine-readable text, underpinning digital archiving, automated indexing, and large-scale document analytics. Traditional OCR engines rely on hand-crafted rules and language-specific resources, while modern deep-learning approaches, using convolutional neural networks and transformer architectures, deliver remarkable performance. However, these methods depend heavily on extensive annotated corpora and lexicons, which are scarce or nonexistent for many languages [ 1 ] . As a result, OCR performance degrades sharply for low-resource languages, highlighting the need for tailored datasets and models. In recent years, the research community has begun addressing these gaps: [ 2 ] has worked on 60 low-resource languages; [ 3 ] has focused on handwritten Tamil, Kurdish, Swahili, and Amharic scripts. Similarly, Ethiopian [ 4 ] , Indic languages [ 5 ] , [ 6 ] and Khmer [ 7 ] have also been explored. Pashto is an Indo-European language of the Perso-Arabic script family, spoken by over 50 million people worldwide. It is the official language of Afghanistan, and the second-largest language of Pakistan by the number of native speakers [ 8 ] . Written in cursive script from right to left (RTL), Pashto comprises 44 letters that can take up to four contextual forms: initial, medial, final, and isolated, which raises considerable challenges for OCR [ 9 ] . Pashto also makes use of ligatures and diacritical marks, expanding its range of glyph shapes [ 10 ] . Furthermore, there are no clear word boundaries in Pashto, and challenges arise from the inconsistent use of diacritical marks that further complicates the training of OCR models to recognize diverse orthographic conventions [ 11 ] , [ 12 ] , [ 13 ] . Synthetic data has emerged as an invaluable resource for addressing data scarcity issues in OCR research, particularly for low-resource languages and scripts. Generating synthetic datasets allows researchers to develop and benchmark robust AI models without the substantial time and resource investments required for manual annotation. This approach has proven effective across diverse languages and scripts; for instance, [ 14 ] demonstrated that incorporating synthetic data alongside real-world data can lead to performance improvements. Similarly, [ 15 ] introduced a comprehensive synthetic dataset for 23 Indic languages, significantly improving accuracy through fine-tuning on synthetic samples. Moreover, [ 16 ] applied synthetic data generation techniques for Arabic OCR, facilitating extensive system evaluation and comparative studies. These studies underscore the importance and effectiveness of synthetic data in enhancing OCR model performance. In this work, we introduce PsOCR, a comprehensive dataset for training and evaluating LMMs on Pashto OCR tasks. PsOCR is composed of one million synthetic images, annotated at word, line, and document levels, featuring around a thousand unique font families, varied color schemes, and diverse layouts. A curated subset of 10K images is used as an evaluation benchmark to examine the performance of various flagship LMMs. We have evaluated a total of eleven models, including seven open-source models: InternVL [ 17 ] , MiniCPM [ 18 ] , Llama [ 19 ] , Janus [ 20 ] , Florence [ 21 ] , and Qwen (3B and 7B) [ 22 ] , and four proprietary models: Grok [ 23 ] , Claude [ 24 ] , GPT-4o [ 25 ] , and Gemini [ 26 ] . Detailed experimental analyses reveal each model’s strengths and weaknesses in Pashto text extraction, paving the way for further research in this domain. The key contributions of this study are as follows: • Introduced the first publicly available comprehensive Pashto OCR dataset consisting of one million synthetic images annotated at word, line, and document-level granularity, covering extensive variations including 1000 unique font families, diverse colors, image sizes, and text layouts. • Developed the first publicly available OCR benchmark comprising 10K images, facilitating systematic evaluation and comparison of OCR systems for the low-resource Pashto. • Conducted a pioneering evaluation and comparison of state-of-the-art LMMs on Pashto OCR, providing crucial insights into their zero-shot capabilities, strengths, and limitations for low-resource languages written in Perso-Arabic scripts. 2 Related Work 2.1 Pashto OCR One of the foundational works in Pashto OCR is [ 10 ] , identifying the primary recognizable units in Pashto script. They suggested “ligature” as the basic unit for OCR and claimed that 7,681 basic shapes were adequate for representing all Pashto ligatures, thus simplifying recognition challenges. [ 27 ] proposed a robust OCR method addressing scale, rotation, and location invariances in Pashto script; they developed a holistic recognition framework and created a dataset of 8K images covering 1K unique ligatures. Recognizing dataset availability as a significant bottleneck, [ 28 ] introduced the KPTI database containing 17K handwritten and printed text-line images, facilitating benchmarking of deep-learning models. [ 29 ] used Sequential Minimal Optimization with horizontal and vertical projections for line, word, and character segmentation and Local Binary Patterns for feature extraction; this model achieved satisfactory performance and emphasized the importance of precise segmentation and robust feature extraction. [ 30 ] leveraged transfer learning with pre-trained CNNs, highlighting the efficacy of fine-tuned deep-learning models augmented by data-enhancement techniques. [ 31 ] developed a CNN-based classifier specifically targeting handwritten numerals, capable of handling variations in style and orientation. Similarly, [ 32 ] and [ 33 ] used CNNs and developed their own benchmark datasets for evaluation. [ 34 ] tested feed-forward neural networks with varying ReLU layers on a custom dataset, providing crucial insights into neural architectures, while [ 32 ] used LSTM for character recognition. [ 35 ] developed the publicly accessible HPCID dataset, comprising 15K handwritten samples, offering critical resources for training and evaluation. Lastly, [ 36 ] created the PHTI dataset, encompassing 36K segmented text-line

Read the complete text