KazakhOCR: A Synthetic Benchmark for Evaluating Multimodal Models in Low-Resource Kazakh Script OCR
Henry Gagnier, Sophie Gagnier, Ashwin Kirubakaran · 2026
Read KazakhOCR: A Synthetic Benchmark for Evaluating Multimodal Models in Low-Resource Kazakh Script OCR by Henry Gagnier, Sophie Gagnier, Ashwin Kirubakaran (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13238.txt.
Introduction
Henry Gagnier, Sophie Gagnier, and Ashwin Kirubakaran build KazakhOCR, a synthetic benchmark for Kazakh script recognition. Kazakh uses Arabic, Cyrillic, and Latin scripts, yet no OCR benchmark covered two of them. Their dataset holds 7,219 images with font, color, and noise variation across all three scripts. Three multimodal models were tested on OCR and language identification tasks. All failed on Latin and Arabic script OCR, and Arabic script was misread as other languages. A classical OCR baseline beat the multimodal models on character error rates.
Catalog record
| CREATOR | Henry Gagnier, Sophie Gagnier, Ashwin Kirubakaran |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16194 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13238
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KazakhOCR: A Synthetic Benchmark for Evaluating Multimodal Models in Low-Resource Kazakh Script OCR Henry Gagnier, Sophie Gagnier, Ashwin Kirubakaran Abstract Kazakh is a Turkic language using the Arabic, Cyrillic, and Latin scripts, making it unique in terms of optical character recognition (OCR). Work on OCR for low-resource Kazakh scripts is very scarce, and no OCR benchmarks or images exist for the Arabic and Latin scripts. We construct a synthetic OCR dataset of 7,219 images for all three scripts with font, color, and noise variations to imitate real OCR tasks. We evaluated three multimodal large language models (MLLMs) on a subset of the benchmark for OCR and language identification: Gemma-3-12B-it, Qwen2.5-VL-7B-Instruct, and Llama-3.2-11B-Vision-Instruct. All models are unsuccessful with Latin and Arabic script OCR, and fail to recognize the Arabic script as Kazakh text, misclassifying it as Arabic, Farsi, and Kurdish. We further compare MLLMs with a classical OCR baseline and find that while traditional OCR has lower character error rates, MLLMs fail to match this performance. These findings show significant gaps in current MLLM capabilities to process low-resource Abjad-based scripts and demonstrate the need for inclusive models and benchmarks supporting low-resource scripts and languages. 1 Introduction Kazakh is a Turkic language spoken in Kazakhstan, China, Mongolia, Russia, Kyrgyzstan, and Uzbekistan by 16 million people 9 . Kazakh is written in different scripts depending on the geographic and political context. In Central Asia, a Cyrillic script is used for Kazakh, while in Europe, America, and Turkey, a Latin script is used, and in China, Afghanistan, Pakistan, and Iran, an Arabic script is used 7 . The use of these different writing scripts makes Kazakh unique and makes optical character recognition (OCR) particularly complex, considering Cyrillic, Latin, and Arabic scripts. OCR refers to the electronic translation of handwritten, typewritten, or printed text, and its conversion into a machine-readable form 2 . It enables the recognition of text in digital images, scanned documents, and videos, converting images into machine-coded text. OCR has many applications, including document scanning and form processing. Convolutional neural networks (CNNs) and transformer architectures have recently improved OCR performance across many scripts and languages. Multimodal models have been used for OCR, enabling text extraction without segmentation and allowing for greater robustness with noise. Research on these models has primarily focused on high-resource languages and has not been performed with the Kazakh Arabic script. Prior work for Kazakh OCR has been primarily done on the Cyrillic script. 15 created a KOTHD, a Kazakh offline handwritten text dataset using Cyrillic script. 11 developed a handwritten text recognition system for handwritten Cyrillic script using CNNs and RNNs. 10 evaluates a neural network on the recognition of Latin letters of the Kazakh alphabet. 16 use a Generative Adversarial Network (GAN) to improve Kazakh handwriting recognition using Cyrillic script. The Kazakh Arabic script uses additional letters, modified graphemes, and different orthographic conventions, making it different from high-resource Arabic varieties and an important test case for Abjad-based low-resource NLP. While work has been performed on Kazakh OCR, it does not focus on the Arabic script and scarcely focuses on the Latin script. Synthetic datasets are emerging as a solution to a lack of images and content in low-resource languages. The creation of a synthetic OCR dataset often consists of the collection of text corpora, fonts, and noise filters 12 . 5 developed a synthetic benchmark for Pashto and the Pashto script. 8 proposed a method for the creation of synthetic data for Arabic OCR. 12 synthetically created a benchmark of 90k images for 23 Indic languages. Synthetic data has also been used for post-OCR correction, and has been shown to have advantages over traditional OCR training in low-resource languages 4 . Synthetic data is particularly useful when limited resources for OCR are available, such as in low-resource Kazakh scripts. Research on OCR for Kazakh using the Arabic script is scarce, but extremely necessary. The purpose of this study is to (1) construct synthetic OCR benchmarks for low-resource Kazakh scripts, (2) evaluate multimodal models on the OCR of low-resource Kazakh scripts, and (3) identify challenges in the OCR of Kazakh. This study also aims to encourage the inclusion and research of low-resource scripts, particularly languages using the Arabic Kazakh script. 2 Methodology 2.1 Kazakh Text Corpora While it is possible to obtain perfect transliterations in Kazakh using pre-defined rules, cultural differences in the Cyrillic and Arabic scripts make transliteration from high-resource scripts into low-resource scripts sub-optimal for training 18 . We obtained authentic Kazakh text in the Arabic script from the first Kazakh corpus in the Arabic script, the MC² corpus 18 . Wikipedia is also a good resource for low-resource languages such as Kazakh. For Cyrillic text, we obtained authentic data from the Kazakh Wikipedia 1 1 1 https://huggingface.co/datasets/wikimedia/wikipedia . We cleaned the Cyrillic script data from Wikipedia using kaznlp 17 , and transliterated it to the Latin script using the QazNLTK package 2 2 2 https://pypi.org/project/qaznltk/ . As the number of characters in each sample of the Arabic (3746 characters) script text was much greater than the number of characters in the Cyrillic (689 characters) and Latin (718 characters) script texts, we implemented random balanced subsampling to make the text lengths similar in each of the scripts. After balance subsampling, we had 2,417 Arabic texts, 2,402 Cyrillic texts, and 2,400 Latin texts, with similar text length distributions across scripts, which were all used in the final benchmark. 2.2 Benchmark Construction In order to make synthetic images representative of authentic OCR tasks, such as document scanning, we implement many variations across images, using the Pillow Python library. Examples of the benchmark and its variation can be seen in Figure 1 , 2 , and 3 . • Fonts: We downloaded 2,491 fonts from Google Fonts and filtered all fonts for compatibility with Kazakh Cyrillic, Latin, and Arabic script. We found 135 fonts compatible with Arabic, 431 with Cyrillic, and 1376 with Latin. These fonts were randomly selected to create synthetic images. We also varied font size randomly from 24 to 56 across images. • Noise: We set the noise level to be random between 4 and 18, and a blur effect to be randomly between 0 and 0.8. Images are all rotated randomly between -3 and 3 degrees randomly. • Color: We designed two color palettes, each with 40 colors. One contained light background colors, and one contained darker text colors. When these colors were randomly selected, a minimum contrast ratio of 4.5 was used as a standard in all images. To foster reproducibility and encourage future work on low-resource Kazakh scripts, we release the full benchmark, which is available at https://huggingface.co/datasets/henrygagnier/kazakh-ocr . Figure 1: Example of an OCR task in the Kazakh Arabic script Figure 2: Example of an OCR task in the Kazakh Cyrillic script Figure 3: Example of an OCR task in the Kazakh Latin script 2.3 Multimodal Large Language Models Multimodal large language models (MLLMs) preserve the reasoning and decision-making capabilities of large language models (LLMs), but also empower multimodal tasks, such as OCR 19 . We evaluated gemma-3-12b-it 14 , Qwen2.5-VL-7B-Instruct 1 , and Llama-3.2-11B-Vision-Instruct 3 using a subset of the benchmark, with 200 images per script, or a total of 600