Finetuning Vision-Language Models as OCR Systems for Low-Resource Languages: A Case Study of Manchu

Yan Hon Michael Chung, Donghyeok Choi · 2025

Read Finetuning Vision-Language Models as OCR Systems for Low-Resource Languages: A Case Study of Manchu by Yan Hon Michael Chung, Donghyeok Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06761.txt.

Introduction

Yan Chung and Donghyeok Choi build a text reader for the Manchu language in a 2025 paper. Manchu is critically endangered, yet its records are key to early modern East Asian history. No strong optical character recognition tools existed for real Manchu documents.

The team fine-tuned three open vision-language models on sixty thousand synthetic Manchu word images. The largest model read synthetic words with over ninety-eight percent accuracy. More telling, it kept over ninety-three percent accuracy on real handwritten pages. An older method aced synthetic tests but fell to about seventy-two percent on real documents. The approach runs on modest hardware, which lowers the bar for archives and scholars.

Catalog record

CREATORYan Hon Michael Chung, Donghyeok Choi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06761

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Finetuning Vision-Language Models as OCR Systems for Low-Resource Languages: A Case Study of Manchu Yan Hon Michael Chung, Donghyeok Choi Abstract Manchu, a critically endangered language essential for understanding early modern Eastern Eurasian history, lacks effective OCR systems that can handle real-world historical documents. This study develops high-performing OCR systems by fine-tuning three open-source vision-language models (LLaMA-3.2-11B, Qwen2.5-VL-7B, Qwen2.5-VL-3B) on 60,000 synthetic Manchu word images using parameter-efficient training. LLaMA-3.2-11B achieved exceptional performance with 98.3% word accuracy and 0.0024 character error rate on synthetic data, while crucially maintaining 93.1% accuracy on real-world handwritten documents. Comparative evaluation reveals substantial advantages over traditional approaches: while a CRNN baseline achieved 99.8% synthetic accuracy, it suffered severe degradation to 72.5% on real documents. Our approach demonstrates effective synthetic-to-real domain transfer, providing a cost-effective solution deployable on accessible infrastructure. This work establishes a transferable framework for endangered language OCR that removes technical and financial barriers in digital humanities, enabling historians and linguists to process historical archives without specialized computing resources. 1 1 1 Code and model weights are available at https://github.com/mic7ch1/ManchuAI-OCR K eywords Manchu \cdot OCR \cdot Vision-Language Model \cdot Low Resource Language 1 Introduction In 1993, Pamela Kyle Crossley and Evelyn S. Rawski published a seminal article in the Harvard Journal of Asiatic Studies titled “A Profile of the Manchu Language in Ch’ing History”, in which they argued that the Manchu language and its archival sources are essential for a comprehensive understanding of Qing political, administrative, cultural, and frontier histories. The Qing, the last imperial dynasty of China, was ruled by the Manchus, an ethnic minority whose now critically endangered native language, Manchu, served as both a court and administrative language of the empire. Challenging the long-standing assumption that Manchu-language materials merely duplicated their Chinese counterparts, Crossley and Rawski demonstrated that many Manchu documents contained unique content—particularly in sensitive political, military, and ideological contexts [ 1 , p. 102] . With growing access to Manchu archival materials, they anticipated that “the relevance of the language (Manchu) to any particular category of research (of Qing history) is not as predictable as it was once assumed to be”. This call for a methodological shift was reinforced in Rawski 1996 ’s 1996 presidential address to the Association for Asian Studies, where she highlighted how a Manchu-centered perspective fundamentally altered prevailing narratives of the Qing. [ 2 , pp. 829–830] Together, these interventions catalyzed the emergence of what is now known as the “New Qing History”, a scholarly approach that repositions the Qing as a multiethnic empire rather than a purely sinicized Chinese dynasty. That said, the language barrier has remained a major obstacle to the broader utilization of the vast Manchu archival corpus. Proficiency in Manchu remains limited to a small circle of specialists, and only a fraction of the available sources has been systematically catalogued or studied. Recent advances in Vision Language Models (VLMs) offer new possibilities for expanding access to these materials. This study introduces a high-performing OCR system for the Manchu language, developed by fine-tuning the Llama-3.2-11B-Vision model on a synthetically generated dataset comprising 60,000 Manchu word images. The resulting system achieves a remarkably low character error rate (CER) of 0.0024 and a word-level accuracy of 98.3% on synthetic data, while effectively maintaining a CER of 0.0219 and word-level accuracy of 93.1% on real-world handwritten Manchu word images. This proposed Manchu OCR system not only facilitates the reading of Manchu-language texts but also helps promote their wider use in historical research. Additionally, this study presents a simple and transferable framework which requires only modest synthetic data and computing resources for building robust OCR systems for low-resource and endangered languages. The recent advances in VLM AI thus open promising opportunities for the preservation and revitalization of many other historically significant, low-resource, and endangered languages that remain underserved by existing OCR technologies. 2 Literature Review 2.1 Earlier OCR Works Efforts to develop an Optical Character Recognition (OCR) system for the Manchu language began in the mid-2000s. Early research focused on recognition of primitive Manchu characters at the character level. They proposed a stroke-based method that decomposed handwritten Manchu characters into basic stroke elements, matched these to a predefined stroke library and reassembled them into Manchu-Roman code using fuzzy string matching [ 3 ] . However, this approach was constrained by the complexity of stroke combination rules and the high variability of handwriting styles, which led to recognition errors in cases of distorted or noisy inputs. Subsequent efforts continued along the character-level trajectory, developing recognition systems that extracted and classified Manchu Character Units (MCUs), a sub-character structural unit [ 4 , pp. 802-803] [ 5 , pp. 3339-3340] . While these models introduced some advances, they still relied on manual segmentation and struggled with compounded character forms and the lack of spacing between letters in the vertical Manchu script. A significant methodological shift occurred around 2017 with the adoption of deep neural network architectures, particularly convolutional neural networks (CNNs). Researchers began to move away from character-level segmentation and instead adopted word-level, segmentation-free recognition frameworks. [ 6 , pp. 46-49] Building on this direction, researchers further improved word-level recognition accuracy by training models on large synthetic Manchu word datasets, generated through augmentation techniques such as rotation, blur, and elastic distortion. These datasets were derived from 2,135 word images representing 666 unique Manchu words extracted from the textbook 365 Sentences in Manchu, with each word class expanded into 1,000 variants to create a corpus of 666,000 samples. These CNN models could achieve a peak accuracy of 97.68% and an average accuracy above 91% in printed fonts. [ 7 , 8 ] A subsequent study conducted in 2021 adopted a similar segmentation-free approach using a deep CNN architecture and a sliding window mechanism. Their system was trained on a slightly expanded dataset of 671 classes, also with 1,000 samples per class, and achieved a peak accuracy of 98.84%. [ 9 , pp. 5-6] However, it is important to note that all these models were trained and evaluated on variations of a closed set of fewer than 700 Manchu words, and their performance on unseen vocabulary remains untested. Consequently, while these studies demonstrate strong performance in controlled environments, their generalizability to the full range of historical Manchu texts has yet to be established. Beyond academic research, several institutions and individuals have also developed practical OCR systems for Manchu. The First Historical Archives of China in Beijing has reportedly created an OCR system tailored for scanned handwritten Manchu documents. However, as the system is restricted to internal institutional use, its technical specifications and recognition accuracy remain unavailable for public evaluation. In 2022, Zhang Zhuohui released an open-source OCR tool for Manchu developed in Python on GitHub [ 10 ] . While the system demonstrates promising functionality in processing printed Manchu texts, no formal assessment of its recognition accuracy has yet been published. More recent research has explored the integration of linguistic understanding and visual features to overcome the limitations of purely visual OCR approaches in low-resource scripts. The most advanced contribution to date is the work of Wang et al. 2024 ’s team, which introduced the Visual-Language framework for Manchu word Recognition (VLMR). The model fuses visual features with semantic reasoning carried out

Read the complete text