Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records
Seorin Kim, Julien Baudru, Wouter Ryckbosch, Hugues Bersini, Vincent Ginis · 2025
Read Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records by Seorin Kim, Julien Baudru, Wouter Ryckbosch, Hugues Bersini, Vincent Ginis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11623.txt.
Introduction
Seorin Kim and four co-authors test large language models on historical handwriting in a paper from two thousand twenty-five. Archives hold vast handwritten records in tables, and traditional recognition tools often struggle with them. The team pitted two leading models against four conventional systems.
Documents were fed line by line or as whole scans. On character error and match scores, the language models beat every traditional tool. Human judges then rated the whole-scan outputs to check what the scores miss. Two-shot prompting worked best: one model led on line images, the other on whole scans. The authors present this as early evidence that general models now out-transcribe specialist software on historical records.
Catalog record
| CREATOR | Seorin Kim, Julien Baudru, Wouter Ryckbosch, Hugues Bersini, Vincent Ginis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-11623
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records Seorin Kim, Julien Baudru, Wouter Ryckbosch, Hugues Bersini, Vincent Ginis Abstract We explore the ability of two LLMs – GPT-4o and Claude Sonnet 3.5 – to transcribe historical handwritten documents in a tabular format and compare their performance to traditional OCR/HTR systems: EasyOCR, Keras, Pytesseract, and TrOCR. Considering the tabular form of the data, two types of experiments are executed: one where the images are split line by line and the other where the entire scan is used as input. Based on CER and BLEU, we demonstrate that LLMs outperform the conventional OCR/HTR methods. Moreover, we also compare the evaluated CER and BLEU scores to human evaluations to better judge the outputs of whole-scan experiments and understand influential factors for CER and BLEU. Combining judgments from all the evaluation metrics, we conclude that two-shot GPT-4o for line-by-line images and two-shot Claude Sonnet 3.5 for whole-scan images yield the transcriptions of the historical records most similar to the ground truth. K eywords Large Language Models ⋅ \cdot Historical Documents ⋅ \cdot Handwritten Text Recognition ⋅ \cdot Optical Character Recognition ⋅ \cdot Character Error Rate 1 Introduction Digitalizing historical documents has been a tedious task for many governments and has allowed research opportunities for many. Besides manual efforts, optical character recognition (OCR) and handwritten text recognition (HTR) models have been the mainstays of digitizing historical documents. Given the increasingly expanding capabilities of the large language models (LLMs), we aim to study how much of the transcription process can be facilitated by the sole use of LLMs compared to that of the traditional OCR/HTR pipelines . For this study, we focus on a tabulated historical record where the table’s header is filled with optical characters while the rest contains handwritten texts. Historical records are more challenging to transcribe than modern documents due to cursive handwriting styles, the degraded quality of the texts (e.g., faded inks or damaged paper) , language changes, and document layouts. The current OCR/HTR pipelines typically begin by pre-processing scans of the documents by adjusting the contrast and color of the image and cropping if necessary. Then, they analyze their layout and segment and then recognize the texts. In addition, layout analysis/segmentation and text recognition often accompany fine-tuning the user’s data set. Unlike the three-step process of the classical OCR/HTR pipelines, LLMs can provide a one-step solution from the user’s view, where only the final outputs need to be checked. Yet, their performability compared to the conventional methods remains unexplored . In this study, we aim to compare LLMs and OCR/HTR tools in terms of how accurately they reproduce the texts in the scanned images of the historical records. Note that our goal is to replicate the texts in the image as they are, not to understand and guess what the information should be in the table. We utilize Character Error Rate (CER) and Bilingual Evaluation Understudy (BLEU) to evaluate the performance of the models: GPT-4o, Claude Sonnet 3.5, EasyOCR, Keras, Pytesseract, and TrOCR. The study also compares different strategies to enhance LLMs’ transcription performance and two TrOCR variants with fine-tuning. 2 Similar work In [ 3 ] , the authors introduce a new method for text recognition called Decoder-only Transformer for Optical Character Recognition (DTrOCR). DTrOCR uses only a decoder, using a pre-trained generative language model, in contrast to traditional encoder-decoder methods [ 14 ] . The authors tested whether a successful natural language processing model could be applied to text recognition in computer vision. Their experiments showed that DTrOCR significantly outperformed current state-of-the-art methods in recognizing printed, handwritten, and scene text in both English and Chinese. In [ 6 ] , the author introduces a method to digitize over 100,000 historic plans from the Swiss Archive for Landscaping Architecture using AI models. The approach employs a three-model architecture: a layout model to identify text, an OCR model to extract words, and a named entity recognition (NER) model to label key information. K-means clustering groups text blocks for OCR processing. Various deep-learning models were evaluated, including German BERT for NER, and retrained on the NVIDIA DGX-2 system. The pipeline achieved an F1 score of 48%, with the NER model scoring 86% and the OCR model correctly extracting 54% of words. In [ 2 ] , the authors propose a novel tokenized representation of digital ink for online handwriting recognition, addressing the shortcomings of naive OCR with vision-language models (VLMs). Integrating stroke sequences and images, this approach achieves state-of-the-art quality on three public datasets. Their findings show that VLMs benefit from multimodal inputs, that images are crucial when text representations are too long, and that multiple handwriting tasks can be combined effectively. The method is compatible with both parameter-efficient tuning and fine-tuning, suggesting future exploration of various handwriting task combinations in large VLMs. The following studies have adopted LLMs in their OCR/HTR process as an assistant to correct the outputs after OCR. In [ 12 ] , the authors address the challenge of poor OCR quality in digitized historical documents, which is a barrier to humanities research. Traditional post-OCR correction methods use sequence-to-sequence models. Instead, the authors propose using generative language models with a prompt-based approach. They demonstrate significant improvements in OCR error correction by tuning Llama 2 with prompts and comparing it to a fine-tuned BART model on 19th-century British newspaper articles. Llama 2 achieves a 55 % reduction in the CER, outperforming BART’s 23 % reduction. This approach shows promise for researchers in improving the accessibility of historical texts by an LLM. Similarly, in [ 1 ] , the authors conducted a comparative study of the ability of fourteen LLMs to correct transcriptions produced using OCR, HTR, and ASR. They then evaluate these corrections by comparing them with ground truths from each document. They conclude that, although GPT-4 appears to be the best model among those tested, all the models degrade rather than improve transcriptions. And that, on the whole, LLMs are better at detecting errors than at correcting them, as they are subject to overcorrection. While these studies have explored different methods to improve OCR systems, to our knowledge, not many studies have used LLMs as a main transcription tool. [ 10 ] was an earlier adopter in exploring the capabilities of GTP-4 Vision in several OCR and HTR tasks for English and Chinese datasets, and [ 4 ] used German typed text datasets to explore the same model for OCR tasks. However, since the experiments were conducted at the early stage of GPT-4 Vision, the authors in [ 10 ] used the web-based dialogue interface for the experiments, hindering them from examining different strategies, including prompting, few shots, and fine-tuning. Also, [ 4 ] did not explore different strategies, and the experiment was limited to reading typed texts rather than handwritten texts. Therefore, our contribution lies in exploring the pure performability of LLMs in transcribing historical records , and different strategies to improve their performance. 3 Methodology 3.1 Dataset We use Belgian probation data from 1921. The dataset includes 20 scanned pages of Déclaration de Succession from Nivelles, a French-speaking city in Belgium. The main language of the record is French, but