Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documents
Gavin Greif, Niclas Griesshaber, Robin Greif · 2025
Read Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documents by Gavin Greif, Niclas Griesshaber, Robin Greif (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00414.txt.
Introduction
Gavin Greif, Niclas Griesshaber, and Robin Greif test whether image-aware language models can read old city directories. The books were printed in German from 1754 to 1870. Their 2025 paper sets the models three tasks: raw transcription, fixing flawed machine reads, and pulling out named people and places. Off-the-shelf tools were used, with no fine-tuning.
The best model read pages far more cleanly than standard OCR tools. A new trick stood out: feeding a flawed read back to the model for repair. That step drove error rates under one percent. The models also sorted names and facts into neat dataset forms. The team sees early proof that such models could reshape how archives turn old print into data.
Catalog record
| CREATOR | Gavin Greif, Niclas Griesshaber, Robin Greif |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-00414
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documents Gavin Greif, Niclas Griesshaber, Robin Greif Abstract We explore how multimodal Large Language Models (mLLMs) can help researchers transcribe historical documents, extract relevant historical information, and construct datasets from historical sources. Specifically, we investigate the capabilities of mLLMs in performing (1) Optical Character Recognition (OCR), (2) OCR Post-Correction, and (3) Named Entity Recognition (NER) tasks on a set of city directories published in German between 1754 and 1870. First, we benchmark the off-the-shelf transcription accuracy of both mLLMs and conventional OCR models. We find that the best-performing mLLM model significantly outperforms conventional state-of-the-art OCR models and other frontier mLLMs. Second, we are the first to introduce multimodal post-correction of OCR output using mLLMs. We find that this novel approach leads to a drastic improvement in transcription accuracy and consistently produces highly accurate transcriptions ( < < 1% CER), without any image pre-processing or model fine-tuning. Third, we demonstrate that mLLMs can efficiently recognize entities in transcriptions of historical documents and parse them into structured dataset formats. Our findings provide early evidence for the long-term potential of mLLMs to introduce a paradigm shift in the approaches to historical data collection and document transcription. August 24, 2026 1 Introduction The rate at which researchers can collect historical evidence is constrained by their ability to access the relevant sources, accurately transcribe them, and extract the desired information. While a global effort to scan historical sources is gradually reducing access barriers, the need for manual labor in the transcription and extraction of historical information remains a major bottleneck. To remedy this, researchers have come up with a wide range of (semi-)automated solutions to transcribe images of historical documents and extract the desired information from the generated transcriptions. However, existing solutions are either inaccurate or require significant technical expertise and manual annotations. In this paper, we introduce a new approach, which addresses these existing problems by leveraging multimodal Large Language Models (mLLMs). Our approach accurately transcribes historical documents and efficiently parses the information they contain into structured datasets. Since it is impossible to exhaustively benchmark our approach for all languages, fonts, and layouts found in historical sources in a single paper, we focus our attention on historical Latin-alphabet prints. While this may seem restrictive, by 1800, already over 1.7 million books that fall into this category had been published across Europe [ 1 ] . The rates of book production further increased across the nineteenth and twentieth centuries, and by 1910, Germany alone was publishing well over 30,000 new titles every year, almost as many as France, England, and the US combined [ 2 ] . Importantly, Latin-alphabet Europe was divided until the mid-twentieth century, with German-speaking and Lutheran Europe predominantly using Blackletter fonts (e.g., Fraktur) and the rest using Roman fonts (e.g., Antiqua) [ 3 ] . While automating the accurate transcription is undoubtedly easier for printed than handwritten texts, historical typefaces, especially Blackletter ones, have posed a significant challenge to automating historical document transcription [ 4 ] . To demonstrate how exactly our generalized approach outperforms existing solutions, we use a corpus of thirty pages drawn from city directories published across German-speaking Europe between 1754–1870. City, trade, and business directories are an important source for economists, historians, and genealogists, as they offer individual-level micro-data for periods and places where alternative sources (such as censuses) are not available [ 5 ] . For German-speaking Europe alone, well over 40,000 surviving directories, each containing thousands of entries, have been located [ 6 ] . Yet, this cornucopia of historical data has, thus far, not been economic to transcribe. The thirty pages in our corpus include the typical challenges associated with the machine transcription of historical printed texts, including multiple historical typefaces, excess visual information in the scans, skewed texts, handwritten marginalia, discolored backgrounds, print errors, different historical fonts on the same page, and various visual artifacts. All pages in the corpus were manually transcribed and corrected several times to generate the ground truth for the evaluation of our approach. We make three contributions in this paper. First, we benchmark the transcription capabilities of mLLMs. We compare the transcription performance of two leading mLLMs (Gemini 2.0 Flash; GPT-4o) against the leading off-the-shelf models for nineteenth century historical prints (Transkribus’ Text Titan I and Print M1) and a widely used conventional OCR engine (Tesseract 5.5.0, "deu_frak"). We find that in a one-shot setting, Gemini 2.0 Flash achieves the highest accuracy out of all tested models and that its off-the-shelf transcription accuracy is comparable to that of corpus-fine-tuned OCR models reported in the literature for similar documents. Second, we are the first to introduce mLLM OCR Post-Correction for historical texts. We show that by combining the original source image and a noisy transcription, mLLMs can substantially increase the accuracy of OCR output, thus demonstrating that, by focusing on text-only post-correction, existing research has missed the full potential of mLLMs. Finally, we explore the capabilities of mLLMs in recognizing entities in our irregularly structured historical sources. We provide early evidence indicating that mLLMs offer an accessible and efficient solution to identifying and parsing historical information. We find that, while a direct parsing of information from images into the dataset is possible, separating out the steps and including mLLM-based OCR Post-Correction yields the highest accuracy. We release all code and data used in this paper. 1 1 1 Available at: https://github.com/niclasgriesshaber/llm_historical_dataset_benchmarking.git and https://github.com/niclasgriesshaber/gemini_historical_dataset_pipeline.git 2 Related Work 2.1 The Traditional OCR Pipeline While research on automating the transcription of historical documents is vast, existing research typically focuses on one or multiple parts of what we call the traditional pipeline . This standard workflow follows a sequential process, consisting of separate pre-processing, layout recognition, character recognition, and post-processing stages. Depending on the project, a separate Named Entity Recognition (NER) stage may be employed to classify information contained in the post-processed transcription. While existing work on this traditional pipeline is too vast to be discussed here in full, a brief review of it is necessary to highlight the challenges of automating historical document transcriptions, the existing approaches to resolving them, and the way in which mLLMs have been used in this context. The first step of the traditional pipeline is the pre-processing of the images of the historical source. If present, excess visual information, such as parts of another page, the book frame, dark spaces, or an archive footer, are removed during this step. Moreover, the image may be de-skewed and processed through denoising and gray-scaling or binarization to remove unwanted artifacts and increase the contrast between the text and the page background [ 7 , 8 , 9 ] . For cases in which images are degraded, up-sampling has also been applied during pre-processing [ 10 ] . Despite many OCR engines having built-in pre-processing which automatically carries out some of these image manipulations and some researchers using tools such as You Only Look Once (YOLO) to automate some of the pre-processing [ 11 ] , recent scholarship continues to rely on extensive tool-assisted manual pre-processing of source images [ 12 ] . The second step in the traditional pipeline is the recognition of the document layout. This involves,