Multimodal LLMs for Historical Dataset Construction from Archival Image Scans: German Patents (1877–1918
Niclas Griesshaber, Jochen Streb · 2025
Read Multimodal LLMs for Historical Dataset Construction from Archival Image Scans: German Patents (1877–1918 by Niclas Griesshaber, Jochen Streb (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19675.txt.
Introduction
Niclas Griesshaber and Jochen Streb turned multimodal language models loose on a mountain of German patent records. Their source material is 9,562 archival page scans holding 306,070 patents from 1877 to 1918, each page packed with twenty to fifty entries in double columns and mixed Gothic and Roman type. A pipeline built on Gemini models read the scans and structured the contents into a usable dataset.
Benchmarking against human research assistants brought a surprise: the models produced higher quality data, hundreds of times faster and cheaper. The authors open-source the datasets, benchmark, and pipeline so other historians can adapt them, then discuss the economics of this workflow. They present it as a paradigm shift for dataset construction in economic history.
Catalog record
| CREATOR | Niclas Griesshaber, Jochen Streb |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-19675
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal LLMs for Historical Dataset Construction from Archival Image Scans: German Patents (1877–1918) Thanks: Paper website: https://historymind.ai Thanks: Code: https://github.com/niclasgriesshaber/llm_patent_pipeline.git Thanks: Archival Image Corpus: https://digi.bib.uni-mannheim.de/sammlungen/patentregister Niclas Griesshaber, Jochen Streb Abstract We leverage multimodal large language models (LLMs) to construct a dataset of 306,070 German patents (1877–1918) from 9,562 archival image scans using our LLM-based pipeline powered by Gemini-2.5-Pro and Gemini-2.5-Flash-Lite. Our benchmarking exercise provides tentative evidence that multimodal LLMs can create higher quality datasets than our research assistants, while also being more than 795 times faster and 205 times cheaper in constructing the patent dataset from our image corpus. About 20 to 50 patent entries are embedded on each page, arranged in a double-column format and printed in Gothic and Roman fonts. The font and layout complexity of our primary source material suggests to us that multimodal LLMs are a paradigm shift in how datasets are constructed in economic history. We open-source our benchmarking and patent datasets as well as our LLM-based data pipeline, which can be easily adapted to other image corpora using LLM-assisted coding tools, lowering the barriers for less technical researchers. Finally, we explain the economics of deploying LLMs for historical dataset construction and conclude by speculating on the potential implications for the field of economic history. K eywords Multimodal Large Language Models \cdot Information Extraction \cdot Dataset Construction \cdot German Patents 1 Introduction For a long time, economic historians focused on compiling and interpreting long macroeconomic time series, with GDP per capita playing the most prominent role ( 42 ; 7 ) . This inevitably led to methodological limitations, as microeconomic data is needed to understand how consumers and producers react to economic policy changes or exogenous shocks. In particular, it is important to consider the heterogeneity of historical actors. People differ in terms of age, gender, education, social status, income, and wealth, among other things, and depending on these differences, they make different decisions under the same external conditions. In order to take this heterogeneity into account, large-scale microeconomic datasets are required. The fact that economic historians rarely use microeconomic data are not due to their inaccessibility. Socioeconomic data on consumers can be obtained, for example, from historical census data ( 40 ; 50 ; 1 ) , savings books ( 38 ) , information on companies from patent statistics ( 19 ) , stock market newspapers ( 37 ) , city directories ( 2 ) , or trade registers ( 27 ) . However, accessing these data was associated with high and sometimes prohibitive costs because the manual construction of the dataset by research assistants was slow, expensive, and error-prone. To accelerate historical dataset construction, methods from the digital humanities and computer science, such as optical character recognition (OCR), named-entity recognition (NER), or information extraction (IE), have become increasingly attractive within the field of economic history ( 51 ; 15 ; 6 ) . However, these digital methods require advanced programming and labeled data to build custom pipelines that work only on homogeneous image corpora. Recent advances in artificial intelligence (AI) have the potential to remove these barriers and make large-scale dataset construction accessible to non-technical researchers, thereby eliminating the field’s major bottleneck: transforming archival image scans into a structured dataset. Multimodal large language models (LLMs) are neural networks that can jointly process multiple types of data. For example, in this paper we send images and raw text instructions to multimodal LLMs to construct a large-scale historical patent dataset. Multimodal LLMs have shown remarkable performance across a wide range of tasks, including language and vision ( 16 ; 8 ; 20 ) . They are—like many of the existing methods from OCR, NER, and IE—based on the deep learning paradigm, which refers to neural networks with many layers that can learn complex patterns from data ( 36 ; 22 ) . When LLMs are scaled with data, parameters, and computing capacity, they become surprisingly powerful ( 32 ; 29 ; 57 ) , offering a unified approach to OCR, NER, and IE. These immense economies of scale are also the reason why the most capable LLMs are almost exclusively developed by private AI companies. The advancements and commercialization of multimodal large language models have enabled us to make several contributions. First, to construct a dataset that contains all 306,070 patent entries that are included in 41 physical volumes of historical patent registers published between 1877 and 1918, we used LLM-assisted coding tools to develop an LLM-based data pipeline that processed our 9,562 image scans. We then extracted five variables from each patent entry: patent_id , assignee , location , title , and date . Moreover, our research assistants manually created a student-constructed benchmarking dataset from 41 randomly selected images, one from each of the annual patent registers used. Initially, we compared the LLM-generated dataset against the student-constructed dataset to evaluate the reliability of our LLM-based pipeline—especially to check for hallucinations that would undermine the data quality and bias any subsequent econometric analysis. Surprisingly, we realized that many highlighted differences between both datasets were not due to errors made by the LLM, but due to mistakes made by our research assistants. For this reason, we also constructed a second, perfect benchmarking dataset. By comparing the student-constructed and LLM-generated dataset to the perfect dataset, we provide tentative evidence that multimodal LLMs are able to produce higher quality datasets from our image corpus than our research assistants. Finally, we open-source both our LLM-based data pipeline and the new historical patent dataset to accelerate empirical research in economic history. This paper is structured as follows: Section 2 provides the historical background of our data. Section 3 then reviews related work on the construction of patent databases, including manual work, custom machine learning approaches, and the emerging literature on LLMs in economic history. Section 4 describes our primary sources that form our archival image corpus. Section 5 presents our LLM-based data pipeline to construct the patent dataset, followed by Section 6 which addresses data quality and hallucination concerns using our benchmarking datasets. Section 7 explains the economics of constructing datasets from image scans using multimodal LLMs. Section 8 concludes and discusses the potential implications of multimodal LLMs for the field of economic history. 2 Historical Background Modern economic growth, which began in Great Britain in the early nineteenth century and has since been emulated by many countries around the world, is based on a never-ending stream of innovation. A central question in economic history is therefore why it took so long for innovations to evolve from a rare occurrence to a constant driver of growth ( 21 ; 44 ) . To understand which factors promote and hinder the emergence of innovation, it is first necessary to measure the type and scope of innovations as precisely as possible. In economic history research, this is done primarily on the basis of patent statistics. The origins of the patent system date back at least to mid-fifteenth century Venice. However, it was not until the end of the eighteenth century that patents evolved from a privilege into an intellectual property right that could be obtained by registration with a patent authority. All patent authorities charged a filing fee for granting a patent;