Open Korean Historical Corpus: A Millennia-Scale Diachronic Collection of Public Domain Texts
2025
Read Open Korean Historical Corpus: A Millennia-Scale Diachronic Collection of Public Domain Texts (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24541.txt.
Introduction
Seyoung Song and seven co-authors built a large open set of Korean texts from the past. Their corpus spans about thirteen hundred years and six tongues, with texts from the seventh century to 2025. It also holds rare scripts, such as Idu and mixed Hanja and Hangul forms.
The set holds 17.7 million texts and 5.1 billion words drawn from nineteen sources. With it, the team tracks big shifts in written Korean over time. They show Idu use peaked in the 1860s, Hangul rose fast from about 1890, and North Korean word choice now stands apart.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24541
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Open Korean Historical Corpus: A Millennia-Scale Diachronic Collection of Public Domain Texts Abstract The history of the Korean language is characterized by a discrepancy between its spoken and written forms and a pivotal shift from Chinese characters to the Hangul alphabet. However, this linguistic evolution has remained largely unexplored in NLP due to a lack of accessible historical corpora. To address this gap, we introduce the Open Korean Historical Corpus, a large-scale, openly licensed dataset spanning 1,300 years and 6 languages, as well as under-represented writing systems like Korean-style Sinitic (Idu) and Hanja-Hangul mixed script. This corpus contains 17.7 million documents and 5.1 billion tokens from 19 sources, ranging from the 7th century to 2025. We leverage this resource to quantitatively analyze major linguistic shifts: (1) Idu usage peaked in the 1860s before declining sharply; (2) the transition from Hanja to Hangul was a rapid transformation starting around 1890; and (3) North Korea’s lexical divergence causes modern tokenizers to produce up to 51 times higher out-of-vocabulary rates. This work provides a foundational resource for quantitative diachronic analysis by capturing the history of the Korean language. Moreover, it can serve as a pre-training corpus for large language models, potentially improving their understanding of Sino-Korean vocabulary in modern Hangul as well as archaic writing systems. Keywords: Korean, Hanja, Diachronic corpus, Historical linguistics, Mixed script, Low-resource languages Seyoung Song 1 Nawon Kim 2 Songeun Chae 1 Kiwoong Park 1 Jiho Jin 1 Haneul Yoo 1 Kyunghyun Cho 3 Alice Oh 1 1 KAIST 2 Korea University 3 New York University [email protected] , [email protected] , { songeun , marspak , jinjh0123 , haneul.yoo }@kaist.ac.kr, [email protected] , [email protected] Abstract content 1. Introduction Written communication on the Korean peninsula before the 20th century is linguistically compelling due to the inherent discrepancy between its spoken language and writing systems. Prior to the creation of the native alphabet, Hangul (한글), Koreans borrowed Hanja (漢字) and used Classical Chinese as the primary literary language. Because the grammatical structure of Korean (SOV) differs from that of Classical Chinese (SVO), unique transcription systems like Idu were developed to better reflect native grammar Handel (2019) . Even after the invention of Hangul in the 15th century, a centuries-long transition involving mixed script usage preceded the predominantly Hangul-based writing of today. The very success of this transition, however, has rendered centuries of Hanja-based records illegible to most contemporary Koreans, making its accessibility a significant challenge for modern NLP. Figure 1: Overview of Open Korean Historical Corpus, which covers major languages and distinct writing systems used on the Korean peninsula. The abbreviations for each source are explained in Table 2 . The blue cells indicate the corpora that we organized and released for the first time under open license terms. Period Language Text 19c– Modern Korean (Hangul) Not rendered in arXiv But it was already several years ago. 20c– North Korean (Hangul) Not rendered in arXiv I won’t eat chicken dishes. Let’s eat vegetable dishes. 19c– Modern Korean (Hanja-Hangul Mixed Script) Not rendered in arXiv Chart-Based Method for Interpreting Classical Chinese 19–20c Japanese (Colonial Era) Not rendered in arXiv It also seemed that they greatly won the hearts of the people. 17–19c Early Modern Korean (Hanja-Old Hangul Mixed Script) Not rendered in arXiv Frightened, they fled, and prevention was impossible. We respectfully request your examination and action. 17–19c Early Modern Korean (Old Hangul) Not rendered in arXiv I’m busy, so I’ll leave it at this. 10–16c Middle Korean (Old Hangul) Not rendered in arXiv How are you? As I cannot hear news from afar these days, my worries are endless. 7–20c Korean-style Sinitic (Idu) Not rendered in arXiv The matter is hereby submitted to the authorities for review and determination. –20c Classical Chinese (Hanja) Not rendered in arXiv On Gyeongjin day in the 8th month, a red and yellow lunar halo appeared with Jupiter. Table 1: Textual diversity in the Open Korean Historical Corpus. Selected texts illustrate the linguistic diversity across different eras (periods are approximate). Highlighting distinguishes key scripts and lexical features—such as Hanja, Japanese, Old Hangul, and North Korean vocabulary—from standard Hangul. An English translation is provided for each example. However, computational research on Korean has predominantly focused on modern, Hangul-based texts, largely neglecting the intermediate period of Hanja-Hangul mixed script. A significant barrier to this research is the lack of accessible historical corpora. Most corpora created by the Korean government ( e.g ., Sejong Corpus, Modu Corpus) are distributed under restrictive licenses that make releasing derivative works nearly impossible Hwang and Choi (2016) . While some institutions release digitized documents online, they often do not offer them as downloadable datasets. As a result, interested researchers have been forced to individually crawl numerous websites to compile their own datasets. To address this problem, we introduce the Open Korean Historical Corpus, a large-scale, openly licensed dataset. We collected and processed 17.7 million documents from 19 distinct sources, totaling 5.1 billion tokens and covering the period from the 7th century through 2025. As illustrated in Figure 1 , the collection encompasses six languages: Korean (Middle, Early Modern, Modern, North), Classical Chinese, and Modern Japanese. This is, to our knowledge, the first openly licensed corpus to offer broad temporal coverage of these historical stages, particularly Middle Korean and Early Modern Korean, as well as under-represented writing systems, including Korean-style Sinitic (Idu) and Hanja-Hangul mixed script. Table 1 provides selected examples that illustrate this linguistic diversity across different eras. Using this corpus, we quantitatively analyze major linguistic shifts. Our analysis of Korean-style Sinitic (Idu) shows its usage peaked in the 1860s before declining sharply, particularly after the 1894 Kabo Reform. We also trace the diachronic transition from Hanja to Hangul, finding it was a rapid transformation rather than a gradual shift; writing exclusively in Classical Chinese dominated written records until 1890, but by 1980, Hangul comprised over 93% of characters. Furthermore, we quantify the lexical divergence of North Korean, which causes modern tokenizers to produce up to 51 times higher out-of-vocabulary (OOV) rates due to its unique orthography for loanwords and distinct native vocabulary. We release the corpus and our processing code under the CC BY-NC 4.0 and MIT licenses, respectively, to facilitate further research in Korean diachronic linguistics and historical NLP. 1 1 1 Dataset and code available at https://github.com/seyoungsong/OKHC . 2. Background In this section, we examine the historical background of the Korean language and writing system and discuss how they have changed over time. 2.1. Historical Development of Korean The Korean language, with its earliest extant records dating back to the early 5th century Kim (1998) ; Bailblé (2016) , has undergone substantial changes. While multiple periodization schemes have been proposed, reflecting different scholarly foci and periodization criteria, we adopt the four-period scheme NIKL (2010) for consistency and clarity. This provides a useful overview by combining dynastic transitions and major linguistic changes as criteria for division. Ancient (–10C) This period spans from before the Common Era to the establishment of the Goryeo dynasty in 918. During this period, Chinese characters were introduced to the Korean peninsula and became the primary medium of written Korean, known as Hanja Choo (2016) . Only a few written records from this era have survived. Medieval (10C–16C) The