A Comparative Analysis of Word Segmentation, Part-of-Speech Tagging, and Named Entity Recognition for Historical Chinese Sources, 1900-1950

Zhao Fang, Liang-Chun Wu, Xuening Kong, Spencer Dean Stewart · 2025

Read A Comparative Analysis of Word Segmentation, Part-of-Speech Tagging, and Named Entity Recognition for Historical Chinese Sources, 1900-1950 by Zhao Fang, Liang-Chun Wu, Xuening Kong, Spencer Dean Stewart (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19844.txt.

Introduction

Zhao Fang, Liang-Chun Wu, Xuening Kong, and Spencer Dean Stewart compare computer tools for parsing old Chinese texts, in a 2025 paper. Texts from 1900 to 1950 lack spaces between words, and the language shifted a lot in those decades. The team tested three basic jobs: splitting words, tagging word roles, and finding names.

Their samples came from a Shanghai Library collection of Republican-era journals. Traditional tools such as Jieba and spaCy faced large language models including GPT-4o and Claude 3.5. The models won on every measure, though at far higher computing cost. They also handled hard cases better, such as poems and texts from before versus after 1920. Their skill at learning from context cuts the need for specially trained data. The authors frame the choice as accuracy bought with computing power.

Catalog record

CREATORZhao Fang, Liang-Chun Wu, Xuening Kong, Spencer Dean Stewart
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-19844

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Comparative Analysis of Word Segmentation, Part-of-Speech Tagging, and Named Entity Recognition for Historical Chinese Sources, 1900-1950 Zhao Fang, Liang-Chun Wu, Xuening Kong, Spencer Dean Stewart Abstract This paper compares large language models (LLMs) and traditional natural language processing (NLP) tools for performing word segmentation, part-of-speech (POS) tagging, and named entity recognition (NER) on Chinese texts from 1900 to 1950. Historical Chinese documents pose challenges for text analysis due to their logographic script, the absence of natural word boundaries, and significant linguistic changes. Using a sample dataset from the Shanghai Library Republican Journal corpus, traditional tools such as Jieba and spaCy are compared to LLMs, including GPT-4o, Claude 3.5, and the GLM series. The results show that LLMs outperform traditional methods in all metrics, albeit at considerably higher computational costs, highlighting a trade-off between accuracy and efficiency. Additionally, LLMs better handle genre-specific challenges such as poetry and temporal variations (i.e., pre-1920 versus post-1920 texts), demonstrating that their contextual learning capabilities can advance NLP approaches to historical texts by reducing the need for domain-specific training data. 1 Introduction With the large-scale digitization of historical documents, researchers are increasingly interested in how Natural Language Processing (NLP) methods might be used and adapted to address the unique characteristics of older texts Guldi 2023 ; Ehrmann et al. 2023 ; Manjavacas and Fonteyn 2022 ; Piotrowski 2012 . Classification models for tasks such as Named Entity Recognition (NER) have improved significantly with the development of neural-based approaches. However, their precision for historical materials still lags behind that of models trained on contemporary texts Ehrmann et al. 2023 . Recent applications of language model-based approaches to NLP tasks have shown mixed results for using large language models (LLMs) such as ChatGPT to generate universal NER output Qin et al. 2023 , including for historical documents González-Gallardo et al. 2023 . More targeted, domain-specific approaches have also proven effective Polak and Morgan 2024 , including classification tasks common in digital humanities research Bamman et al. 2024 and in low-resource settings Frei and Kramer 2023 ; Wang et al. 2023 . The processing of historical Chinese documents presents unique challenges for NLP tasks due to the logographic writing system, the absence of natural word boundaries, and the rich morphological structures embedded within individual characters Cui et al. 2020 . Previous work on relatively “simple” tasks, such as Chinese word segmentation, has evolved through three paradigm shifts: rule-based systems, statistical machine learning models, and LLMs based on the transformer architecture Fang 2024 . Traditional machine learning methods such as Jieba and spaCy rely on dictionary matching and hidden Markov models to identify word boundaries. The dramatic linguistic and logographical transformations that occurred in China during the late nineteenth and twentieth centuries Liu 1995 ; Tsu 2023 pose particular challenges for these models, which struggle to handle out-of-vocabulary terms. Some researchers have approached this problem by first converting historical sources into standardized simplified Chinese before performing NLP tasks Stewart 2025 . Others have drawn from domain-specific approaches to manually curate datasets from historical sources to improve tasks such as segmentation Luo et al. 2019 ; Blouin et al. 2023 . The advent of LLMs capable of detecting contextual patterns from large corpora presents new opportunities for processing classical and modern Chinese texts. Although there has been growing interest in BERT-based models and the development of domain-specific tools to process historical Chinese sources Yu and Wang 2020 ; Cui et al. 2020 ; Fang 2024 , further research is needed to evaluate LLMs’ performance on NLP tasks. This short research paper presents a comparative analysis of machine learning and LLM-based tools for word segmentation, part-of-speech (POS) tagging, and NER on a diverse set of sample texts taken from the Shanghai Library Republican Journal corpus. 1 1 1 https://textual-optics-lab.uchicago.edu/shanghai-library-republican-journal-corpus This study finds that, for transitional-era Chinese texts, LLM-based approaches outperform traditional NLP tools on segmentation, POS tagging, and NER tasks. However, these improvements come with notable increases in computational costs, highlighting a trade-off between performance and efficiency. 2 Methodology To create our ground truth files, we extracted a random sample of passages from a large textual dataset of Late-Qing and Republican periodicals held by the Shanghai Library. We identified 208 passages spanning the decades 1900 to 1950. These passages include a variety of genres and topics, such as government reports, academic writing, social and political commentary, and literary texts such as short stories and poetry. To assess the ability of existing tools to handle different genres and textual changes over time, our sample included 41 passages identified as poetry, with the the remaining 167 passages distributed across five decades: 1900 (23 passages), 1910 (32 passages), 1920 (40 passages), 1930 (34 passages) and 1940 (38 passages). The passages ranged in length from 6 to 170 characters, with an average of 41.3 characters and a total of 8,610 characters. From this sample, the authors collectively segmented and tagged the passages, with each passage verified by two authors. Discrepancies were noted and resolved after further discussion. We selected widely used and reputable tools for Chinese segmentation and NER, as well as several popular LLMs, to evaluate out-of-the-box performance of these tools in our comparative analysis. To evaluate their effectiveness, we generated consistent prompts for each LLM and utilized their APIs to ensure standardized conditions. The prompts included clear, precise instructions requiring that the results be provided in a structured JSON format. This enabled a straightforward comparison with our established ground-truth dataset. LLM API Prompt You are a spaCy-style NLP annotator for Traditional Chinese text from 1900-1950. Do not remove any text, including punctuation and brackets. Don’t treat spaces as tokens. Tasks: 1. Segment the input text into tokens. 2. Annotate each token with: - text: the exact token string - pos: a coarse POS tag (POS tags are exclusively: {list of tags}) - ent: the entity label if the token is part of a named entity (NER types are exclusively: {list of tags}, otherwise "") 3. Return the result as a JSON list of objects. Here is an example of expected input and output: input text = "此問題為英國政治上第一棘手之難問題。" expected output = [ {"text": "此", "pos": "DET", "ent": ""}, {"text": "問題", "pos": "NOUN", "ent": ""}, {"text": "為", "pos": "VERB", "ent": ""}, {"text": "英國", "pos": "PROPN", "ent": "GPE"}, {"text": "政治", "pos": "NOUN", "ent": ""}, {"text": "上", "pos": "ADP", "ent": ""}, {"text": "第一", "pos": "NUM", "ent": ""}, {"text": "棘手", "pos": "ADJ", "ent": ""}, {"text": "之", "pos": "PART", "ent": ""}, {"text": "難", "pos": "ADJ", "ent": ""}, {"text": "問題", "pos": "NOUN", "ent": ""}, {"text": "。", "pos": "PUNCT", "ent": ""} ] Nothing else but valid JSON in the final response. The performance of each approach was assessed based on several key metrics: 1. F1 Score: As the standard metric for evaluating Chinese tokenization, the F1 score effectively balances the risks of over-tokenization and under-tokenization. An F1 score of 90% or higher is generally considered indicative of high accuracy. 2. Part-of-Speech (POS) Accuracy (%): This metric measures the accuracy of POS tagging for those tokens that were correctly segmented. 3. Named Entity Recognition (NER) Accuracy (%): This measures the precision of named entity tagging for those tokens that were correctly segmented. 4. Time (in seconds): The processing speed for each approach

Read the complete text