Modern Models, Medieval Texts: A POS Tagging Study of Old Occitan

Matthias Schöffel, Marinus Wiedner, Esteban Garces Arias, Paula Ruppert, Christian Heumann, Matthias Aßenmacher · 2025

Read Modern Models, Medieval Texts: A POS Tagging Study of Old Occitan by Matthias Schöffel, Marinus Wiedner, Esteban Garces Arias, Paula Ruppert, Christian Heumann, Matthias Aßenmacher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07827.txt.

Introduction

Matthias Schöffel and co-authors test whether large language models can tag parts of speech in Old Occitan, a medieval language with loose spelling rules. Their 2025 study uses two text collections, one of saints' lives and one of medical writing. The language is low-resource, meaning little training data exists, and its forms shift a lot over time.

The models struggled with the wide spelling and grammar variation in the old texts. A close look at the errors shows where the tools break down and why. The authors give concrete advice for doing better with historical languages. Their findings matter for scholars who want modern AI help with old manuscripts.

Catalog record

CREATORMatthias Schöffel, Marinus Wiedner, Esteban Garces Arias, Paula Ruppert, Christian Heumann, Matthias Aßenmacher
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-07827

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modern Models, Medieval Texts: A POS Tagging Study of Old Occitan Matthias Schöffel, Marinus Wiedner, Esteban Garces Arias, Paula Ruppert, Christian Heumann, Matthias Aßenmacher Abstract Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing, yet their effectiveness in handling historical languages remains largely unexplored. This study examines the performance of open-source LLMs in part-of-speech (POS) tagging for Old Occitan, a historical language characterized by non-standardized orthography and significant diachronic variation. Through comparative analysis of two distinct corpora—hagiographical and medical texts—we evaluate how current models handle the inherent challenges of processing a low-resource historical language. Our findings demonstrate critical limitations in LLM performance when confronted with extreme orthographic and syntactic variability. We provide detailed error analysis and specific recommendations for improving model performance in historical language processing. This research advances our understanding of LLM capabilities in challenging linguistic contexts while offering practical insights for both computational linguistics and historical language studies. (a) (b) Figure 1: (a) Geographic distribution of Old Occitan with its principal dialect zones ( Sibille, 2024 ) . (b) Orthographic diversity in Old Occitan texts, as evidenced by multiple graphical variants of the same term, illustrating inherent challenges for modern LLMs. 1 Introduction Old Occitan, also known as Old Provençal, was widely spoken from the 11th to the 16th century across southern France, northeastern Spain, and northwestern Italy (cf. Fig. 1 (a)). This language played a pivotal role in shaping both Romance linguistics and medieval European literature, particularly through its renowned troubadour tradition. However, computational analysis and digital preservation of Old Occitan face significant challenges, primarily due to the limited availability of digitized manuscripts and annotated corpora compared to contemporary medieval languages such as Old French ( Scrivner and Kübler, 2012 ) . A key obstacle in processing Old Occitan texts is their pronounced orthographic variation, as illustrated in Figure 1 (b) through the term abeurador (’watering place’), which exhibits substantial regional and textual variations in spelling. These variations, while historically significant, present particular challenges for automated text processing tasks such as Part-of-Speech (POS) tagging, which is the focus of the present work. The imperative for accurate POS tagging in low-resource languages like Old Occitan extends beyond mere technical curiosity. POS tagging is a foundational step in numerous NLP applications, from syntactic parsing and information extraction to more advanced tasks in digital humanities. For historical languages, reliable tagging is critical not only for linguistic analysis but also for reconstructing the evolution of language, understanding regional variation, and supporting interdisciplinary research that bridges history and computational methods. Moreover, the performance of LLMs on such texts offers insights into the adaptability of modern models when confronted with non-standardized data – a challenge that remains largely unaddressed in contemporary NLP research. In this study, we systematically evaluate a range of LLMs using various prompting strategies—Zero-shot, few-shot, and instruction-tuned approaches—on a corpus comprising 91,953 tokens. Beyond a mere exploration of current capabilities, our work elucidates key factors influencing model performance and offers a rigorous error analysis and practical recommendations to mitigate the effects of input modifications and enhance POS tagging accuracy. Research Questions: Our study addresses the following research questions: (i) RQ1: How effectively can current LLMs perform POS tagging on Old Occitan texts, given the challenges posed by non-standardized orthography and sparse annotated resources? (discussed in Section 5.1 ) (ii) RQ2: Which prompting strategies—Zero-shot, few-shot, or instruction-tuned—yield the most robust performance on this low-resource, historical language? (discussed in Section 5.2 ) (iii) RQ3: What specific error patterns and model biases emerge during POS tagging, and how can these insights inform practical improvements? (addressed in Sections 6 and 7 ). By answering these questions, we aim to bridge modern NLP techniques with the nuanced demands of historical linguistics. Contributions: We summarize our contributions as follows: 1. We provide the first comprehensive evaluation of multiple LLMs for POS tagging on Old Occitan texts, establishing a robust baseline for historical Romance languages. 2. We systematically compare concrete prompting strategies—including Zero-shot, few-shot, and instruction-tuned approaches—to adapt LLMs to the irregularities of non-standardized historical data. 3. We perform a detailed error analysis to uncover model-specific biases and limitations, offering targeted recommendations to improve POS tagging performance on low-resource texts. 4. We release a novel POS Tagging dataset for Old Occitan, along with our code and experimental results, to facilitate future research in historical NLP. 1 1 1 Link to our GitHub Repository. 2 Related work Part-of-speech (POS) tagging for low-resource languages presents unique challenges that have gained increasing attention in computational linguistics. Several approaches have emerged to address data scarcity in these settings, with varying degrees of success. Cardenas et al. (2019) proposed a grounded unsupervised universal POS tagger for low-resource languages, framing tagging as a clustering problem followed by decipherment-based grounding. This approach requires no labeled training data and demonstrates reasonable performance across diverse languages. Building on this work, Plank et al. (2018) demonstrated that integrating conventional lexical information can significantly improve neural cross-lingual POS tagging, suggesting that even small amounts of symbolic lexical resources can be valuable when gold-standard corpora are unavailable. However, Kann et al. (2020) challenged the effectiveness of weakly supervised approaches for truly low-resource languages. Their evaluation across 15 typologically diverse languages revealed that state-of-the-art weakly supervised POS taggers perform significantly worse under realistic resource constraints than previously reported, with accuracy below 50% for most languages. This skepticism is further supported by Moeller et al. (2021) , who found that the presence or absence of POS tags does not significantly impact performance in morphological learning tasks, with some cases showing improved performance when POS tags were removed. For endangered languages specifically, Anastasopoulos et al. (2018) evaluated POS tagging techniques on Griko, achieving 72.9% accuracy through combined semi-supervised methods and cross-lingual transfer. Similarly, Gore and Khatavkar (2022) demonstrated success with the endangered Indian tribal language Katkari, achieving 86.84% accuracy using Hidden Markov Models and the Viterbi algorithm, suggesting that traditional statistical approaches remain viable for low-resource scenarios. Recent work has focused particularly on languages with dialectal variation. The creation of CorpusArièja by Poujade et al. (2024) provides a valuable resource for Occitan, containing 41,000 tokens with POS tags and handling both dialectal and spelling variations. Building on this, Hopton and Aepli (2024) demonstrated that large multilingual models can effectively handle dialectal variation in Occitan without requiring spelling normalization, particularly when fine-tuned for POS tagging. More recently, there have been efforts to ramp up the availability of resources for Old Occitan, including the creation of a digital version of the Old Occitan dictionary 2 2 2 DOM: Dictionnaire de l’occitan médiéval http://www.dom-en-ligne.de/ at the Bavarian Academy of Sciences. Building on handwritten resources, Garces Arias et al. (2023) tackled automatic transcription, combining a custom-trained Swin image encoder with a BERT-based text decoder to enhance digitization of Old Occitan spelling variations. 3 Data Our benchmark comprises two corpora drawn from distinct domains: a hagiographical text and a medical treatise. The former is represented by the Vida de Sant Honorat , while the latter is embodied by On surgery and instruments by Abū l-Qāsim al-Ḥalaf al-Zahrāwī (Albucasis). For the hagiographical corpus, the primary source is the manuscript Nouvelle Acquisition Française 6195 (NAF6195, also known as manuscript M of the Vida de Sant Honorat

Read the complete text