Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages
Micha Elsner, David Liu · 2025
Read Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages by Micha Elsner, David Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09778.txt.
Introduction
Micha Elsner and David Liu use large language models to help document endangered languages, in a 2025 paper. Interlinear glossed text, where each word gets a translation and grammar label, is basic infrastructure for linguistics. Making it by hand is slow. The authors argue that language models, good at following plain instructions, can make the process more accessible.
Their system prompts the model word by word, with retrieved examples as guidance, across the seven languages of a shared benchmark. It beat the previous best automated baseline on morpheme-level scores for every language. A simple oracle picking among three guesses beat the challenge winner on word-level scores in five languages. In a case study on Tsez, the model even wrote and followed its own instructions, cutting errors on a tricky grammar feature. The authors see interactive glossing help as the prize.
Catalog record
| CREATOR | Micha Elsner, David Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09778
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages Micha Elsner, David Liu Abstract Partly automated creation of interlinear glossed text (IGT) has the potential to assist in linguistic documentation. We argue that LLMs can make this process more accessible to linguists because of their capacity to follow natural-language instructions. We investigate the effectiveness of a retrieval-based LLM prompting approach to glossing, applied to the seven languages from the SIGMORPHON 2023 shared task. Our system beats the BERT-based shared task baseline for every language in the morpheme-level score category, and we show that a simple 3-best oracle has higher word-level scores than the challenge winner (a tuned sequence model) in five languages. In a case study on Tsez, we ask the LLM to automatically create and follow linguistic instructions, reducing errors on a confusing grammatical feature. Our results thus demonstrate the potential contributions which LLMs can make in interactive systems for glossing, both in making suggestions to human annotators and following directions. 1 Introduction At least half of the world’s languages are under-documented Bird 2011 and at least 2500 are endangered Moseley 2012 . A multitude of factors contribute to language endangerment and death: a lack of available resources; pressure from dominant languages, and governmental policies that impair linguistic diversity all contribute to the gradual loss of a language Bromham et al. 2022 . As speaker populations dwindle, documentation has become a priority in order to enable future revitalization projects. Interlinear glossed text (IGT) is a widely-used format for documentary projects, utilizing grammatical labels and morpheme segmentation in order to construct translations for individual words and morphemes. While IGT efforts can be beneficial from a revitalization standpoint, the burdens of cost and time create the need for an expedited procedure that can surpass the rate of language death Moeller and Hulden 2018 . Semi-automated methods using computational assistance for glossing can potentially help linguists to annotate more data faster Palmer et al. 2009 ; Moeller and Hulden 2018 . The best current approaches to glossing ( Yang et al. 2024a , e.g.) use purpose-built sequence models, but there has been some recent interest in LLMs as glossing models Ginn et al. 2024a . We believe this is an important direction because LLMs are (in principle) capable of interacting with linguists and native-speaker consultants in a more natural way than sequence models: they can follow instructions. Rather than finding and annotating examples of a problematic construction, the user could explain the appropriate generalization to the system or supply a passage from a descriptive grammar, changing the terms on which users can interact with language processing technology Meighan 2021 . But recent work on low-resource translation has cast doubt on LLMs’ ability to understand this kind of instruction in practice Court and Elsner 2024 ; Aycock et al. 2024 . We explore what types of information, examples and instructions most benefit LLMs in performing linguistic analysis, and provide a first demonstration that linguistic instructions can reduce error rates for labeling a specific construction in Tsez. We believe our findings also shed light on previous results involving LLMs’ failure to follow linguistic instructions while translating. We use word-by-word retrieval based prompting to gloss sentences in the seven languages of the SIGMORPHON 2023 shared task Ginn et al. 2023 . This approach differs somewhat from whole-sentence prompting Ginn et al. 2024a and prompting for post-processing Yang et al. 2024a . Word-by-word prompting enables easy elicitation of k -best options and we show that the LLM is often nearly right even when its top guess is wrong— a feature that could be very helpful for human annotators, since machine suggestions can speed up manual annotation Palmer et al. 2009 . 1 1 1 Whether machine suggestions are helpful or not depends on how examples are selected and how fluent the annotator is in the object language. Our work thus makes three major contributions. We replicate earlier results showing that prompt-based glossing performs acceptably (beating the 2023 baseline in every language) for inter-linear glossing, but not at SOTA; we show these results apply to word-by-word as well as sentence-level retrieval. 2 2 2 Code and results at https://github.com/dfdddfressd/glossing-project . We find that a simple 3-best oracle beats the challenge winner for all but two languages, indicating that LLM prompting has high potential to assist a human annotator by suggesting high-quality options. Finally, a case study on Tsez shows that automatically-generated linguistic instructions can reduce errors involving a particular set of commonly-confused tags by 10%. 2 Related work 2.1 Automated glossing Models for automated glossing have used both non-neural Palmer et al. 2009 ; Moeller and Hulden 2018 and neural Moeller and Hulden 2018 ; Zhao et al. 2020 learning methods. The SIGMORPHON challenge baseline Ginn 2023 uses the RoBERTa architecture, while the challenge winner Girrbach 2023 uses an encoder-decoder model with hard attention which automatically induces morphological segmentations from the data. Such relatively small task-specific and language-specific models still represent the state of the art for the glossing task, because they can learn efficiently from the small datasets available and because they are cheap to apply in the computational sense. Neither Ginn 2023 nor Girrbach 2023 use sentence translations. Although this simplicity is in some sense an advantage, it is also a limitation of the approach, since in some cases, the correct gloss is more evident from the translation, for instance in disambiguating homophones or syncretic morphemes Zhao et al. 2020 . Yang et al. 2024a adds embeddings of the sentence translations on top of the Girrbach 2023 model and obtain an improvement of 4% in the word-level accuracy score. The utility of translations motivates Ginn et al. 2024a to apply LLMs in the glossing task. 3 3 3 Yang et al. 2024a report preliminary and unsuccessful attempts to use Llama2 for translation embeddings. Like ours, their attempt is retrieval-based. They use one prompt per sentence, retrieving up to 100 similar sentences per instance; the most effective similarity metric is chrF Popović 2015 . In the four languages they ran, their results lie between the challenge baseline and Girrbach 2023 except in Gitksan, which has the lowest absolute scores due to its very high OOV rate. Aycock et al. 2024 also apply LLMs to glossing the Kalamang language using glossed examples from a grammar book. Their glossing system improves on the challenge baseline, and benefits slightly from the addition of generated text explaining some typological features of Kalamang (such as the absence of definite articles). Unlike this work, they do not attempt to disambiguate particular constructions. Yang et al. 2024b and Shandilya and Palmer 2025 use LLMs in a different way, to post-correct glosses proposed by an encoder-decoder model. Shandilya and Palmer 2025 find that retrieval-based post-correction can improve the glosses proposed by a competitive encoder-decoder model. Recent LLMs are capable of producing informative explanations of their decisions, and surprisingly in light of the results surveyed in the next subsection, are able to incorporate information from a linguistic grammar to do so. Like this paper, their results show some promise for incorporating linguistic instructions into the glossing process. A third approach is taken in Ginn et al. 2024b . Rather than