DAG: Dictionary-Augmented Generation for Disambiguation of Sentences in Endangered Uralic Languages using ChatGPT
Mika Hämäläinen · 2024
Read DAG: Dictionary-Augmented Generation for Disambiguation of Sentences in Endangered Uralic Languages using ChatGPT by Mika Hämäläinen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01531.txt.
Description
Abstract We showcase that ChatGPT can be used to disambiguate lemmas in two endangered languages ChatGPT is not proficient in, namely Erzya and Skolt Sami. We augment our prompt by providing dictionary translations of the candidate lemmas to a majority language - Finnish in our case.
Catalog record
| CREATOR | Mika Hämäläinen |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01531
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DAG: Dictionary-Augmented Generation for Disambiguation of Sentences in Endangered Uralic Languages using ChatGPT Mika Hämäläinen Abstract We showcase that ChatGPT can be used to disambiguate lemmas in two endangered languages ChatGPT is not proficient in, namely Erzya and Skolt Sami. We augment our prompt by providing dictionary translations of the candidate lemmas to a majority language - Finnish in our case. This dictionary augmented generation approach results in 50% accuracy for Skolt Sami and 41% accuracy for Erzya. On a closer inspection, many of the error types were of the kind even an untrained human annotator would make. 1 Introduction Morphological disambiguation is a critical task in natural language processing (NLP), especially for morphologically rich and endangered languages. Skolt Sami (sms) and Erzya (myv), both belong to Uralic language family and they are classified as critically and definitely endangered respectively by Unesco Moseley 2010 . This poses significant challenges in this domain due to their complex morphological systems and limited available linguistic resources (see Hämäläinen 2021 ). In languages like these, each word form can have multiple possible morphological interpretations and lemmas, and determining the correct one in context is essential for accurate language processing. Traditional approaches to morphological disambiguation for Uralic languages often rely on finite-state transducers (FSTs) and constraint grammars (CGs) that list all potential lemmas for a word, but these systems struggle to accurately select the appropriate lemma in ambiguous contexts - not to mention that CG disambiguators have not even been created to a majority of these languages. Additionally, while some modern NLP techniques, such as machine learning models, have been successful in languages with large datasets (see Shen et al. 2016 ; Zalmout and Habash 2017 ), such methods are less effective for languages like Skolt Sami and Erzya, which suffer from limited annotated corpora and lexicographical resources. This paper presents a novel method for performing morphological disambiguation for Skolt Sami and Erzya that leverages a combination of a traditional FST-based analyzer, a bilingual dictionary and a state-of-the-art language model, namely ChatGPT. Our approach involves passing each sentence through an FST to generate a list of possible lemmas for every word. These lemmas are then translated into a majority language (Finnish in our case) using a dictionary. Finally, we utilize ChatGPT, a powerful transformer-based language model, to analyze the translated sentence, disambiguate the lemmas, and select the most contextually appropriate form for each word. The dictionary needs to be provided given that ChatGPT is not proficient in these languages. By integrating the structured linguistic knowledge from FSTs with the contextual understanding of large language models, this method aims find a novel way that does not need a time consuming rule-writing or data annotation process for morphological disambiguation for Skolt Sami and Erzya. The proposed approach is particularly valuable for endangered languages, where data scarcity hinders the development of purely data-driven models. This paper details the methodology, presents an evaluation of the approach, and discusses the potential for applying this approach to other morphologically complex languages. We have released our disambiguation code as an addition to UralicNLP 1 1 1 https://github.com/mikahama/uralicNLP/wiki/Disambiguation . Prompt template Actual prompt Your task is to disambiguate a sentence in [LANGUAGE] You will be given the sentence, a table that has all of the words of the sentence in separate rows and a comma separated list of possible lemmas. You will need to pick the correct lemma for each word so that every word will have only one lemma. To help you understand [LANGUAGE] you will also get a second table that gives you translations of the words in [LANGUAGE2]. Sentence: [SENTENCE] Table of lemmas: [TABLE1] [LANGUAGE] - [LANGUAGE2] vocabulary: [TABLE2] Please write out the steps of your decision process and provide a list of lemmas in JSON format at the very end of your answer. Example: {"lemmas": ["lemma 1", "lemma 2", "lemma 3"]} Your task is to disambiguate a sentence in Skolt Sami. You will be given the sentence, a table that has all of the words of the sentence in separate rows and a comma separated list of possible lemmas. You will need to pick the correct lemma for each word so that every word will have only one lemma. To help you understand Skolt Sami you will also get a second table that gives you translations of the words in Finnish. Sentence: Päärna mõ \prime nne mååusat . Table of lemmas: +————————–+ | Word | Lemmas | +——-+——————+ | Päärna| päärnaž | +——-+——————+ | mõ \prime nne|mõõnnâd, mõ \prime nn’jed| +——-+——————+ |mååusat| mååusat | +——-+——————+ | . | . | +————————–+ Skolt Sami - Finnish vocabulary: +—————————————+ |Skolt Sami| Finnish | +———-+—————————-+ | päärnaž |poikanen, lapsi, pieni poika| +———-+—————————-+ | mõõnnâd | mennä | +———-+—————————-+ | mõ \prime nn’jed| munata | +———-+—————————-+ | mååusat | takaisin | +———-+—————————-+ | . | | +—————————————+ Please write out the steps of your decision process and provide a list of lemmas in JSON format at the very end of your answer. Example: {"lemmas": ["lemma 1", "lemma 2", "lemma 3"]} Table 1: The prompt template and an example of it filled 2 Related Work Constraint grammars (CGs) Karlsson 1990 have been widely used in the context of Uralic languages for disambiguation given their compatibility with the output of FSTs. They are, however, not used widely anymore in the mainstream NLP research. In this section, we will go through some of the more modern NLP approaches to this task in the context of endangered languages. In a work focusing on Uralic languages Ens et al. 2019 , the authors propose a Long Short-Term Memory (LSTM) model that automatically ranks morphological readings of sentences based on their quality. This ranking can be used either to evaluate existing CG disambiguators or to directly disambiguate sentences. Notably, their approach relies on morphological abstraction and can be effectively trained with minimal data. Apertium’s approach Khanna et al. 2021 is to employ statistical methods based on patterns learned from a corpus in addition to CG. Two prominent methods include a bigram-based first-order Hidden Markov Model (HMM), which selects analyses based on a probabilistic model of part-of-speech tag sequences in context, and an Averaged Perceptron tagger, which assigns weights to features defined by language-pair developers. The paper by Keleg et al. 2020 introduces a method for weighting the outputs of an FST-based morphological analyzer to disambiguate its results. The approach uses a word2vec model, trained in an unsupervised manner on raw, untagged corpora, to capture semantic meaning. Unlike traditional methods that require manually constructed tagged corpora, this method disambiguates morphological analyses without relying on such resources. Additionally, it focuses on token-level information rather than context, differing from most approaches that heavily depend on contextual features for disambiguation. 3 Method The method itself does not require any training or additional annotated data. However, to evaluate our method, we use the Universal Dependences treebanks for Erzya Rueter and Tyers 2018 and Skolt Sami Nivre et al. 2022 . These treebanks have word forms and their correct lemmas for each word in each sentence. Given that we do not need to do training, we concatenate the training and test datasets into one dataset for both languages. Next, we used the Skolt Sami Rueter