Leveraging Transformer-Based Models for Predicting Inflection Classes of Words in an Endangered Sami Language

Khalid Alnajjar, Mika Hämäläinen, Jack Rueter · 2024

Read Leveraging Transformer-Based Models for Predicting Inflection Classes of Words in an Endangered Sami Language by Khalid Alnajjar, Mika Hämäläinen, Jack Rueter (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02556.txt.

Description

Abstract This paper presents a methodology for training a transformer-based model to classify lexical and morphosyntactic features of Skolt Sami, an endangered Uralic language characterized by complex morphology. The goal of our approach is to create an effective system for understanding and analyzing Skolt Sami, given the limited data availability and linguistic intricacies inherent to the language.

Catalog record

CREATORKhalid Alnajjar, Mika Hämäläinen, Jack Rueter
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-02556

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging Transformer-Based Models for Predicting Inflection Classes of Words in an Endangered Sami Language Khalid Alnajjar, Mika Hämäläinen, Jack Rueter Abstract This paper presents a methodology for training a transformer-based model to classify lexical and morphosyntactic features of Skolt Sami, an endangered Uralic language characterized by complex morphology. The goal of our approach is to create an effective system for understanding and analyzing Skolt Sami, given the limited data availability and linguistic intricacies inherent to the language. Our end-to-end pipeline includes data extraction, augmentation, and training a transformer-based model capable of predicting inflection classes. The motivation behind this work is to support language preservation and revitalization efforts for minority languages like Skolt Sami. Accurate classification not only helps improve the state of Finite-State Transducers (FSTs) by providing greater lexical coverage but also contributes to systematic linguistic documentation for researchers working with newly discovered words from literature and native speakers. Our model achieves an average weighted F1 score of 1.00 for POS classification and 0.81 for inflection class classification. The trained model and code will be released publicly to facilitate future research in endangered NLP. 1 Introduction Skolt Sami is a minority language in the Uralic family, spoken primarily in Finland, and is characterized by complex morphosyntactic properties and rich morphological forms (see Koponen and Rueter 2016 ). Minority languages like Skolt Sami face significant challenges in the field of natural language processing (NLP) due to their endangered nature, including a lack of extensive annotated datasets and linguistic resources. This scarcity complicates the development of computational models capable of effectively understanding and analyzing the language. Moreover, the morphology of Skolt Sami is highly intricate, with numerous inflections and derivations that present additional challenges for automated processing Rueter and Hämäläinen 2020 . Despite these challenges, developing NLP models for minority languages is essential to preserve linguistic diversity and support language revitalization. Accurate part-of-speech (POS) and inflection class classification are fundamental steps in understanding the grammatical and semantic structure of a language. Such classifications enable downstream NLP applications like machine translation, morphological analysis, and syntactic parsing, which are particularly important for languages with rich morphology. Additionally, effective classifiers can assist in improving the current state of FSTs by providing greater lexical coverage, ultimately enhancing their ability to handle the full range of morphological variations found in Skolt Sami. Classifiers can also aid researchers in systematically documenting new words collected from literature and native speakers, which is crucial for tracking linguistic evolution in endangered contexts. For Skolt Sami, POS and inflection class classification can contribute to building digital resources and educational tools, making the language more accessible to both linguists and speakers. To address these challenges, we propose a transformer-based model designed to automate the analysis of Skolt Sami, specifically for POS and inflection class classification. Our approach includes data extraction, preprocessing, augmentation, model training and evaluation. We employed advanced transformer architectures to learn the linguistic features of Skolt Sami effectively. Additionally, we provide both the trained model and the accompanying code publicly to support future research efforts on endangered languages 1 1 1 https://github.com/mokha/predict-inflection-class . The contributions of this work are as follows: 1. Data Augmentation Using Miniparadigms : We employed data augmentation techniques, including the generation of morphological forms, to mitigate data scarcity and improve model robustness. 2. Transformer-Based Model : We designed a transformer-based model for POS and inflection class classification in Skolt Sami, utilizing shared embedding layers and task-specific output heads. 2 Related work Skolt Sami has received a moderate amount of NLP research interest as a result of Dr Jack Rueter’s amazing work on building the fundamental NLP building blocks for Skolt Sami 2 2 2 https://researchportal.helsinki.fi/en/projects/ koltansaamen-elvytys-kieliteknologia-avusteisen-kielenoppimisohje as a result, Skolt Sami has an FST Rueter and Hämäläinen 2020 an online dictionary (see Hämäläinen et al. 2021a ), a Universal Dependencies treebank Nivre et al. 2022 and some neural models to identify cognates Hämäläinen and Rueter 2019 . An empirical study by Wu et al. 2020 reveals that the transformer’s performance on character-level transduction tasks, such as morphological inflection generation, is significantly influenced by batch size, unlike in recurrent models. By optimizing batch size and introducing feature-guided transduction techniques, the transformer can outperform RNN-based models, achieving state-of-the-art results on tasks such as grapheme-to-phoneme conversion, transliteration, and morphological inflection. This study demonstrates that, with appropriate modifications, transformers are highly effective for character-level tasks as well. Recent research Abudouwaili et al. 2023 has introduced a joint morphological tagger specifically designed for low-resource agglutinative languages. By leveraging multi-dimensional contextual features of agglutinative words and employing joint training, the proposed model mitigates the error propagation typically seen in part-of-speech tagging while enhancing the interaction between part-of-speech and morphological labels. Furthermore, the model predicts part-of-speech and morphological features separately, using a graph convolution network to capture higher-order label interactions. Experimental results demonstrate that this approach outperforms existing models, showcasing its effectiveness in low-resource language settings. One notable contribution in this area is a transformer-based inflection system that enhances the standard transformer architecture by incorporating reverse positional encoding and type embeddings proposed by Yang et al. 2022 . To address data scarcity, the model also leverages data augmentation techniques such as data hallucination and lemma copying. The training process is conducted in two stages: initial training on augmented data using standard backpropagation and teacher forcing, followed by further training with a modified version of scheduled sampling, termed student forcing. Experimental results demonstrate that this system achieves competitive performance across both small and large data settings, highlighting its efficacy in diverse morphological inflection tasks. Recent work Hämäläinen et al. 2021b on morphological analysis, generation, and lemmatization for morphologically rich languages has focused on training recurrent neural network (RNN)-based models. A notable contribution in this area is the development of a method for automatically extracting large amounts of training data from finite-state transducers (FSTs) for 22 languages, including 17 endangered ones. These neural models are designed to follow the same tagset as the FSTs, ensuring compatibility and allowing the neural models to serve as fallback systems when used in conjunction with the FSTs. This approach enhances the accessibility and preservation of endangered languages by leveraging both neural and rule-based systems. 3 Methodology 3.1 Data Collection and Preparation Data extraction and preprocessing are particularly critical when working with an endangered language like Skolt Sami. This phase involved extracting linguistic data from available resources and transforming it into a structured format suitable for further processing. We extracted a total of 28,984 lexemes from Ve \prime rdd Alnajjar et al. 2020 , an online tool designed for editing and managing dictionaries for endangered languages. Ve \prime rdd offers a structured and efficient way to curate linguistic resources, making it an invaluable asset for our dataset creation process. The extracted lexemes included diverse entries from the dictionary, which were parsed and transformed into a tabular format for further analysis and training. This structured dataset stored each lexeme along with its POS and contextual lexical information, ensuring consistency and accessibility for subsequent processing. 3.2 Data Cleaning and Filtering Data cleaning and filtering are crucial in the context of endangered languages to ensure data quality and improve model performance. We filtered the dataset to include only nouns ( N ) and verbs

Read the complete text