YAD: Leveraging T5 for improved automatic diacritization of Yorùbá text

Akindele Michael Olawole, Jesujoba O. Alabi, Aderonke Busayo Sakpere, David I. Adelani · 2024

Read YAD: Leveraging T5 for improved automatic diacritization of Yorùbá text by Akindele Michael Olawole, Jesujoba O. Alabi, Aderonke Busayo Sakpere, David I. Adelani (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20218.txt.

Description

A 2024 arXiv research paper by Akindele Michael Olawole, Jesujoba O. Alabi, Aderonke Busayo Sakpere, David I. Adelani. In this work, we present Yorùbá automatic diacritization (YAD) benchmark dataset for evaluating Yorùbá diacritization systems. In addition, we pre-train text-to-text transformer, T5 model for Yorùbá and showed that this model outperform several multilingually trained T5 models.

Catalog record

CREATORAkindele Michael Olawole, Jesujoba O. Alabi, Aderonke Busayo Sakpere, David I. Adelani
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE11 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20218

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

YAD: Leveraging T5 for improved automatic diacritization of Yorùbá text Akindele Michael Olawole, Jesujoba O. Alabi, Aderonke Busayo Sakpere, David I. Adelani Abstract In this work, we present Yorùbá automatic diacritization (YAD) benchmark dataset for evaluating Yorùbá diacritization systems. In addition, we pre-train text-to-text transformer, T5 model for Yorùbá and showed that this model outperform several multilingually trained T5 models. Lastly, we showed that more data and larger models are better at diacritization for Yorùbá Introduction Yorùbá, a language spoken predominantly in West Africa, is renowned for its tonal nature which is characterized by a heavy use of diacritics to signify tone variations. In Yorùbá and many other languages, diacritics play a crucial role in disambiguating word meanings and in word pronunciation, making accurate diacritization essential for effective communication and language processing tasks ( Skiredj & Berrada 2024 ) . However, manual diacritization is time-consuming and requires specialized linguistic expertise, motivating the development of automatic diacritization systems. In recent years, significant progress has been made in natural language processing (NLP) techniques, leading to the exploration of various approaches to automate the diacritization process for languages using diacritics ( Náplava et al. 2018 ; Mubarak et al. 2019 ; Náplava et al. 2021 ; Stankevicius et al. 2022 , inter alia ) including Yorùbá ( Orife 2018 ; Orife et al. 2020 ) . Despite these efforts, there is the absence of a standard benchmark dataset for evaluating and comparing developed Yorùbá diacritization systems. Furthermore, the emphasis has predominantly been on model development rather than assessing the usability of these systems, particularly in the context of lightweight models amidst the prevalence of large neural networks in contemporary NLP research. In this study, we address these gaps by introducing YAD (Yorùbá Automatic Diacritization Dataset), which we curated by leveraging the MENYO-20k ( Adelani et al. 2021 ) dataset. Moreover, we focus on developing lightweight diacritizers for Yoruba, leveraging T5 ( Raffel et al. 2020 ) a text-to-text transformer architecture, to improve usability and efficiency in the era of large neural networks. For reproducibility, we release our code and data on GitHub. 1 1 1 https://github.com/ajesujoba/YAD Yorùbá language Yorùbá orthography is based on the Latin alphabet. Its alphabets include 25 Latin letters excluding characters c, q, v, x and z, and its has five additional letter (ẹ, gb, ṣ , ọ) representing certain sounds. Diacritics including accents and undots are used in Yorùbá to capture tone and vowel height. The tones include low, middle and high which are denoted by the grave (e.g. “à”), optional macron (e.g. “ā”) and acute (e.g. “á”) accents respectively. Yorùbá automatic diacritization dataset (YAD) In this work, we present YAD, a dataset created for evaluating diacritization systems for Yorùbá. YAD is constructed based on the MENYO-20k dataset, which is a benchmark for English-Yorùbá translation, containing 20,000 parallel sentences. We divided the Yorùbá side of MENYO-20k’s test data into two halves, allocating one half as development set (Dev.) and the other half as test set (Test). We ensured that both halves of the data had equal representation across all domains present in MENYO-20k. Given these splits, we applied four heuristics on the data, considering the sequence-to-sequence nature of the diacritization task. This task requires a source side and a target side, where the source side can consist of Yorùbá text without diacritics or partially diacritized texts, and the target side is fully diacritized Yorùbá texts. To create and prepare the source side of the data, the following heuristics were applied with specific proportions on the original, well-diacritized texts: (i) removal of all diacritics for 60\% of the data, (ii) removal of only tone marks for 20\% of the data, (iii) a combination of (ii) and the removal of any random word for 10\% , and (iv) a combination of (ii) and swapping any two distinct words (applicable only for sentences with more than two words). Where (iii) and (iv) can be seen as infilling and grammatical error correction tasks respectively. Example Yorùbá automatic diacritization dataset (YAD) , shows an example from the development set where (iv) was applied where sibẹsibẹ, and Ṣugbọn were swapped on the source side. \enumsentence source : sibẹsibẹ, Ṣugbọn Mama o gbagbọ. reference : Ṣùgbụ́n síbẹ̀síbẹ̀, Màmá ò gbàgbọ́ Diacritization model We pre-train T5 models from scratch using Yorùbá WURA corpus ( Oladipo et al. 2023 ) and JW300 ( Agić & Vulić 2019 ) with different sizes: tiny (number of layers encoder/decoder (L=4), heads (H=4)) , mini (L=4, H=4), small (L=8, H=6), base (L=12, H=12) corresponding to 14M, 18M, 60M and 280M parameters. All models make use of a vocabulary size of 32K except the tiny model. We called our model o yo -T5. 2 2 2 Oyo Yorùbá (YO) dialect of T5. Oyo dialect is the standardized dialect for writing Yorùbá . All models were pre-trained using HuggingFace Flax code and TPU v3-8 for 100-200K steps, which took around one day each. After pre-training, we trained our baseline diacritizer models by fine-tuning existing multilingually trained language models, MT5 ( Xue et al. 2021 ) , AfriMT5 ( Adelani et al. 2022 ) , AfriTeVa-V2 ( Oladipo et al. 2023 ) , and UMT5 ( Chung et al. 2023 ) . We use MT5-base, Afri-MT5-base, the base and large variants of AfriTeVa-V2, and UMT5. we fine-tune these models using HuggingFace Transformers ( Wolf et al. 2020 ) . These eventual diacritization models are evaluated using the test and development SacreBLEU ( Post 2018 ) 3 3 3 “intl” tokenizer, all data comes untokenized. and ChrF ( Popović 2015 ) . Training and evaluation data For our experiments, we trained diacritization models using data from three sources, which are, the Yorùbá Bible, combination of training and development split of MENYO-20k (now referred to as YAD training split from here onwards), and JW300 ( Agić & Vulić 2019 ) . Table 2 shows the number of sentences in each split. Different combinations of these datasets were used for training our models, and unless specified otherwise we implemented the four previously described heuristics on the training data also. We evaluated our models on the YAD development and test splits, as well as on an existing Global Voices (GV) test set from Orife et al. 2020 . Additionally, we used a sampled 10\% of the Bible as a test set. For reproducibility, we release our code, data and models on GitHub 4 4 4 https://github.com/ajesujoba/YAD . Baseline models and result We trained diacritization models by fine-tuning MT5, Afri-MT5, AfriTeVa-V2, and UMT5, on the YAD training split. The experimental results as presented in Table 2 shows that AfriTeVa-V2-large outperform other massively multilingual language models. Interestingly, Oyo-T5-base demonstrates competitiveness on both BLEU and CHRF similar to AfriTeVa-V2-large, despite having fewer parameters compared to AfriTeVa-V2-large. Corpus Train Dev Test Bible 27739 - 3083 JW300 459,871 - - YAD 13,467 3,326 3,330 Previous test set: Orife et al. 2020 GV - - 619 Table 1: Data split of different corpus. Dev. eval. Test eval. Model Size BLEU CHRF BLEU CHRF UMT5-base 580M 38.1 62.5 39.3 64.0 MT5-base 580M 48.1 69.4 49.5 71.2 AfriMT5-base 580M 53.3 72.8 54.0 74.2 AfriTeVa-base 313M 56.9 72.9 57.2 74.0 AfriTeVa-large 871M 67.2 80.8 66.8 81.3 Oyo-T5-base 280M

Read the complete text