Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training
H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Egemen Uzun, Atahan Uz, Elif İnce, Yusuf Erdem, Osama Shbib, Ahmed Zeer, M. Fatih Amasyali · 2024
Read Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training by H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Egemen Uzun, Atahan Uz, Elif İnce, Yusuf Erdem, Osama Shbib, Ahmed Zeer, M. Fatih Amasyali (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02775.txt.
Introduction
Toprak Kesgin and co-authors build better language models for Turkish, in a 2024 study. Most large language models train mainly on English, so they handle Turkish less well. The team set out to close that gap with smarter choices about training text and training steps.
They adapted model-made data sets and translated English ones into Turkish, then mixed these into training. The adapted models scored better on tasks with a few examples and with none. Merging several adapted models lifted results further. Human judges also rated the new models as stronger at Turkish understanding and logic questions. The recipe may help other languages that lack huge text collections.
Catalog record
| CREATOR | H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Egemen Uzun, Atahan Uz, Elif İnce, Yusuf Erdem, Osama Shbib, Ahmed Zeer, M. Fatih Amasyali |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-02775
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Egemen Uzun, Atahan Uz, Elif İnce, Yusuf Erdem, Osama Shbib, Ahmed Zeer, M. Fatih Amasyali Abstract In this study, we develop and assess new corpus selection and training methodologies to improve the effectiveness of Turkish language models. Specifically, we adapted Large Language Model generated datasets and translated English datasets into Turkish, integrating these resources into the training process. This approach led to substantial enhancements in model accuracy for both few-shot and zero-shot learning scenarios. Furthermore, the merging of these adapted models was found to markedly improve their performance. Human evaluative metrics, including task-specific performance assessments, further demonstrated that these adapted models possess a greater aptitude for comprehending the Turkish language and addressing logic-based queries. This research underscores the importance of refining corpus selection strategies to optimize the performance of multilingual models, particularly for under-resourced languages like Turkish. PubID: pubid: 979-8-3315-3149-2/24/$31.00 ©2024 IEEE I Introduction In recent years, the development of language models has been an important area of research in artificial intelligence. In particular, multilingual models have the potential to overcome data gaps in different languages and improve language learning processes [ 1 ] . To realize this potential, it is important to evaluate and optimize the performance of multilingual models. Multilingual models leverage shared knowledge across languages, allowing for more robust and comprehensive language understanding [ 2 ] . However, for these models to be effective, there must be sufficient quantity and quality of data in each language. There can be large differences between the quantity and quality of data in different languages, which can cause models to underperform in some languages. For languages with limited data sources, such as Turkish, multilingual models must be trained and optimized to overcome these shortcomings. The focus of this study is to improve the performance of Turkish language in multilingual models and enable them to produce more accurate responses. In this context, we investigate how we can improve the performance of existing multilingual models for Turkish. In this study, English datasets, which were found to enhance the performance of the models, were translated into Turkish and utilized in the training of the models. After training, the performance of the models was evaluated both by human evaluation and specifically on Turkish adapted datasets with few-shot datasets such as HellaSwag [ 3 ] and ARC [ 4 ] . In particular, Accuracy metric were used as corpus selection, model training and evaluation criteria. The results of the study demonstrate the observed improvements in the performance of Turkish language models. Furthermore, illustrates the positive effects of large-scale models on small-scale language models, as well as the benefits of synthetic and translation datasets. In conclusion, this study makes significant contributions to corpus selection methodologies and training strategies for the development of Turkish language models, with the aim of increasing the effectiveness of Turkish in the field of language technologies. The main contributions presented in this paper are as follows: • By enhancing existing corpus selection methodologies and adapting them to Turkish, we have devised novel, optimized methods for language modeling. • New adaptation datasets in Turkish were also created. New datasets, meticulously translated and harmonized from English to Turkish, were created for the purpose of training Turkish language models. • New pre-training corpora have been designed for Turkish language models, with the objective of improving model performance. • A comprehensive comparison has been conducted between existing models and models trained with the proposed method, in addition to other Turkish language models. This comparison has been conducted using both human voting and evaluations based on few-shot approaches. II Corpus Creation Zero-shot and few-shot methods are commonly employed to assess the performance of large language models. In this study, we concentrated on enhancing the performance of Turkish models on these datasets. In our initial study, we sought models that are relatively small in size but demonstrate relative proficiency on few-shot evaluation datasets. Model selection was primarily based on the Cosmopedia dataset [ 5 ] , which also served as the main training dataset for the Cosmo1b model. The Cosmopedia dataset comprises approximately 30 million files and 25 billion tokens. It was created using the Mixtral-8x7B-Instruct-v0.1 model [ 6 ] and comprises eight subsets of varying sizes, derived from sources including synthetic textbooks, blog posts, stories, posts, and WikiHow articles. The number of examples and the size of the text in each subset are provided in Table I . Furthermore, the OpenOrca dataset [ 7 ] , an open-source instruction completion dataset comprising 3.7GB of text, was employed for the model to execute human instructions. All these datasets are in English and translated into Turkish using the Google Translate API. TABLE I: Number of examples and size of text in each subset of the Cosmopedia dataset Subset Sample Size auto_math_text 1.95M 8.8GB khanacademy 24.1K 125MB openstax 126K 700MB stanford 1.02M 6.6GB stories 4.99M 21.8GB web_samples_v1 12.4M 71.9GB web_samples_v2 10.3M 30.5GB wikihow 179K 1GB Given their relatively small sizes, the Stanford, Khan Academy, WikiHow, and OpenStax datasets were combined to achieve meaningful results. This combination of datasets was used in our evaluation process as detailed in Table II . ’SKWO’ stands for the combined datasets of Stanford, Khan Academy, WikiHow, and OpenStax. Training models with parameters as large as 7 billion can present a significant challenge, due to the limitations of both hardware and time. To mitigate these challenges, we employed smaller yet effective versions of the models for initial testing phases. Accordingly, in the present study, we employed the following methodology to ascertain which types of datasets enhance the few-shot performance of LLMs. A model was trained ten times smaller with the specified dataset, and the performance of the model on few-shot datasets was compared before and after training. It is hypothesized that if a dataset can enhance the performance of a smaller model, it can also enhance the performance of a larger model. In all experiments, the ytu-ce-cosmos/turkish-gpt2-large model [ 8 ] , trained exclusively for the Turkish language with 750 million parameters, was utilized. The datasets selected for evaluation were chosen based on three criteria: (1) they are widely used, (2) they are datasets that will not lose their meaning when translated, and (3) the ytu-ce-cosmos/turkish-gpt2-large model used can perform better than random prediction. These criteria ensured that the selected datasets were both relevant and suitable for assessing the performance of the model when adapted to Turkish. The model demonstrated comparable performance to random prediction in certain datasets. Consequently, the datasets selected for evaluating the model were COPA [ 9 ] , XStoryCloze [ 10 ] , ARC Easy, ARC [ 4 ] , and HellaSwag [ 3 ] . TABLE II: Evaluation of Model Performance on Different Datasets Data Copa Xstory ARC Easy ARC HellaSwag Avg Base 60.00 55.33 37.71 23.65 36.39 42.62 SKWO 59.20 57.64 39.31 27.92 36.19 44.05 AutoMath 57.20 53.47 36.06 27.50 33.62 41.57 Stories 59.40 60.95 42.14 25.70 37.80 45.20 Web1 57.00 55.85 38.04 24.34 36.36 42.32 Web2 58.00 56.32 39.10 24.77 36.57 42.95 OpenOrca 59.40 56.05 38.47 24.77 37.22 43.18 To ensure a fair and comprehensive evaluation, ARC dataset