BgGPT 1.0: Extending English-centric LLMs to other languages
Anton Alexandrov, Veselin Raychev, Dimitar I. Dimitrov, Ce Zhang, Martin Vechev, Kristina Toutanova · 2024
Read BgGPT 1.0: Extending English-centric LLMs to other languages by Anton Alexandrov, Veselin Raychev, Dimitar I. Dimitrov, Ce Zhang, Martin Vechev, Kristina Toutanova (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10893.txt.
Introduction
Alexandrov and co-authors present language models retrained for Bulgarian, published in 2024, built on an open English-centric model family. The team continued training two sizes of the base model on over a hundred billion tokens of Bulgarian and English text. Extra fine-tuning tuned them for instruction following. The results set a new standard on Bulgarian language tasks.
Care was taken not to wreck the original English skill. Training tricks, including a branch-and-merge schedule, preserved the base abilities while the new language took root. Most model development favors a handful of big languages; Bulgarian speakers had no top-tier local model. This project shows the adaptation recipe works for a mid-sized language. It also releases the models for public use. Other language communities can copy the playbook.
Catalog record
| CREATOR | Anton Alexandrov, Veselin Raychev, Dimitar I. Dimitrov, Ce Zhang, Martin Vechev, Kristina Toutanova |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10893
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BgGPT 1.0: Extending English-centric LLMs to other languages Anton Alexandrov, Veselin Raychev, Dimitar I. Dimitrov, Ce Zhang, Martin Vechev, Kristina Toutanova Abstract We present BgGPT-Gemma-2-27B-Instruct and BgGPT-Gemma-2-9B-Instruct : continually pretrained and fine-tuned versions of Google’s Gemma-2 models, specifically optimized for Bulgarian language understanding and generation. Leveraging Gemma-2’s multilingual capabilities and over 100 billion tokens of Bulgarian and English text data, our models demonstrate strong performance in Bulgarian language tasks, setting a new standard for language-specific AI models. Our approach maintains the robust capabilities of the original Gemma-2 models, ensuring that the English language performance remains intact. To preserve the base model capabilities, we incorporate continual learning strategies based on recent Branch-and-Merge techniques as well as thorough curation and selection of training data. We provide detailed insights into our methodology, including the release of model weights with a commercially-friendly license, enabling broader adoption by researchers, companies, and hobbyists. Further, we establish a comprehensive set of benchmarks based on non-public educational data sources to evaluate models on Bulgarian language tasks as well as safety and chat capabilities. Our findings demonstrate the effectiveness of fine-tuning state-of-the-art models like Gemma 2 to enhance language-specific AI applications while maintaining cross-lingual capabilities. 1 Introduction 0 0 footnotetext: ∗ Correspondence author: [email protected] Table 1: Performance of open large language models on 10 Bulgarian benchmarks, as well as the average scores for the English version of the bencmarks. Model English Bulgarian Standard LLM evaluation benchmarks translated to Bulgarian Native Bulgarian benchmarks (average) (average) Wino* Hellaswag Arc-C† Arc-E† MMLU MathQA GSM-8K TriviaQA MON EXAMS Gemma-2-2b-it 57.29 34.92 57.9 45.7 31.4 42 41.7 28.6 25.9 9.5 34.2 32.3 Mistral-7B-Instruct-v0.2 61.54 43.14 60.7 56.4 39.3 55.2 47.3 32.5 29.7 24.5 43.5 42.2 BgGPT-Instruct-7B-v0.2 62.89 54.36 69.7 69.6 49.7 69.5 53.8 35.2 56.9 40.9 48.9 49.3 Mistral-Nemo-Instruct-2407 (12B) 69.11 50.24 66.8 56.7 42.2 59.6 56 35.6 58.8 31.6 45.0 50.1 Qwen2.5-7B-Instruct 62.22 40.2 57.7 47.1 34 45.3 57.6 39.7 35.3 10.4 38.4 36.4 Llama-3.1-8B-Instruct 67.89 46.69 60.6 52.8 36.9 54.9 52.5 38.4 57.6 25.5 42.2 45.5 Gemma-2-9b-it 70.92 55.36 70 64.6 48.2 64.6 63.1 39.2 75.7 31.3 47.3 49.6 BgGPT-Gemma-2-9B-Instruct 71.53 61.27 72.6 71.5 54.2 75 64.4 45 73.8 43.6 52.7 \mathbf{59.7} Gemma-2-27b-it 75.08 60.51 72.1 71 49.9 63.4 69.8 43.1 81 46.8 51.2 56.7 BgGPT-Gemma-2-27B-Instruct 75.12 \mathbf{64.67} \mathbf{73.5} \mathbf{74.9} \mathbf{59.3} \mathbf{78.1} 69.8 47 77.3 \mathbf{52.9} \mathbf{54.2} 59.2 Qwen2.5-72B-Instruct 77.53 58.25 67.4 66.2 44.3 56.5 \mathbf{78.8} \mathbf{58.0} \mathbf{81.7} 31.8 46.0 51.8 Llama-3.1-70B-Instruct \mathbf{77.75} 61.49 67.6 67.3 52.4 70.2 75.3 53.9 68.5 51.2 52.2 56.1 * - Winogrande challenge, †- ARC-Challenge and ARC-Easy Large Language Models (LLMs) have shown remarkable capabilities, particularly in the English language, due to the abundance of digitalized English text. Moreover, with recent developments ( Grattafiori et al., 2024 ; Gemma-Team et al., 2024 ; Team, 2024 ; Yang et al., 2024 ) we are seeing that the open-source/weight community has been closing the gap compared to closed, black box platforms in most tasks. A large portion of LLM users, however, are not native English speakers or require text generation in low-resource languages, and English-centric models do not always meet their needs. Some such models do exhibit strong multi-lingual understanding, stemming from their large, diverse pretraining data, especially in more popular European languages, but often do not meet the required standard for text generation nowadays. This work presents the development process and evaluation of our language models, specialized for Bulgarian and English use, named BgGPT-Gemma-2 , based on the Gemma-2 architecture ( Gemma-Team et al., 2024 ) . This is the third iteration of Bulgarian models developed by INSAIT , designed to be powerful yet accessible on common consumer hardware. Advancing Bulgarian while retaining base capabilities The BgGPT-Gemma-2 models are trained through continued pretraining of strong base models not specialized in Bulgarian. Our training data also includes English data but has an emphasis on Bulgarian local knowledge and language understanding. Since the base models demonstrate strong performance in English, coding, mathematics, and other domains, we have applied Branch-and-Merge ( Alexandrov et al., 2024 ) , which significantly reduces catastrophic forgetting from the base model while minimally impacting the learning of Bulgarian capabilities. Our training procedure also incorporates techniques such as curriculum learning and experience replay ( Scialom et al., 2022 ; Zhang et al., 2023 ; Ibrahim et al., 2024 ) . Challenges and goals Previous releases of open models such as Llama, Llama-2, Mistral 7B and even GPT-2 Touvron et al. (2023a) ; Touvron et al. (2023b) ; Jiang et al. (2023) ; Radford et al. (2019) have led to numerous specialized models targeting non-English languages Plüster (2023) ; Nikolich et al. (2024) ; Vo et al. (2024) ; Toraman (2024) ; Nguyen et al. (2023) ; Csaki et al. (2023) ; Masala et al. (2024) ; Aleksa (2024) ; Bacciu et al. (2023) . Even though the best open-weight foundational models are mostly English-centric, they exhibit reasonable multi-lingual understanding and even often manage to generate, from a linguistic point of view, excellent text in other languages. We find that with the emergence of more capable models like Llama-3.1 and Gemma-2, language-specific fine-tuning has become more demanding in order to produce significant improvements without decreasing the base English performance ( Choudhury et al., 2024 ) . Our primary objective with BgGPT-Gemma-2 is to enhance the Bulgarian capabilities of a state-of-the-art model while maintaining its strong performance across other tasks. At the same time, we observe that even powerful models exhibit reduced mathematical, few-shot, knowledge retrieval and other capabilities when prompted in Bulgarian (see section 5.1 ), instead of English. Additionally, sometimes these models default to English responses or exhibit language confusion ( Marchisio et al., 2024 ) for Bulgarian prompts. This work focuses on addressing these challenges to produce the best existing open Bulgarian models. Model capabilities Our primary results for Bulgarian are presented in Table 1 . We evaluate on a set of 10 standard benchmarks, translated to Bulgarian using machine translation with manual edits and corrections by professional translators, or coming from authentically Bulgarian sources. Each benchmark has a maximum score of 100.0 , which would correspond to all tasks being correctly completed. Eight of the benchmarks contain multiple-choice questions, whose answers are computed using token probability and two evaluate the models’ ability to generate text. Since log probabilities are typically not available for commercial models, we only include open models in this table. The benchmarks are publicly available 2 2 2 https://github.com/insait-institute/lm-evaluation-harness-bg and can be executed on any open model. We provide more details about our benchmarking settings in Section 4 . Overall, BgGPT-Gemma-2 performs well on Bulgarian tasks, surpassing even larger models such as Qwen-2.5-72B ( Team, 2024 ) and Llama-3.1-70B in specific benchmarks. Among models larger than BgGPT-Gemma-2-27B-Instruct , we observe varying levels of Bulgarian proficiency. Specifically, Qwen-2.5 excels in mathematics, coding, and logical reasoning, while Llama-3.1 demonstrates superior performance in general language understanding and knowledge. At approximately 27 B parameters, BgGPT-Gemma-2-27B-Instruct achieves the highest average performance on Bulgarian tasks while maintaining most of the English capabilities of its base model. In the 7 - 12 B parameter range, BgGPT-Gemma-2-9B-Instruct outperforms all models on Bulgarian tasks and also overall on English ones. Applications The BgGPT-Gemma-2 models are available for download with a license