Building a Strong Instruction Language Model for a Less-Resourced Language
Domen Vreš, Tjaša Arčon, Timotej Petrič, Dario Vajda, Marko Robnik-Šikonja, Iztok Lebar Bajec · 2026
Read Building a Strong Instruction Language Model for a Less-Resourced Language by Domen Vreš, Tjaša Arčon, Timotej Petrič, Dario Vajda, Marko Robnik-Šikonja, Iztok Lebar Bajec (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01691.txt.
Introduction
Domen Vres, Tjasa Arcon, Timotej Petric, and colleagues build a strong language model for Slovene, a less-resourced language. Their 2026 work notes that open models train mostly on English, so smaller languages and cultures get weaker service. The team sets out the steps needed to adapt a model well.
Their model, GaMS3-12B, starts from an open base and learns in stages with mixed-language data. Continued training uses Slovene plus English and three close South Slavic languages. Two rounds of fine-tuning on instruction data follow, drawn from English and Slovene examples. Tests cover Slovene benchmarks, translation from English, and a head-to-head arena. The model beats its base at its size and rivals a much larger commercial model in the arena.
Catalog record
| CREATOR | Domen Vreš, Tjaša Arčon, Timotej Petrič, Dario Vajda, Marko Robnik-Šikonja, Iztok Lebar Bajec |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74533 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.01691
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Building a Strong Instruction Language Model for a Less-Resourced Language Domen Vreš, Tjaša Arčon, Timotej Petrič, Dario Vajda, Marko Robnik-Šikonja, Iztok Lebar Bajec Abstract Large language models (LLMs) have become an essential tool for natural language processing and artificial intelligence in general. Current open-source models are primarily trained on English texts, resulting in poorer performance on less-resourced languages and cultures. We present a set of methodological approaches necessary for the successful adaptation of an LLM to a less-resourced language, and demonstrate them using the Slovene language. We present GaMS3-12B, a generative model for Slovene with 12 billion parameters, and demonstrate that it is the best-performing open-source model for Slovene within its parameter range. We adapted the model to the Slovene language using three-stage continual pre-training of the Gemma 3 model, followed by two-stage supervised fine-tuning (SFT). We trained the model on a combination of 140B Slovene, English, Bosnian, Serbian, and Croatian pretraining tokens, and over 200 thousand English and Slovene SFT examples. We evaluate GaMS3-12B on the Slovenian-LLM-Eval datasets, English-to-Slovene translation, and the Slovene LLM arena. We show that the described model outperforms 12B Gemma 3 across all three scenarios and performs comparably to much larger commercial GPT-4o in the Slovene LLM arena, achieving a win rate of over 60 %. keywords GaMS, large language models, language adaptation, less-resourced language D. Vreš][email protected] 1 Introduction Large language models (LLMs), particularly generative LLMs like GPT models, have dramatically transformed natural language processing (NLP), advancing our understanding and ability to generate human language. As a result of this rapid development, new open-source decoder-type transformer LLMs such as Gemma ( 45 ) , Llama ( 9 ) , Nemotron ( 32 ) , and many others are released on a regular basis. These models are trained predominantly on high-resource languages (primarily English) while their attention and performance on less-resourced languages, such as Slovene, is limited. To amend this situation, we present the development of GaMS3-12B ( G ener a tive M odel for S lovene), an open-source generative model for Slovene with 12 billion parameters. We evaluate GaMS3-12B on a suite of Slovene benchmarks, Slovenian-LLM-Eval ( 6 ) , English to Slovene translation, and using the Slovene LLM Arena 1 1 1 https://arena.cjvt.si/en/leaderboard . We demonstrate clear improvement over the base Gemma 3 model, providing a strong indication that it is possible to enhance LLMs for many other less-resourced languages. As many such developments are ongoing or being planned, the description of our development has considerable generalization value beyond a single language. The primary challenge in training LLMs for less-resourced languages is the scarcity of data. For example, the Llama 4 model was trained on 30 trillion tokens, whereas the currently available authentic Slovene corpora contain around 40 billion tokens or approximately one thousand times less. This means that training an LLM from scratch for Slovene (and other less-resourced languages) is unfeasible. Hence, we take the pretrained Gemma 3 model and adapt it for Slovene. We perform three continual pretraining (CPT) stages, followed by two supervised fine-tuning (SFT) stages. To increase the amount of training data, we also include texts from similar languages, i.e., Croatian, Bosnian, and Serbian, which can improve the models’ performance. Additionally, we generate more data by performing an OCR of high-quality Slovene PDFs, such as the repositories of top Slovene universities and the digital archive of the national library. We also translate high-quality English corpora such as Wikipedia and Nemotron datasets. Our contributions are as follows: • GaMS, a State-of-the-art open-source generative model for Slovene; • An open-source code for the OCR of Slovene documents, that can be transferred to other less-resourced languages; • A clear description of data generation and preparation methodology, together with model training steps, making our work transferable to other less-resourced languages. Developing open-source models for less-resourced languages is crucial for preserving the autonomy, cultural identity, and sovereignty of such languages. As these languages are currently only sufficiently supported in commercial models, users who want to use them are forced to send their data to large companies. In contrast, for languages where open LLMs exist, companies and public institutions can set up open-source models on-premises, retaining full control over their customers’ data residency. Additionally, the creation of the Slovene GaMS model is open, well-documented, and offers useful lessons to other less-resourced languages. The rest of the paper is organized as follows. In Section 2 , we present relevant related work for language adaptation of large language models. We offer a detailed description of our continual pretraining approach and data in Section 3 . We describe supervised fine-tuning data and approach in Section 4 . In Section 5 , we present the training environment, hardware resources, and hyperparameters. We evaluate our model in Section 6 , and offer conclusions and ideas for future work in Section 7 . 2 Related work Most state–of–the–art language models ( 4 ; 7 ; 9 ; 45 ) are trained using the same pipeline. The models first go through the pretraining phase, where they are trained to predict the next token on huge corpora. This is followed by the supervised fine-tuning (SFT) phase, where models are trained to respond to users’ prompts. Finally, models are trained to produce outputs that align with user preferences (e.g., safer, factually correct, and helpful responses) using methods such as reinforcement learning with human feedback (RLHF). We follow a similar pipeline, but replace the reinforcement learning with chat-enhancing SFT due to a lack of preference data for Slovene. Multilingual models were developed to extend the benefits of LLMs to languages that lack sufficient training data. These models are typically based on English with additional training data in other languages. The best performing open-source multi-lingual models for Slovene are EuroLLM ( 24 ) , Gemma 2 45 , Gemma 3 ( 46 ) , DeepSeek ( 14 ) , Qwen ( 57 ) , Llama 4 ( 25 ) , Mistral ( 26 ) , and Apertus ( 2 ) . However, the models such as DeepSeek, Llama 4, and the largest versions of Mistral and Qwen are often infeasible to run on-premises due to a large number of parameters that require a significant amount of GPU vRAM. Smaller versions of Mistral and Qwen, on the other hand, perform significantly worse on Slovene and are beaten by other models of similar size. For Slovene, the most interesting family of models mentioned above is Gemma due to its strong performance and EuroLLM due to its focus on EU languages. EuroLLM was trained from scratch on a data mixture consisting of 50~\% of English texts, 5~\% of code/math data, and 45~\% of other languages. These other languages represent the majority of European languages. Slovene data represents 1~\% of the other languages’ data. The percentage of Slovene data used for pretraining of Gemma models is unclear, but it is significant enough that the 27B version of Gemma 3 performs better than the rest of multi–lingual models in the Slovenian-LLM-Eval ( 6 ) suite of benchmarks (see Table 8 ). This justifies our choice of Gemma 3 as our starting model. However, the Gemma 3 models still benefit from training on more Slovene data, as clearly shown by the gain in