Krikri: Advancing Open Large Language Models for Greek

Dimitris Roussis, Leon Voukoutis, Georgios Paraskevopoulos · 2025

Read Krikri: Advancing Open Large Language Models for Greek by Dimitris Roussis, Leon Voukoutis, Georgios Paraskevopoulos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13772.txt.

Introduction

Dimitris Roussis and co-authors released a language model for Greek in two thousand twenty-five. The model, named Krikri, builds on an open model from Meta. It was trained heavily on high-quality Greek text. It handles modern Greek and English side by side. It can also read older polytonic and ancient Greek text.

The chat version went through several rounds of extra training. Those rounds used both human and synthetic teaching data. The team also built three new public tests for Greek. Scores on old and new tests gauge its skill. An open model of this strength aids Greek schools and firms. It also supports research on less-served languages. The release includes the benchmarks for others to use.

Catalog record

CREATORDimitris Roussis, Leon Voukoutis, Georgios Paraskevopoulos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13772

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Krikri: Advancing Open Large Language Models for Greek Dimitris Roussis, Leon Voukoutis, Georgios Paraskevopoulos Abstract We introduce Llama-Krikri-8B, a cutting-edge Large Language Model tailored for the Greek language, built on Meta’s Llama 3.1-8B. Llama-Krikri-8B has been extensively trained on high-quality Greek data to ensure superior adaptation to linguistic nuances. With 8 billion parameters, it offers advanced capabilities while maintaining efficient computational performance. Llama-Krikri-8B supports both Modern Greek and English, and is also equipped to handle polytonic text and Ancient Greek. The chat version of Llama-Krikri-8B features a multi-stage post-training pipeline, utilizing both human and synthetic instruction and preference data created via various synthesis techniques, such as MAGPIE, in order to address the issue of data scarcity for Greek. In addition, for evaluation, we propose three novel public benchmarks for Greek which are suitable for instruction-tuned models. Our evaluation on existing as well as the proposed benchmarks shows notable improvements over comparable Greek and multilingual LLMs in both natural language understanding and generation as well as tasks related to math, code, instruction-following, etc. 1 Introduction Recent advancements in AI have been largely driven by the development of large-scale foundation models. Meta’s Llama 3 Grattafiori et al. (2024) fostered a new generation of open models, designed for strong multilingual capabilities, code generation, reasoning, and tool use. With extended context windows, and refined training strategies, models based on Llama 3 have achieved performance comparable to proprietary systems like GPT-4. A critical aspect in this evolution is the development of multilingual and language-specific models, democratizing access to AI technologies and preserving linguistic diversity. While substantial progress has been made for widely spoken languages, low and medium resource languages remain underrepresented. Greek, in particular, has received limited attention despite its linguistic complexity, rich cultural heritage, and historical significance. Addressing this gap, we present Llama-Krikri-8B, a cutting-edge open Large Language Model tailored for the Greek language. Built on Meta’s Llama 3.1-8B architecture, Llama-Krikri has been continually pretrained on a diverse, high-quality Greek corpus. This allows the model to effectively capture the syntactic and semantic nuances of Greek, while retaining the multilingual strengths of the base model. Notably, Llama-Krikri also supports English and is capable of handling polytonic and Ancient Greek texts, addressing not only contemporary but also historical forms of the language. Compared to Meltemi-7B Voukoutis et al. (2024) , the previous state-of-the-art open Greek LLM built on Mistral 7B Jiang et al. (2023) , Llama-Krikri-8B significantly increases the number of parameters, context length, and training data scale. Additionally, it features an enhanced post-training pipeline using both human and synthetic data. By utilizing various data synthesis methodologies, such as MAGPIE Xu et al. (2024) , we generate high-quality synthetic data for Supervised Fine-Tuning (SFT) and alignment via Direct Preference Optimization (DPO) Rafailov et al. (2024) . This pipeline addresses the scarcity of relevant data for Greek and ensures that the model produces helpful, honest, and harmless outputs. To evaluate Llama-Krikri-8B, we also introduce three novel public benchmarks specifically designed for Greek instruction-tuned models. These, alongside existing evaluation suites, show that Llama-Krikri outperforms comparable Greek and multilingual LLMs in both natural language understanding and generation, as well as other challenging tasks related to math, code, instruction-following, etc. Moreover, it supports function calling and agentic behavior, opening new application domains for Greek users. Llama-Krikri-8B is available under the Llama 3.1 Community License Agreement 1 1 1 https://www.llama.com/llama3_1/license/ . Our key contributions are: • We present Llama-Krikri-8B, a state-of-the-art open Greek foundation model based on Llama 3.1, demonstrating strong capabilities in Modern and Ancient Greek, math, code, instruction-following, etc., while also supporting function calling and agentic behavior. • We incorporate additional steps aiming to increase efficiency, reduce training costs, and improve performance, such as expanding the original Llama 3.1 tokenizer and embeddings, and conducting an annealing phase with high-quality and synthetic data. • We collect and curate a large pretraining corpus and utilize synthetic data creation strategies for post-training, in order to address the scarcity of Greek data. • We implement a multi-step post-training pipeline incorporating Supervised Fine-Tuning (SFT) and alignment through Direct Preference Optimization (DPO), which leads to Llama-Krikri-8B-Instruct surpassing Llama-3.1-8B-Instruct in several English tasks. • We introduce three new benchmarks for evaluating instruction-tuned Greek LLMs, covering instruction-following, multi-turn chats, code/math tasks, etc. • We show that Llama-Krikri-8B and Llama-Krikri-8B-Instruct significantly outperform existing open multilingual models in Greek tasks across several domains, while also being competitive with 3-4 times larger models. 2 Background and Related Work Large Language Models (LLMs) have achieved state-of-the-art performance across a wide variety of natural language processing (NLP) tasks. These models are typically trained on massive corpora dominated by English, leading to strong performance in English-language tasks but comparatively weaker capabilities in other languages Devlin et al. (2019) ; Brown et al. (2020) . As a result, the development of language-specific LLMs has become an active area of research, particularly for under-represented languages. One prominent strategy for developing such models is continual pretraining, where a pretrained base model is further trained on data in the target language. This approach allows researchers to leverage the general capabilities of large base models while improving performance in specific linguistic domains, without the prohibitive cost of training from scratch Gururangan et al. (2020) . Several recent projects have successfully applied continual pretraining to adapt existing models to new languages. BgGPT-GEMMA-2-27B-Instruct Alexandrov et al. (2024) fine-tunes Google’s Gemma-2 model Riviere et al. (2024) for Bulgarian, combining over 100B tokens of Bulgarian and English data and applying techniques such as Branch-and-Merge to mitigate catastrophic forgetting. Similarly, LeoLM LAION (2023) adapts a Llama-based architecture for German, while the Sabiá Pires et al. (2023) and Bielik models Ociepa et al. (2024) adapt Mistral-based architectures for Portuguese and Polish respectively, through targeted continual pretraining and post-training. For Greek, Meltemi-7B represents the first open generative LLM tailored to the language Voukoutis et al. (2024) . It was developed by continually pretraining Mistral-7B on a substantial Greek corpus, followed by instruction fine-tuning. While effective, Meltemi’s performance is bounded by the size and capabilities of the base model, data scarcity for instruction tuning in Greek, as well as the limited post-training alignment techniques employed at the time. Beyond language adaptation, alignment of LLMs to generate helpful, harmless, and honest outputs has become increasingly central. Early approaches such as InstructGPT Ouyang et al. (2022) and Constitutional AI Bai et al. (2022) rely on multi-stage fine-tuning pipelines involving human feedback or rule-based constraints. More recently, DPO and data synthesis methods like MAGPIE have enabled scalable and effective post-training. MAGPIE, in particular, leverages already-aligned models to generate large volumes of instruction-response pairs, demonstrating that synthetic data can rival or surpass human-curated datasets. These advancements highlight a trend toward bootstrapping high-quality training data using strong base models, especially in low-resource languages. Our work builds on this foundation by employing Llama 3.1 as a base architecture, employing various data synthesis techniques (e.g., MAGPIE), and applying a more rigorous post-training pipeline, including DPO for alignment. Original Upsampled Subcorpus Tokens (B) Percentage Tokens (B) Percentage Greek 56.7 62.3% 66.1 60.0% English 21.0 23.1% 25.2 22.9% Parallel 5.5 6.0% 8.8 8.0% Math/Code 7.8 8.6% 10.1 9.1% Total 91.0 100% 110.2 100% Table 1: Composition of the pretraining corpus - original and upsampled 3

Read the complete text