SozKZ : Training Efficient Small Language Models for Kazakh from Scratch
Saken Tukenov · 2026
Read SozKZ : Training Efficient Small Language Models for Kazakh from Scratch by Saken Tukenov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20854.txt.
Introduction
Tukenov presents SozKZ, a family of small language models built from scratch for Kazakh. Kazakh is a Turkic language spoken by over 22 million people. Large multilingual models give it little capacity, and their tokenizers fit its word structure poorly. SozKZ models range from 50 million to 600 million parameters, trained on 9 billion tokens of Kazakh text. The largest model nears a model twice its size on cultural questions. It beats all tested multilingual models up to 2 billion parameters on topic sorting. All models are openly released.
Catalog record
| CREATOR | Saken Tukenov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23842 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20854
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SozKZ : Training Efficient Small Language Models for Kazakh from Scratch Saken Tukenov Abstract Аннотация Kazakh, a Turkic language spoken by over 22 million people, remains underserved by existing multilingual language models, which allocate minimal capacity to low-resource languages and employ tokenizers ill-suited to agglutinative morphology. We present SozKZ , a family of Llama-architecture language models (50M–600M parameters) trained entirely from scratch on 9 billion tokens of Kazakh text with a dedicated 50K BPE tokenizer. We evaluate all models on three Kazakh benchmarks—multiple-choice cultural QA, reading comprehension (Belebele), and topic classification (SIB-200)—alongside five multilingual baselines ranging from 500M to 3B parameters. Our 600M model achieves 30.3% accuracy on Kazakh cultural QA, approaching the 32.0% of Llama-3.2-1B (2 × \times larger), and 25.5% on SIB-200 topic classification, surpassing all evaluated multilingual models up to 2B parameters. We observe consistent scaling from 50M to 600M, with MC QA accuracy rising from 22.8% to 30.3%, suggesting that further scaling remains beneficial. These results demonstrate that small, dedicated models trained from scratch with a language-appropriate tokenizer offer a viable path for low-resource language technology, achieving competitive performance at a fraction of the computational cost. All models and the tokenizer are released under open licenses. 1 Introduction Kazakh is a Turkic language spoken by approximately 22 million people, primarily in Kazakhstan and neighboring countries. As an agglutinative language with rich morphology, Kazakh poses particular challenges for natural language processing: words carry extensive inflectional and derivational suffixes, leading to a large effective vocabulary that general-purpose multilingual tokenizers handle poorly. Despite a growing body of NLP research for Kazakh [ 25 , 26 ] , the language remains underserved by the current generation of large language models, which typically allocate only a fraction of their capacity to low-resource languages. Modern multilingual models such as Llama 3 [ 23 ] , Gemma [ 9 ] , Qwen 2.5 [ 4 ] , and Mistral [ 11 ] are trained on corpora dominated by English and other high-resource languages. When applied to Kazakh, these models face two compounding inefficiencies. First, their tokenizers fragment Kazakh text into disproportionately many subword tokens—a phenomenon quantified by fertility , the average number of tokens per word. Second, the vast majority of model parameters encode knowledge about languages other than Kazakh, yielding a poor ratio of useful capacity to total model size. We argue that small, dedicated models can achieve competitive performance with much larger multilingual models on Kazakh language tasks at a fraction of the computational cost . To test this thesis, we train SozKZ , a family of Llama-architecture causal language models at four parameter scales—50M, 150M, 300M, and 600M—entirely from scratch on a curated Kazakh corpus of approximately 9 billion tokens. Our contributions are as follows: 1. We present an end-to-end pipeline for training Kazakh language models from scratch at four scales (50M–600M parameters), including data collection, cleaning, and pre-tokenization. 2. We design a dedicated ByteLevel BPE tokenizer with a 50K vocabulary 1 1 1 https://huggingface.co/stukenov/kazakh-bpe-32k that achieves a 2–3 \times fertility advantage over multilingual tokenizers on Kazakh text. 3. We evaluate SozKZ models against five multilingual baselines (Qwen, Llama 3, Gemma) spanning 0.5B–3B parameters on three Kazakh benchmarks, showing competitive performance and a clear advantage on topic classification. 4. We report empirical scaling curves for Kazakh language modeling, identifying the efficiency sweet spot where dedicated small models offer the best performance-per-parameter trade-off. 5. We release all models, the tokenizer, and the training pipeline under open licenses to support further research on Kazakh NLP. The rest of this paper is organized as follows. Section 2 surveys related work on Kazakh NLP, low-resource language modeling, scaling laws, and tokenizer design. Section 3 describes our data pipeline, tokenizer, model architectures, and training procedure. Section 4 details the benchmark suite and evaluation protocol. Section 5 presents and analyzes results. Section 6 summarizes our findings and outlines future directions. 2 Related Work 2.1 Kazakh NLP Research on Kazakh natural language processing has accelerated in recent years, driven largely by the IS2AI research group at Nazarbayev University. Yeshpanov et al. 2022 introduced KazNERD, a large-scale named entity recognition dataset for Kazakh covering 25 entity classes across news, fiction, and legal domains. Yeshpanov et al. 2024 subsequently released KazQAD, a question-answering dataset for Kazakh built from Wikipedia and educational texts. These resources have enabled systematic evaluation of NLP models on Kazakh for the first time. Beyond datasets, several efforts have targeted Kazakh language modeling directly. Multilingual models such as mBERT [ 8 ] and XLM-R [ 7 ] include Kazakh in their training data but allocate minimal capacity to it. More recently, KazLLM [ 28 ] explored continued pre-training of large models on Kazakh corpora. Our work differs in training dedicated models entirely from scratch, allowing full control over architecture, tokenizer, and data composition. 2.2 Low-Resource Language Models Training language models for low-resource languages has emerged as an active research area. Ogueji et al. 2021 demonstrated with AfriBERTa that small transformer models trained on modest corpora (less than 1 GB) can achieve competitive performance on African languages, challenging the assumption that large-scale data is a prerequisite. For Arabic, Sengupta et al. 2023 trained Jais, a 13B-parameter model from scratch on a bilingual Arabic-English corpus, showing that dedicated models outperform multilingual ones on Arabic-specific benchmarks. SEA-LION [ 3 ] adopted a similar philosophy for Southeast Asian languages, training models at multiple scales with language-specific tokenizers. The BLOOM project [ 6 ] took a different approach, training a single 176B-parameter multilingual model on 46 languages with the goal of equitable coverage. Our work follows the dedicated-model approach of Jais and AfriBERTa but at substantially smaller scales (50M–600M parameters), testing how far efficiency gains from specialization can compensate for reduced model capacity. 2.3 Scaling Laws Kaplan et al. 2020 established power-law relationships between model size, dataset size, compute budget, and cross-entropy loss for neural language models. Hoffmann et al. 2022 refined these findings with the Chinchilla scaling laws, demonstrating that many large models are significantly undertrained relative to their parameter count and that a compute-optimal model should be trained on approximately 20 tokens per parameter. In practice, the Llama model family [ 23 , 24 ] demonstrated that training smaller models on substantially more data than the Chinchilla-optimal ratio yields models that are more efficient at inference time. Our training regime follows this over-training philosophy: the SozKZ -600M model is trained on approximately 9B tokens, yielding a tokens-to-parameters ratio of roughly 15:1—slightly below the Chinchilla optimum but reflective of the available Kazakh data. These scaling laws have been derived primarily from English-language experiments. Whether they transfer to morphologically rich, low-resource languages with different tokenizer efficiencies remains an open question that our scaling analysis addresses. 2.4 Tokenizer Design Subword tokenization, introduced by Sennrich et al. 2016 with Byte-Pair Encoding (BPE) and refined by Kudo and Richardson 2018 with SentencePiece, is now standard in language model pre-training. The choice of tokenizer has outsized effects on model efficiency for morphologically rich languages [ 17 ] : a tokenizer trained on English-dominated data will fragment agglutinative languages like Kazakh, Turkish, or Finnish into far more tokens per word than necessary, effectively reducing the model’s context window and increasing inference cost per semantic unit. Fertility