Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series

Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, Adrian Gwoździej · 2026

Read Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series by Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, Adrian Gwoździej (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10799.txt.

Introduction

Ociepa and four colleagues advance Polish language modeling through tokenizer tuning in the Bielik v3 7B and 11B models. One-size-fits-all tokenizers miss Polish word-shape nuances, raising token counts and running costs while shrinking usable context windows. The report details the move from Mistral token sets to a dedicated Polish-optimized vocabulary. It covers FOCUS-based vector setup and a multi-stage training plan. Post-training tuning uses supervised fine-tuning, direct preference optimization, and group-based policy optimization with reward checks.

Catalog record

CREATORKrzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, Adrian Gwoździej
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52057
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10799

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, Adrian Gwoździej Abstract The development of the Bielik v3 PL series, encompassing both the 7B and 11B parameter variants, represents a significant milestone in the field of language-specific large language model (LLM) optimization. While general-purpose models often demonstrate impressive multilingual capabilities, they frequently suffer from a fundamental architectural inefficiency: the use of universal tokenizers. These tokenizers, typically designed to cover a broad spectrum of languages, often fail to capture the morphological nuances of specific languages like Polish, leading to higher fertility ratios, increased inference costs, and restricted effective context windows. This report details the transition from the universal Mistral-based tokenization to a dedicated Polish-optimized vocabulary for the Bielik v3 models, exploring the FOCUS-based embedding initialization, the multi-stage pretraining curriculum, and the subsequent post-training alignment involving Supervised Fine-Tuning, Direct Preference Optimization, and Reinforcement Learning through Group Relative Policy Optimization with verifiable rewards. 1 Introduction Recent years have witnessed rapid progress in the development of large language models (LLMs), including a growing focus on languages that remain underrepresented in global AI systems. Within Europe, multiple initiatives have emerged to support linguistic diversity and improve access to high-quality language technologies across a wide range of languages. Our work extends the Bielik model family, building upon the Bielik 11B v3 model 21 and the Bielik Minitron 7B v3 model 13 . The approach presented in this paper leverages prior experience and methodologies developed during earlier iterations of smaller Bielik v3 models 22 . This research is aligned with broader European efforts aimed at advancing multilingual and accessible AI systems. Notable examples include EuroLLM 18 , which focuses on multilingual capabilities across European Union languages, Apertus 10 , which promotes open and inclusive LLM development, and the PLLuM family 15 , which targets Polish language modeling specifically. In this paper, we introduce Bielik v3 PL models in both 7B and 11B parameter configurations, designed with a tokenizer optimized specifically for the Polish language. The main contributions of this work are as follows: • We propose a method for replacing the tokenizer using the FOCUS 7 approach, while mitigating the risk of catastrophic forgetting. • We describe a comprehensive multi-stage pre-training and post-training pipeline that preserves performance comparable to models using the original tokenizer. • We release the weights of both models under the Apache 2.0 license. 2 Model Architecture The Bielik v3 family is based on the Transformer architecture 37 , adopting and extending key design principles introduced in the Mistral 7B models 11 . The architecture incorporates several optimizations aimed at improving both computational efficiency and training robustness, while maintaining strong performance across a wide range of tasks. A central component of the design is the use of Grouped-Query Attention (GQA) 1 , which reduces memory bandwidth usage and computational overhead during inference. This is achieved by sharing key-value projections across multiple query heads, effectively lowering the number of key-value heads without significantly impacting model quality. This approach has become a standard technique in modern efficient large-scale models, particularly for handling long input sequences. To support extended context lengths, the models employ Rotary Positional Embeddings (RoPE) 34 , which provide improved generalization of positional information compared to traditional embedding methods. This enables the Bielik v3 models to operate with a native context window of up to 32,768 tokens, while preserving sensitivity to token order over long sequences. The flagship Bielik 11B v3 model 21 was scaled using the Depth Up-Scaling (DUS) strategy 12 . Starting from a 32-layer Mistral-based backbone, the architecture was expanded by duplicating layers, followed by a structured reduction in which selected layers from both the lower and upper parts of the network were removed. This process resulted in a 50-layer model, balancing increased representational capacity with practical deployment constraints. The final architecture was specifically designed to fit within the memory limits of widely available 24 GB GPUs, while providing sufficient depth for advanced reasoning capabilities. The Bielik Minitron 7B v3 model 13 was obtained through compression of the 11B variant rather than being trained independently. This process combined structured pruning with knowledge distillation, enabling a substantial reduction in model size and computational requirements. As a result, the compressed model retains approximately 90% of the original model’s performance, while achieving up to 50% faster inference. This approach significantly lowers both development cost and environmental impact, and demonstrates an effective strategy for building high-quality models for underrepresented languages. The Bielik v3 PL variants retain the same architectural design as their base models, differing only in the tokenizer and vocabulary, which are specifically adapted for the Polish language. 3 Tokenizers Tokenization defines the boundary between raw text and its numerical representation, making it a critical component of any language model. Its design is particularly important for morphologically rich languages such as Polish, which exhibits complex inflectional patterns, frequent use of diacritics, and a high degree of lexical variation. In such settings, suboptimal tokenization can significantly degrade model efficiency and performance. General-purpose tokenizers, including those used in models such as Mistral 7B, are typically optimized for multilingual coverage rather than language-specific efficiency. As a result, Polish text is often segmented into an excessive number of subword units. This behavior is commonly measured using the fertility ratio , defined as the average number of tokens required to represent a given text 29 . A high fertility ratio leads to reduced information density within the context window and increased computational cost during inference. At the opposite end of the design spectrum are tokenizers with very large vocabularies, often ranging from 150k to 250k tokens. While such approaches can reduce fragmentation, they introduce significant overhead in terms of model size and memory consumption. In monolingual or language-focused applications, a substantial portion of these embeddings remains unused, resulting in inefficient utilization of both memory and compute resources, as well as slower inference. The original Bielik v3 tokenizer employed a vocabulary of 32,128 tokens. Although effective, it frequently required multiple tokens to encode single Polish words that could otherwise be represented more compactly. This limitation negatively impacts both context utilization and generation speed. To address this issue, the Bielik v3 PL models adopt a dedicated Polish tokenizer with a comparable vocabulary size of 32,000 tokens. The primary objective of this design is to reduce the fertility ratio for Polish while preserving reasonable coverage of English and other European languages. Beyond vocabulary size, the segmentation strategy itself plays a crucial role. In particular, the handling of digits, punctuation, and special characters can influence both token efficiency and downstream generation quality. Taking these factors into account, we developed and adopted the APT4 tokenizer, which extends and refines the design of the earlier APT3 tokenizer introduced with the Polish APT3 model 20 . To quantitatively evaluate tokenizer performance, we use the preamble of the Constitution of the Republic of Poland (see Appendix A ) as a benchmark. This text provides a representative example of formal Polish, characterized by complex syntax and rich morphology, while its official English translation enables controlled cross-linguistic comparison. Table 1 reports key efficiency metrics, including the total number of tokens, characters per token (CpT), and

Read the complete text