Bielik v3 Small: Technical Report

Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, Adrian Gwoździej · 2025

Read Bielik v3 Small: Technical Report by Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, Adrian Gwoździej (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02550.txt.

Description

Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, and Adrian Gwoździej reported on small language models built for Polish in two thousand twenty-five. The Bielik v3 models come in two sizes and aim to match much larger models while using far less compute. Polish has less training material than English, so care in design matters.

Key steps include a custom Polish tokenizer, a loss that balances types of instruction, and a learning rate that adapts as training goes on. The models trained on two hundred ninety-two billion tokens. The larger model rivaled models two to three times its size on Polish tests, and the smaller one stayed strong for its size.

Catalog record

CREATORKrzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, Adrian Gwoździej
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-02550

Browse the full library · About the Tower of Babel