LLMic: Romanian Foundation Language Model

Vlad-Andrei Bădoiu, Alexandru M. Gherghescu, Alexandru Agache, Mihai-Valentin Dumitru, Costin Raiciu · 2025

Read LLMic: Romanian Foundation Language Model by Vlad-Andrei Bădoiu, Alexandru M. Gherghescu, Alexandru Agache, Mihai-Valentin Dumitru, Costin Raiciu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-07721.txt.

Introduction

Vlad-Andrei Badoiu and co-authors present a computing paper from 2025 on a language model for Romanian. Big open models often lag in tongues with little training text. Romanian is one such case. The team built a bilingual base model tuned to Romanian from the start.

The paper records the full build: gathering the text corpus, picking the model shape, and tuning its settings. After base training, the model was fine-tuned for translation. On English-to-Romanian tasks it beat the existing tools the team compared. Its scores also rivaled much larger open models on Romanian tasks. The authors say a small, focused model can serve a language community well at low cost.

Catalog record

CREATORVlad-Andrei Bădoiu, Alexandru M. Gherghescu, Alexandru Agache, Mihai-Valentin Dumitru, Costin Raiciu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-07721

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLMic: Romanian Foundation Language Model Vlad-Andrei Bădoiu, Alexandru M. Gherghescu, Alexandru Agache, Mihai-Valentin Dumitru, Costin Raiciu Abstract . Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks with commercial models leading the way. While open models usually operate at a smaller scale, they maintain competitiveness through specialization and fine-tuning. However, a significant challenge persists: open models often underperform in low-resource languages due to limited representation in the training corpus. In this paper, we present LLMic, a bilingual foundation language model designed specifically for the Romanian Language. We document the complete process of pretraining a foundation model for a low-resource language, including corpus construction, architecture selection, and hyper-parameter optimization. Our evaluation demonstrates that LLMic can be specialized for tasks in the target language, achieving results comparable to other much larger open models. We show that fine-tuning LLMic for language translation after the initial pretraining phase outperforms existing solutions in English-to-Romanian translation tasks. This opens the path for efficient large-scale processing for the Romanian language community, using the much smaller LLMic model. 1. Introduction The emergence of Large Language Models (LLMs) has transformed the landscape of natural language processing. We are witnessing an ever-growing number of open and proprietary models. Leveraging hyperscale computing infrastructure, commercial models such as GPT-4, Claude, and Gemini attained the trillion-parameters threshold ( Hesse, 2024 ) achieving never seen before capabilities ( Brown et al., 2020 ; Wei et al., 2022 ) . At this size, models are great general purpose problem-solvers, demonstrating sophisticated reasoning. On the open models front, despite operating at significantly lower parameter counts, we have a vibrant ecosystem ( Minaee et al., 2024 ) . Models such as Llama ( Dubey et al., 2024 ) , Deepseek ( Liu et al., 2024 ) , Qwen ( Bai et al., 2023 ) and Mistral ( Jiang et al., 2023 ) are able to be competivitve on a variety of tasks ( Chiang et al., 2024 ) . Through fine-tuning techniques, these open models can be adapted further, for state-of-the-art task-specific performance. The advantages of open models are multiple: reduced operational costs, efficient processing of large-scale data through specialized compact models, privacy through self-deployment, and low memory footprint enabling running on edge devices such as smartphones. A significant challenge in language model development lies in achieving consistent performance across diverse languages. Open-source models, constrained by their reduced parameter counts and English-centric pretraining datasets, typically underperform on low-resource languages compared to commercial alternatives ( Ali and Pyysalo, 2024 ) . While open models leverage public datasets like Dolma ( Soldaini et al., 2024 ) and FineWebEdu ( Penedo et al., 2024 ) , such collections predominantly target widely spoke languages , with limited representation for other languages. While fine-tuning can enhance a model’s capabilities in specific languages ( Masala et al., 2024 ; Garcia et al., 2024 ) , the results are underwhelming compared to the performance in languages that dominate the pretraining data. While fine-tuning can be used to improve the capabilities in specific languages ( Masala et al., 2024 ; Garcia et al., 2024 ) performance, it does provide underwhelming results compared to the languages that make most of the pretraining dataset. However, recent research has shown that it is possible to classify an LLM’s neurons into “language-specific” and “language-agnostic” categories ( Tang et al., 2024 ) , whith language specific-specific neurons being only a tiny fraction of the entire model. The authors of ( Tang et al., 2024 ) go as far as showing that, through manual intervention in the LLM’s inference process, activations of language-specific neurons can be modified to steer the model towards a particular language. This finding has important implications: it suggests that strong cross-lingual performance on non-linguistic tasks might be achievable by combining a comprehensive English “knowledge corpus” with relatively small amounts of language-specific documentation. Our work focuses on Romanian, a language that represents approximately 0.6% of Common Crawl pages ( Common Crawl, 2024 ) and where existing open models demonstrate limited capabilities ( Masala et al., 2024 ) . To address this gap, we introduce LLMic, a 3B-parameter bilingual Romanian-English foundation model. This paper presents a comprehensive study on pretraining a foundation model; we address critical challenges, beginning with the construction of high-quality pretraining corpora for the Romanian language - a resource notably absent from the state-of-the-art open datasets ( Penedo et al., 2024 ; Weber et al., 2024 ; Soldaini et al., 2024 ; Penedo et al., 2023 ; Li et al., 2024 ) . Next, we discuss the process of selecting the model’s architecture, optimization of hyperparameters, and development of an efficient runtime configuration. To showcase the usefulnes of LLMic, we fine-tune the model for translation tasks and evaluate its performance against existing models, including fine-tuned model for the Romanian language. By releasing LLMic to the research community, we aim to bootstrap the development of natural language processing tools and models for the Romanian language. LLMic 1 1 1 https://huggingface.co/faur-ai/LLMic is released under the Apache 2.0 License. 2. Pretraining Data Data scarcity presents a fundamental challenge in developing models for Romanian language processing. Acquiring hundreds of billions of tokens requires extensive filtering and cleaning of CommonCrawl’s petabyte-scale dataset. Obtaining additional curated content often necessitates applying optical character recognition (OCR) ( Smith, 2007 ) to public domain documents, or parsing complex formats (HTML, EPUB) to extract text. We trained LLMic on a mixture of filtered web data and curated sources in both Romanian and English. The data is split in 300B tokens for Romanian and 700B tokens for English, as shown in Table 1 . For the English subset of the pretraining data, we use FineWebEdu ( Penedo et al., 2024 ) due to its LLM-assisted quality filtering, and we further source a curated set of documents from Dolma ( Soldaini et al., 2024 ) ; for Romanian, we use the entirety of FuLG ( Bădoiu et al., 2024 ) , and we further repeat a small number of high-quality documents. We mask all personal identifiable information (PII). A detailed overview of the dataset is discussed below. Source Size Romanian (300B) Web Sources 621 GB Discussions, Curated & Parallel 10 GB English (700B) FineWebEdu – Dolma Subset 109 GB Table 1 . Dataset Composition by Language and Source Web Sources. We leverage two filtered CommonCrawl sources for Romanian language data: FuLG ( Bădoiu et al., 2024 ) , extracted using the CCNet pipeline ( Wenzek et al., 2019 ) , and contains 220B tokens when processed with the Llama tokenizer; and mC4 ( Raffel et al., 2020 ) , a multilingual cleaned version of CommonCrawl, comprising 42B Romanian tokens. While they share the source, the two corpora differ in their language detection algorithms and cleaning and deduplication techniques. To remove duplicates resulted from the merge, we apply both fuzzy and exact deduplication using the RedPajama ( Weber et al., 2024 ; Computer, 2023 ) pipeline. We further augment our dataset by incorporating filtered content from recent CommonCrawl dumps (2024-22, 2024-26, 2024-30, and 2024-33). Curated Data. We extend our Romanian web corpus with curated data from multiple sources, including the Romanian Wikipedia and public documents from Romanian institutions processed. For the English portion, we

Read the complete text