Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering
Vlad Negoi t , ă, Mihai Masala, Traian Rebedea · 2025
Read Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering by Vlad Negoi t , ă, Mihai Masala, Traian Rebedea (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01090.txt.
Introduction
Vlad Negoita, Mihai Masala, and Traian Rebedea study how to build better Romanian language data for pretraining large language models. Data quality is decisive for model quality, and for under represented languages like Romanian, good corpora are scarce, so filtering matters even more than for English.
They first compare the coverage and character of Romanian corpora with English data. Then they train a lightweight multitask model on Romanian texts annotated by language models, rating educational value, topic, and format, and use it to filter at several levels. The filtered data sets improved pretraining performance across multiple benchmarks. The study also reports notable differences in which topics the Romanian and English corpora actually contain.
Catalog record
| CREATOR | Vlad Negoi t , ă, Mihai Masala, Traian Rebedea |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01090
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering Vlad Negoi t , ă, Mihai Masala, Traian Rebedea Abstract Large Language Models (LLMs) have recently exploded in popularity, often matching or outperforming human abilities on many tasks. One of the key factors in training LLMs is the availability and curation of high-quality data. Data quality is especially crucial for under-represented languages, where high-quality corpora are scarce. In this work we study the characteristics and coverage of Romanian pretraining corpora and we examine how they differ from English data. By training a lightweight multitask model on carefully LLM-annotated Romanian texts, we are able to analyze and perform multi-level filtering (e.g., educational value, topic, format) to generate high-quality pretraining datasets. Our experiments show noteworthy trends in the topics present in Romanian and English data, while also proving the effectiveness of filtering data through improved LLM pretraining performance across multiple benchmarks. 1 Introduction Recent advances in Artificial Intelligence, especially in Natural Language Processing (NLP), have been driven by Transformer architectures ( Vaswani et al. 2017 ) and Large Language Models (LLMs). These innovations have transformed how machines process language, enabling applications such as conversational AI, intelligent search, machine translation, and content generation. The success of modern LLMs depends heavily on large, high-quality pretraining datasets Longpre et al. 2024 . With billions of parameters, these models require vast data to capture statistical patterns, semantic nuances, and world knowledge in human language. Although multilingual datasets ( Penedo et al. 2024b ; Nguyen et al. 2023 ; Ortiz Suárez et al. 2019 ) have driven broad NLP progress, language-specific resources are vital for robust performance across diverse languages. Romanian is under-represented in large multilingual corpora, causing models trained on high-resource languages to underperform on Romanian. Dedicated Romanian datasets are therefore essential for training, finetuning, and evaluating language models ( Masala et al. 2024 ) . Recent studies highlight data quality as crucial for strong benchmark performance ( Lozhkov et al. 2024 ; Ali et al. 2025 ; Bai et al. 2025 ) . The FineWeb-Edu framework ( Lozhkov et al. 2024 ) proposed an educational-value-based filtering method effective across various tasks, forming the basis for our Romanian adaptation. To understand potential topic bias from this filtering, we also incorporate additional signals to examine diversity issues. Our entire recipe, including code, data, and models is publicly available 1 1 1 https://huggingface.co/collections/OpenLLM-Ro/pretraining-datasets . Our contributions can be summarized as follows: • We build multidimensional (i.e., educational value, topic, format, reader education level) resources: a small human-annotated dataset (100 samples) and a large (1M samples) LLM-annotated dataset. • We train a lightweight multi-head classifier that enables cost-effective filtering and cross-lingual distribution analysis at scale. • We build the first high-quality pretraining dataset for Romanian - FineWeb2-Edu-Ro - and prove its usefulness by performing continual pretraining. Compared to other approaches, models trained on our filtered dataset exhibit superior performance across a variety of benchmarks. 2 Related Work Recent efforts in pretraining focus on high-quality data, combining filtering (rule-based or ML-driven) with growing interest in synthetic data for its significant benefits. FineWeb ( Penedo et al. 2024a ) , FineWeb2 ( Penedo et al. 2024b ) , and FineWeb-Edu ( Lozhkov et al. 2024 ) represent a collection of high-quality web-based datasets for training large language models. FineWeb initiative offers both English-only datasets (FineWeb - 15T tokens) and multilingual datasets (FineWeb2 - 1000+ languages - 35B Romanian words), cleaned and deduplicated. Quality Filtering. FineWeb-Edu employs classifier-based quality (educational content) filtering using Llama-3-70B ( AI@Meta 2024 ) annotations and Snowflake-Arctic-Embed ( Merrick et al. 2024 ) to train a lightweight regressor for educational value. JQL ( Ali et al. 2025 ) introduces a language-agnostic setup for annotating educational value in text, using manual labeling and automatic translation to build a multilingual dataset. They benchmark several LLMs and select the top three based on Spearman correlation with human labels, then train lightweight models using Snowflake-Arctic-Embed. Instead of relying on a single metric for data quality, MetaRater ( Zhuang et al. 2025 ) uses classifiers for 4 key criteria: professionalism, readability, reasoning, and cleanliness. Reliable identification of lower-quality documents allows alternative approaches such as document rewriting to increase the overall quality of datasets ( Nguyen et al. 2025 ) . Romanian Datasets. Besides FineWeb2, we identify only two other important datasets for Romanian, namely CulturaX ( Nguyen et al. 2023 ) (40B tokens in Romanian) and FuLG ( Bădoiu et al. 2024 ) (150B tokens in Romanian). All three datasets stem from CommonCrawl, with different number of snapshots used and different rules for processing and filtering. Crucially, all datasets employ rather standard rules based on n-gram frequency, stop word ratio or text length, and thus lack a more high-level quality based filtering. Evaluation. Several pretraining efforts evaluate performance across different model sizes (1B–7B) using metrics like perplexity and accuracy on benchmarks such as MMLU ( Hendrycks et al. 2021 ) , ARC ( Clark et al. 2018 ) , HellaSwag ( Zellers et al. 2019 ) , or OpenBookQA ( Mihaylov et al. 2018 ) . FineWeb-Edu trains a 1.71B model on up to 350B tokens reporting improvements over unfiltered datasets on MMLU, ARC, and OpenBookQA. Similarly, JQL pretrains a 2B Llama-based model, outperforming FineWeb2 on various benchmarks. FuLG trains a 1B decoder-only OLMo ( Groeneveld et al. 2024 ) model and reports perplexity gains over mC4 and OSCAR ( Ortiz Suárez et al. 2019 ) . Our approach builds upon recent advancements in data curation for LLMs, but distinguishes itself by jointly predicting multiple signals for a given text in Romanian, as educational value is insufficient for efficient pretraining ( Bai et al. 2025 ) . 3 Taxonomy definition For educational quality, we utilize the validated 5-point grading scale from FineWeb-Edu for its effectiveness and to enable comparison between Romanian and English distributions. Regarding additional signals, we follow the taxonomy of WebOrganizer ( Wettig et al. 2025 ) , which developed topic 2 2 2 https://huggingface.co/WebOrganizer/TopicClassifier-NoURL (e.g., Health, Politics) and format 3 3 3 https://huggingface.co/WebOrganizer/FormatClassifier-NoURL (e.g., News Article, Creative Writing) classifiers to analyze large-scale English web distributions. Finally, we also extract the required educational level per text (preschool through post-graduate studies), which enables progressive learning or curriculum learning for LLMs Mukherjee et al. 2023 and could enhance quality prediction. Full taxonomies are provided in Appendix A . Figure 1: Flowchart detailing the multi-stage pipeline for building an educational Romanian pretraining dataset. The process includes initial human annotation and LLM benchmarking, distillation of a lightweight annotator for full-scale labeling, and final filtering with a subsequent cross-lingual distribution analysis. 4 Approach We start by manually annotating a subset of 50 organic Romanian texts from FineWeb2. Furthermore, we add another 50 texts from FineWeb-Edu (translated into Romanian and retaining their classifier’s educational scores) to enable score alignment and provide a less skewed distribution. We use the resulting dataset (100 samples) to select the best performing LLM (e.g., model, prompt, prompt language) that we will use for annotating a larger dataset, dataset that will be used for training a lightweight classifier. Figure 1 illustrates the pretraining dataset construction process. 4.1 Training dataset Following manual annotation, we evaluated a wide range of models, including Llama-3, 3.1, and