HPLT 3.0: Very Large-Scale Multilingual Resources for LLM and MT Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models

2025

Read HPLT 3.0: Very Large-Scale Multilingual Resources for LLM and MT Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01066.txt.

Introduction

Stephan Oepen and a very large team present HPLT 3.0, an open collection of text data for training language and translation models in almost two hundred languages. At thirty trillion tokens, they describe it as likely the largest multilingual pretraining collection available to all.

The data come from web crawls, processed by a fully open pipeline that selects documents, extracts text, identifies languages, removes duplicates, and annotates register, quality, and personal information. Quality is probed by statistics, by hand inspection in twenty four languages, and by training models end to end. The release also adds test benchmarks for nine European languages and a family of fifty seven monolingual encoder decoder models, plus a large parallel corpus for translation.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01066

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HPLT 3.0: Very Large-Scale Multilingual Resources for LLM and MT Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models Abstract We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data. These datasets are derived from web crawls from different sources and accompanied with a complete, open-source pipeline for document selection from web archives, text extraction from HTML, language identification for noisy texts, exact and near-deduplication, annotation with, among others, register labels, text quality estimates, and personally identifiable information; and final selection and filtering. We report on data quality probes through contrastive and analytical statistics, through manual inspection of samples for 24 languages, and through end-to-end evaluation of various language model architectures trained on this data. For multilingual LLM evaluation, we provide a comprehensive collection of benchmarks for nine European languages, with special emphasis on natively created tasks, mechanisms to mitigate prompt sensitivity, and refined normalization and aggregation of scores. Additionally, we train and evaluate a family of 57 monolingual encoder–decoder models, as well as a handful of monolingual GPT-like reference models. Besides the monolingual data and models, we also present a very large collection of parallel texts automatically mined from this data, together with a novel parallel corpus synthesized via machine translation. Keywords: large language models, multilinguality, pre-training data, evaluation, machine translation, mining parallel texts, synthetic data HPLT 3.0: Very Large-Scale Multilingual Resources for LLM and MT Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models Stephan Oepen ♣ , Nikolay Arefev ♣ , Mikko Aulamo ♠ , Marta Bañón ♡ , Maja Buljan ♣ , Laurie Burchell ♢ , Lucas Charpentier ♣ , Pinzhen Chen ∙ , Mariia Fedorova ♣ , Ona de Gibert ♠ , Barry Haddow ∙ , Jan Hajič ∘ , Jindřich Helcl ♣ , Andrey Kutuzov ♣ , Veronika Laippala ⋆ , Zihao Li ♠ , Risto Luukkonen ⋆ , Bhavitvya Malik ∙ , Vladislav Mikhailov ♣ , Amanda Myntti ⋆ , Dayyán O’Brien ∙ , Lucie Poláková ∘ , Sampo Pyysalo ⋆ , Gema Ramírez Sánchez ♡ , Janine Siewert ♠ , Pavel Stepachev ∙ , Jörg Tiedemann ♠ , Teemu Vahtola ♠ , Dušan Variš ∘ , Fedor Vitiugin ⋆ , Tea Vojtěchová ∘ , Jaume Zaragoza ♡ ♣ University of Oslo, Department of Informatics ♠ University of Helsinki, Department of Digital Humanities ♡ Prompsit Language Engineering ♢ The Common Crawl Foundation ∙ Edinburgh University, School of Informatics ∘ Charles University, Prague, Institute of Formal and Applied Linguistics ⋆ TurkuNLP, University of Turku, Department of Computing [email protected] Abstract content 1. Introduction Massive text collections for pre-training are the “crude oil” of the LLM era. The process of “refining” high-quality datasets from web data at scale presupposes computational infrastructure and technological muscle that oftentimes is characteristic of corporate involvement, as evidenced for example by some notable generally available pre-training datasets: C4 (the Colossal Clean Crawled Corpus, by Google and Allen AI; Raffel et al. 2020 ), FineWeb 1 & 2 (by Hugging Face; Penedo et al. 2024 ; Penedo et al. 2025 ), MADLAD-400 (by Google; Kudugunta et al. 2023 ), or Nemotron-CC (by Nvidia; Su et al. 2025 ). With notable exceptions, this line of work tends to capitalize on the English language. We build on the open-source pipeline of the European academic consortium HPLT (High-Performance Language Technologies; de Gibert et al. 2024 ; Burchell et al. 2025 ), a project that was funded under the Horizon Europe programme in 2022–2025. We revise and improve various of the sub-components, and apply the updated pipeline to a greatly enlarged collection of raw web data. This results in a massive pre-training dataset of high-quality texts in close to 200 distinct language–script combinations. We dub this novel resource HPLT 3.0. The data comprises some 30 trillion sub-word tokens in total, of which close to half represent languages other than English. Following the examples of FineWeb and HPLT 2.0, we make this resource publicly available under permissive licenses. 1 1 1 https://hplt-project.org/datasets/v3.0 We further share the refined open-source processing pipeline, a novel multilingual evaluation framework, as well as various collections of language models pre-trained on HPLT 3.0 data. Finally, while our focus in this report is on the monolingual data and models, we also briefly report on ongoing work to derive novel bilingual datasets for 28 language pairs, provide associated machine translation models, and synthesize additional pre-training data for underrepresented languages by machine translation of very high-quality English documents. In our view, it is the totality of generally available and very large-scale resources and the documentation of the underlying processes that bears promise of “democratizing” the current LLM and MT landscape. In-depth technical and experimental details will be provided through appendices in a forthcoming extended version of this report. 2. Raw Web Archive Data There are few available collections of massive web archives. Our work builds on the same set of so-called “wide crawls” from the Internet Archive (IA) as the HPLT 2.0 release, but combines this with a broader and much larger set of snapshots from the Common Crawl (CC). Specifically, we start from some 3.3 petabytes of IA crawls from the period 2012–2020. We complement this data with 57 CC full snapshots from the period 2014–2025, making sure to include all available data since 2020, together with about half of the earlier CC snapshots. This amounts to almost five times more raw CC data than in HPLT 2.0, bringing our total volume of web archives to about 7.2 petabytes. HPLT 3.0 to date does not include a few experimental data sources, an IA “survey crawl” and a minor sample from the ArchiveBot repository (270 terabytes in HPLT). We plan on reporting on an enlarged dataset in early 2026, augmented with another 3 petabytes from ArchiveBot. 3. Monolingual Data Preparation Figure 1: Schematic overview of data preparation. Extraction of high-quality and richly annotated text from raw web archives proceeds through a sequence of refinement and filtering steps. Figure 1 shows the main components of our data preparation, which is an updated and extended version of the open-source HPLT pipeline by Burchell et al. 2025 . The following paragraphs provide a high-level overview of the refinements we make to the original HPLT 2.0 pipeline. Text Extraction Like FineWeb and HPLT 2.0, we stick to the Trafilatura text extraction framework Barbaresi 2021 , but we conduct a comprehensive data-driven optimization of its many hyper-parameters, emphasizing extraction quality over speed and text recall. Language Identification We predict the language of the text during preprocessing using OpenLID-v2, an updated version of the model described in Burchell et al. 2023 . Compared to the prior version of OpenLID, we revise the inventory of language labels to align with the Flores + evaluation set Burchell et al. 2024 , and include additional training data for Dari, South Azerbaijani, Asturian, and Paraguayan Guaraní. We also revise input preprocessing for language identification to a simpler, language-agnostic pipeline: We normalize whitespace, apply lowercasing, and remove non-word characters and digits. This change increases the robustness of language identification to the non-standard orthography commonly found in web documents. Deduplication Random

Read the complete text