An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT
Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, Bhavitvya Mal · 2025
Read An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT by Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, Bhavitvya Mal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10267.txt.
Introduction
Burchell and co-authors release a vast text trove in 2025 for training language tools across many tongues. Top models need huge, clean, varied text, and building that in many languages at once is hard. Their set, HPLT version two, grows an earlier project of the same name.
The single-language part holds eight trillion tokens across 193 languages. The paired part holds 380 million sentence pairs across fifty-one languages, fit for translation work. The team logs every step of how the data was built and shares the code to redo it. Wide checks cover quality and mix. An open trove of this size helps smaller language communities build tools that only big labs could make before.
Catalog record
| CREATOR | Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, Bhavitvya Mal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10267
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT) Thanks: Work was done prior to joining Amazon. Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, Bhavitvya Malik, Farrokh Mehryary, Vladislav Mikhailov, Nikita Moghe, Amanda Myntti, Dayyán O’Brien, Stephan Oepen, Proyag Pal, Jousia Piha, Sampo Pyysalo, Gema Ramírez-Sánchez, David Samuel, Pavel Stepachev, Jörg Tiedemann, Dušan Variš, Tereza Vojtěchová, Jaume Zaragoza-Bernabeu Abstract Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value. 1 Introduction In order to train the state-of-the-art llm ( llm ) required for modern NLP, large amounts of high-quality textual training data are essential. However, obtaining a sufficient quantity of such data is far from easy. In addition, effective NLP research requires open training data so that results can be replicated and verified. In this paper, we introduce a new set of text corpora dubbed HPLT v2. 1 1 1 https://hplt-project.org/datasets/v2.0 It is extracted from 4.5 petabytes (PB) of the ia ( ia ) 2 2 2 https://archive.org and cc ( cc ) 3 3 3 https://commoncrawl.org data. We build on the work of de Gibert et al. (2024) (hereafter referred to as HPLT v1.2) with an improved extraction pipeline and a much larger set of input crawls to produce the HPLT v2 collection of monolingual and parallel corpora. To our knowledge, our new corpus is the only large-scale text collection extracted from the ia , apart from HPLT v1.2. We release HPLT v2 under the permissive cc0 ( cc0 ) license 4 4 4 https://creativecommons.org/share-your-work/public-domain/cc0/ . We do not claim ownership of any of the text from which this data has been extracted. and provide the code to replicate our pipeline. Our main contributions can be summarised as: • We release monolingual corpora covering 193 languages and containing approximately 52 trillion characters and 8 trillion tokens. • We derive parallel corpora from our monolingual data for 50 languages paired with English, containing over 380 million sentence pairs. • We make the tools and pipelines used to create the collection openly available. 5 5 5 https://github.com/hplt-project/HPLT-textpipes • We conduct an in-depth analysis of our data including descriptive statistics, manual inspection, and automatic register labelling. • We demonstrate the quality of HPLT v2 by using it to train a range of high-performing language and machine translation models. Figure 1: The distribution of documents in the HPLT v2 cleaned dataset by language family and variety. Shortened ISO 639-3 language codes are used here; see the full plots at https://hplt-project.org/datasets/v2.0 2 Related work The increasing data demands of state-of-the-art llm have driven a rapid growth in both the number and the size of text corpora. We provide a summary of some well-known collections in Appendix A . Whilst llm trained on ostensibly English data have shown impressive multilingual capabilities Armengol-Estapé et al. (2022) , of particular relevance to this work is the growing shift towards explicitly multilingual corpora. Compared with earlier efforts (e.g. OSCAR Suárez et al. (2019) , CC-100 Conneau et al. (2020) and mC4 Xue et al. (2021) ), more recent multilingual datasets cover increasing numbers of languages, e.g. CulturaX Nguyen et al. (2024a) and MADLAD-400 Kudugunta et al. (2024) . HPLT v2 continues this trend by aiming for significant coverage of a wide range of languages. We note that the majority of previous multilingual datasets are sourced from cc , whereas much of HPLT v2 is composed of ia crawls. This means that HPLT v2 can be used in conjunction with these existing datasets as a complementary source. Producing large-scale datasets by crawling the Web is helpful for scale, but raises questions around dataset quality such as the prevalence of boilerplate, explicit material or non-linguistic content Kreutzer et al. (2022) . One way to tackle low-quality data is through human audit and curation (e.g. ROOTS Laurençon et al. (2022) , Glot500-c Imani et al. (2023) , Serengeti Adebara et al. (2023) and the MaLA Corpus Ji et al. (2024) ). However, such an approach is difficult to scale. Instead, we ensure the quality of HPLT v2 through a robust dataset construction pipeline ( Section 4 ) and by verifying our data through extensive analysis and downstream evaluation ( Sections 5 and 6 ). In addition to large-scale monolingual data in multiple languages, HPLT v2 contains high-quality parallel data. Whilst clm with decoder-only architectures rely primarily on monolingual data, recent studies have shown that incorporating parallel data during the pretraining stage significantly boosts multilingual, cross-lingual and mt ( mt ) performance for such models Kale et al. (2021) ; Briakou et al. (2023) ; Alves et al. (2024) . Because of this, we expect that there is still significant demand for parallel data. This work is a direct successor of our HPLT v1.2 dataset introduced in de Gibert et al. (2024) . Compared to HPLT v1.2, we feature more data (21 billion vs. 5 billion documents) using an improved pipeline ( Section 4 ), resulting in a significantly larger dataset (52 trillion characters compared to 42 trillion). The HPLT v2 collections are of higher quality than those in HPLT v1.2, as shown through comparative analysis and evaluation ( Sections 5 and 6 ). For clarity, we list the main differences between HPLT v1.2 and HPLT v2 here: • The size of the source web collections is 2.5x larger, totalling 4.5 petabytes of compressed web data. • The text extraction pipeline uses Trafilatura rather than warc2text , which results in more efficient boilerplate removal. • Language identification uses a modified version of OpenLID rather than CLD2 , increasing coverage from 75 to 193 language varieties. • Documents are annotated with their compliance to the robots.txt files of the corresponding websites, which can be used to filter out documents explicitly forbidden for crawling by website owners. The cleaned variant of the HPLT v2 dataset contains only robots.txt compliant documents. • Deduplication is done at collection level rather than globally. • Documents are annotated for PII information. • Document quality scores computed with web-docs-scorer rather than segment-level language model based scores • Simpler and more interpretable filtering and cleaning criteria. 3 Dataset description In this section, we describe the HPLT v2 collection of monolingual and parallel corpora, before explaining how it was constructed in Section 4 . 3.1 Monolingual datasets The monolingual portion of HPLT v2 covers 193 language varieties 6 6 6 Language varieties are labelled with an ISO 639-3 code denoting the variety plus an ISO 15924 four-letter code