Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training

Pierre-Carl Langlais, Carlos Rosas Hinostroza, Mattia Nee, Catherine Arnett, Pavel Chizhov, Eliot Krzystof Jones, Irène Girard, David Mach, Anastasia Stasenko, Ivan P. Yamshchikov, PleIAs, Paris, France https://pleias.fr/ · 2025

Read Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training by Pierre-Carl Langlais, Carlos Rosas Hinostroza, Mattia Nee, Catherine Arnett, Pavel Chizhov, Eliot Krzystof Jones, Irène Girard, David Mach, Anastasia Stasenko, Ivan P. Yamshchikov, PleIAs, Paris, France https://pleias.fr/ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01732.txt.

Description

Pierre-Carl Langlais and co-authors released Common Corpus, the largest open dataset for training language models that is built only from material free of copyright restrictions or openly licensed. Standard training sets swallow trillions of words of uncertain legal status. This corpus, about two trillion tokens, documents where every part came from and how it was filtered.

The collection spans many languages, from major European ones to rarely represented low-resource languages, plus a large body of code. To prove it works, the team trained two small language models on it and matched the performance of similar-sized rivals trained on ordinary data. They present the corpus as shared infrastructure for open, legally clean AI research and business use.

Catalog record

CREATORPierre-Carl Langlais, Carlos Rosas Hinostroza, Mattia Nee, Catherine Arnett, Pavel Chizhov, Eliot Krzystof Jones, Irène Girard, David Mach, Anastasia Stasenko, Ivan P. Yamshchikov, PleIAs, Paris, France https://pleias.fr/
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01732

Browse the full library · About the Tower of Babel