Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training
Pierre-Carl Langlais, Carlos Rosas Hinostroza, Mattia Nee, Catherine Arnett, Pavel Chizhov, Eliot Krzystof Jones, Irène Girard, David Mach, Anastasia Stasenko, Ivan P. Yamshchikov, PleIAs, Paris, France https://pleias.fr/ · 2025
Read Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training by Pierre-Carl Langlais, Carlos Rosas Hinostroza, Mattia Nee, Catherine Arnett, Pavel Chizhov, Eliot Krzystof Jones, Irène Girard, David Mach, Anastasia Stasenko, Ivan P. Yamshchikov, PleIAs, Paris, France https://pleias.fr/ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01732.txt.
Description
Pierre-Carl Langlais and co-authors released Common Corpus, the largest open dataset for training language models that is built only from material free of copyright restrictions or openly licensed. Standard training sets swallow trillions of words of uncertain legal status. This corpus, about two trillion tokens, documents where every part came from and how it was filtered.
The collection spans many languages, from major European ones to rarely represented low-resource languages, plus a large body of code. To prove it works, the team trained two small language models on it and matched the performance of similar-sized rivals trained on ordinary data. They present the corpus as shared infrastructure for open, legally clean AI research and business use.
Catalog record
| CREATOR | Pierre-Carl Langlais, Carlos Rosas Hinostroza, Mattia Nee, Catherine Arnett, Pavel Chizhov, Eliot Krzystof Jones, Irène Girard, David Mach, Anastasia Stasenko, Ivan P. Yamshchikov, PleIAs, Paris, France https://pleias.fr/ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01732