GneissWeb: Preparing High Quality Data for LLMs at Scale
Hajar Emami Gohari, Swanand Ravindra Kadhe, Syed Yousaf Shah †, Constantin Adam, Abdulhamid Adebayo, Praneet Adusumilli, Farhan Ahmed, Nathalie Baracaldo Angel, Santosh Subhashrao Borse, Yuan-Chi Chang, Xuan-Hong Dang, Nirmit Desai, Revital Eres, Ran Iwamoto, Alexei Karve, Yan Koyfman, Wei-Han Lee, · 2025
Read GneissWeb: Preparing High Quality Data for LLMs at Scale by Hajar Emami Gohari, Swanand Ravindra Kadhe, Syed Yousaf Shah †, Constantin Adam, Abdulhamid Adebayo, Praneet Adusumilli, Farhan Ahmed, Nathalie Baracaldo Angel, Santosh Subhashrao Borse, Yuan-Chi Chang, Xuan-Hong Dang, Nirmit Desai, Revital Eres, Ran Iwamoto, Alexei Karve, Yan Koyfman, Wei-Han Lee, (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14907.txt.
Description
Hajar Emami Gohari and co-authors built a web dataset of about ten trillion tokens for training language models, in a 2025 paper. Data quality shapes what such models learn, yet the best training sets stay private and open ones are often small. Their dataset, GneissWeb, was built with a strict recipe: text was split into shards, exact repeated passages were removed, and a team of quality filters kept only strong material.
Models trained on GneissWeb beat models trained on a leading open rival across standard test suites. The margin held on both a core set of eleven benchmarks and a wider set of twenty. The authors argue the recipe strikes a good balance: aggressive cleaning without shrinking the pool too far. They release the approach as a way for open research to narrow the gap with closed labs.
Catalog record
| CREATOR | Hajar Emami Gohari, Swanand Ravindra Kadhe, Syed Yousaf Shah †, Constantin Adam, Abdulhamid Adebayo, Praneet Adusumilli, Farhan Ahmed, Nathalie Baracaldo Angel, Santosh Subhashrao Borse, Yuan-Chi Chang, Xuan-Hong Dang, Nirmit Desai, Revital Eres, Ran Iwamoto, Alexei Karve, Yan Koyfman, Wei-Han Lee, |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-14907