GneissWeb: Preparing High Quality Data for LLMs at Scale

Hajar Emami Gohari, Swanand Ravindra Kadhe, Syed Yousaf Shah †, Constantin Adam, Abdulhamid Adebayo, Praneet Adusumilli, Farhan Ahmed, Nathalie Baracaldo Angel, Santosh Subhashrao Borse, Yuan-Chi Chang, Xuan-Hong Dang, Nirmit Desai, Revital Eres, Ran Iwamoto, Alexei Karve, Yan Koyfman, Wei-Han Lee, · 2025

Read GneissWeb: Preparing High Quality Data for LLMs at Scale by Hajar Emami Gohari, Swanand Ravindra Kadhe, Syed Yousaf Shah †, Constantin Adam, Abdulhamid Adebayo, Praneet Adusumilli, Farhan Ahmed, Nathalie Baracaldo Angel, Santosh Subhashrao Borse, Yuan-Chi Chang, Xuan-Hong Dang, Nirmit Desai, Revital Eres, Ran Iwamoto, Alexei Karve, Yan Koyfman, Wei-Han Lee, (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14907.txt.

Description

Hajar Emami Gohari and co-authors built a web dataset of about ten trillion tokens for training language models, in a 2025 paper. Data quality shapes what such models learn, yet the best training sets stay private and open ones are often small. Their dataset, GneissWeb, was built with a strict recipe: text was split into shards, exact repeated passages were removed, and a team of quality filters kept only strong material.

Models trained on GneissWeb beat models trained on a leading open rival across standard test suites. The margin held on both a core set of eleven benchmarks and a wider set of twenty. The authors argue the recipe strikes a good balance: aggressive cleaning without shrinking the pool too far. They release the approach as a way for open research to narrow the gap with closed labs.

Catalog record

CREATORHajar Emami Gohari, Swanand Ravindra Kadhe, Syed Yousaf Shah †, Constantin Adam, Abdulhamid Adebayo, Praneet Adusumilli, Farhan Ahmed, Nathalie Baracaldo Angel, Santosh Subhashrao Borse, Yuan-Chi Chang, Xuan-Hong Dang, Nirmit Desai, Revital Eres, Ran Iwamoto, Alexei Karve, Yan Koyfman, Wei-Han Lee,
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14907

Browse the full library · About the Tower of Babel