Rewriting Pre-Training Data Boosts LLM Performance in Math and Code

Kazuki Fujii, Yukito Tajima, Sakae Mizuki, Masaki Kawamura, Hinari Shimada, Taihei Shiotani, Koshiro Saito, Masanari Oi, Taishi Nakamura, Takumi Okamoto, Shigeki Ishida, Kakeru Hattori, Youmi Ma, Hiroya Takamura, Rio Yokota, Jun Sakuma, Naoaki Okazaki · 2025

Read Rewriting Pre-Training Data Boosts LLM Performance in Math and Code by Kazuki Fujii, Yukito Tajima, Sakae Mizuki, Masaki Kawamura, Hinari Shimada, Taihei Shiotani, Koshiro Saito, Masanari Oi, Taishi Nakamura, Takumi Okamoto, Shigeki Ishida, Kakeru Hattori, Youmi Ma, Hiroya Takamura, Rio Yokota, Jun Sakuma, Naoaki Okazaki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02881.txt.

Description

Fujii and co-authors show, in a paper published in 2025, that rewriting a language model's training data, rather than just gathering more, lifts its math and coding skill. They release two openly licensed datasets built by systematically cleaning public sources. One refines Python code from a giant public archive through four stages. The stages check syntax, filter style with an automatic linter, then rewrite twice with a language model to enforce consistent style.

A companion dataset applies the same philosophy to mathematics text. Models trained on the rewritten corpora outperform those trained on the raw originals. The lesson is industrial: data curation is a lever as strong as model size. Open licensing lets other teams verify and build on the gains. The paper details the pipeline so it can be copied.

Catalog record

CREATORKazuki Fujii, Yukito Tajima, Sakae Mizuki, Masaki Kawamura, Hinari Shimada, Taihei Shiotani, Koshiro Saito, Masanari Oi, Taishi Nakamura, Takumi Okamoto, Shigeki Ishida, Kakeru Hattori, Youmi Ma, Hiroya Takamura, Rio Yokota, Jun Sakuma, Naoaki Okazaki
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-02881

Browse the full library · About the Tower of Babel