Data-efficient pre-training by scaling synthetic megadocs

Konwoo Kim, Suhas Kotha · 2026

Read Data-efficient pre-training by scaling synthetic megadocs by Konwoo Kim, Suhas Kotha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18534.txt.

Description

Kim and co-authors ask how to stretch scarce pre-training data with synthetic text. Mixing web text with rephrased copies improves loss and benchmark scores without overfitting. Gains level off near 1.48 times data efficiency at 32 rephrases per source text. Joining rephrases of one source into a single long megadoc works better still. Megadocs lift efficiency to 1.80 times and help long-context loss most. The gap over plain rephrasing grows as more synthetic text is added.

Catalog record

CREATORKonwoo Kim, Suhas Kotha
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57564
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18534

Browse the full library · About the Tower of Babel