Data-efficient pre-training by scaling synthetic megadocs
Konwoo Kim, Suhas Kotha · 2026
Read Data-efficient pre-training by scaling synthetic megadocs by Konwoo Kim, Suhas Kotha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18534.txt.
Description
Kim and co-authors ask how to stretch scarce pre-training data with synthetic text. Mixing web text with rephrased copies improves loss and benchmark scores without overfitting. Gains level off near 1.48 times data efficiency at 32 rephrases per source text. Joining rephrases of one source into a single long megadoc works better still. Megadocs lift efficiency to 1.80 times and help long-context loss most. The gap over plain rephrasing grows as more synthetic text is added.
Catalog record
| CREATOR | Konwoo Kim, Suhas Kotha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57564 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18534