Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
Xu Guo, Runyu Peng, Jian Tong, Yunhua Zhou, Haijun Lv, Zhihui Lu, Qipeng Guo, [6pt] Shanghai AI Laboratory, [2pt] Corresponding authors · 2026
Read Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data by Xu Guo, Runyu Peng, Jian Tong, Yunhua Zhou, Haijun Lv, Zhihui Lu, Qipeng Guo, [6pt] Shanghai AI Laboratory, [2pt] Corresponding authors (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10129.txt.
Description
Guo and six colleagues test whether a short synthetic pre-training stage can toughen language models against noisy web data. Web-scale corpora always carry noise, from duplicated boilerplate to formatting errors, and curation cannot remove it all, so noise obscures patterns and degrades performance, sometimes cutting knowledge capacity sharply. Their pre-pre-training stage uses synthetic data with learnable temporal structure before ordinary pre-training begins.
Across corruption settings, the method consistently improved robustness, with bigger relative gains at higher noise levels. In a one-billion-parameter model, a synthetic stage of only sixty-five million tokens matched baseline final loss while using up to forty-nine percent fewer natural-text tokens across noise levels. Mechanistic analysis suggests the stage does not at once suppress attention to noisy tokens; instead, initialised models gradually downweight attention between corrupted tokens, inhibiting noise self-modelling and reshaping later optimisation.
Catalog record
| CREATOR | Xu Guo, Runyu Peng, Jian Tong, Yunhua Zhou, Haijun Lv, Zhihui Lu, Qipeng Guo, [6pt] Shanghai AI Laboratory, [2pt] Corresponding authors |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10129