An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence

Qizhen Zhang, Ankush Garg, Jakob Foerster, Niladri Chatterji, Kshitiz Malik, Mike Lewis · 2026

Read An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence by Qizhen Zhang, Ankush Garg, Jakob Foerster, Niladri Chatterji, Kshitiz Malik, Mike Lewis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02400.txt.

Description

Qizhen Zhang and 5 co-authors test whether noisy web data can make language model training blow up, in a 2026 arXiv paper. Pretraining corpora inevitably contain junk, and trainers suspect it triggers loss divergence, where the training error suddenly spirals. Until now that suspicion lacked controlled evidence.

The team injects synthetic random noise into clean datasets and trains models from under half a billion to over five billion parameters. Noise does cause divergence, and the risk depends strongly on noise type, noise amount, and model scale. Noise-driven failures also light up different internal activation patterns than failures caused by high learning rates. The authors supply diagnostics that tell the two apart. Their results give trainers a firmer grip on a costly, poorly understood failure mode.

Catalog record

CREATORQizhen Zhang, Ankush Garg, Jakob Foerster, Niladri Chatterji, Kshitiz Malik, Mike Lewis
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02400

Browse the full library · About the Tower of Babel