A Bitter Lesson for Data Filtering

Christopher Mohri, John Duchi, Tatsunori Hashimoto · 2026

Read A Bitter Lesson for Data Filtering by Christopher Mohri, John Duchi, Tatsunori Hashimoto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19407.txt.

Introduction

Mohri, Duchi and Hashimoto test a common pretraining habit in the setting where it matters most, plentiful compute paired with scarce data. Scaling runs for large models show that long training lets them absorb low-quality and distracting material without harm, and nominally poor material can even help. That result flips the usual rule: when compute is ample, skipping selection entirely beats selecting at all. Their point is aimed at the belief that only high-quality text belongs in pretraining.

Catalog record

CREATORChristopher Mohri, John Duchi, Tatsunori Hashimoto
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48270
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19407

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Bitter Lesson for Data Filtering Christopher Mohri, John Duchi, Tatsunori Hashimoto Abstract We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an apparently common belief that filtering data to include only high-quality information is essential, our experiments suggest that with enough compute, the best data filter is no data filter. We find that sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally “poor” data. 1 Introduction The standard approach to select pretraining data for language models is to filter text from sources like Common Crawl (CC) ( 12 ) . It is widely documented that in compute-constrained regimes, where one must train on a subset of CC, different data selection strategies can have a large impact on performance. This is intuitive: all else equal, it seems natural to train on “higher-quality” data. As a result, a large body of research has emerged to tackle the data selection problem, with the goal of finding the best subset for pretraining language models ( 1 ; 22 ) . However, not only is large-scale filter ablation heuristic and expensive, but filtering removes data, which is at odds with scaling trends that prescribe ever-increasing amounts of data to improve model performance. For example, the heavily-filtered DCLM-Baseline dataset keeps \sim\!1\% of the original CC, leading to about 3.8 trillion tokens ( 22 ) . While this is still enormous, it falls short of the Chinchilla-optimal token budget for a 1 trillion parameter model, even after accounting for diminishing returns when epoching ( 25 ) . The current trend is also to over-train relative to Chinchilla-optimal, which prescribes even more tokens to allow for (relatively) smaller models that are financially feasible to serve ( 34 ) . We begin by testing the hypothesis that data filtering is necessary at all in the large compute limit. While large-scale machine learning has moved toward task-agnostic pretraining ( 30 ) , and there is anecdotal evidence that larger computational budgets benefit from looser data filters ( 16 ; 25 ) , removing all data filtering would be an extreme intervention that uses data considered to be actively harmful ( 30 ) . Our goal in this work is to take this extreme seriously and study the limits of (low-quality) data for transformer pretraining. We find evidence that rejects the hypothesis that data filtering is necessary, and that eventually, no existing data filter is likely to improve upon training directly on Common Crawl. In our experiments, we scale down both CC and its filtered versions to keep their relative sizes intact, and then scale computational resources for pretraining on these different datasets. Our two main levers to do so are scaling model size (which requires more compute per training step) and training steps (which eventually leads to epoching). When comparing the best achieved performance, regardless of computational cost, our main finding is that the full pool outperforms our selected filters. Our findings are robust as we scale our experiments by 2 orders of magnitude, and we find that we can continue to see the effects from our small pool experiments as long as the models are sufficiently large. Furthermore, we find a predictable relationship between pool size, training steps, and model size which enables us to build scaling laws that predict how much compute is needed for no filter to be optimal for a particular pool size. Using this, we find that the 240 trillion token Common Crawl pool from DCLM-Pool may become optimal as soon as 1e+30 FLOPs. These initial findings lead us to study the robustness of pretraining to “junk” data. Surprisingly, sufficiently large models are highly robust to irrelevant or junk data and can extract useful information even from highly noisy data. We test this using randomly generated strings and documents with shuffled word orders. While performance degrades at low compute budgets, sufficiently trained large models close the gap. Remarkably, these models even benefit from shuffled-word documents, despite only the unigram distribution of the documents remaining intact. Overall, our experiments suggest that sufficiently large models that are trained for sufficiently long can benefit from the full CC dataset. While it is possible to construct harmful data, which could for example be non-factual content that looks identical to high-quality data, we do not find large amounts of this in CC. As a result, data filtering may suffer from the bitter lesson ( 36 ) in which human-designed filters that perform well at the small scale are eventually replaced by simple, no-filter approaches that scale more gracefully with compute. We structure the paper as follows. In Section 2 , we provide the basic experimental setup, followed by experiments on filtering in Section 3 . We then move to adding data to our CC pool in Section 4 , and scaling the pool size in Section 5 . We finish with edge cases in Section 6 and a theoretical model in Section 7 to provide a post-hoc explanation of the observed phenomena. 1.1 Related Work Data-constrained pretraining. Several prior works consider the data-constrained pretraining regime. 25 derive scaling laws that factor data repetition into the original Chinchilla scaling laws, finding diminishing returns after around 4 epochs on the data and that adding code data and using looser perplexity-based filters mitigates data scarcity. However, the authors recommend filtering “noisy datasets” and train on subsets of C4 ( 30 ) , while the current work directly trains on (parsed) Common Crawl and finds evidence in support of no filtering. 20 study the question of algorithmic improvements in a data-constrained but compute-unlimited setting. We share a similar experimental setup (where we take subsets of a dataset, scale compute on this subset, and then scale the subset size) but differ in the object of analysis (dataset filtering). Loose data filters. The closest work to ours is 16 , who argue that filter thresholds should depend on the compute budget, showing evidence for vision-language models. They derive a scaling law to predict the filtering threshold as a function of compute budget, and conclude that “less aggressive filtering is best” with “large compute” but do not identify the parameter scaling interactions that are critical to our work, and do not show our main findings that for language models, no filter can be the best filter. 13 tackle a related question by artificially repeating “high-quality” data to match the scale of loosely filtered data. They find that the former can outperform the latter in low-compute regimes, but the high compute regime studied in this work remains fully speculative in their work. Finally, 14 finds that filtering aggressively can hurt performance, speculating that this follows from Goodhart’s law [ 15 ], and 32 find that filtering with a quality classifier may improve downstream benchmarks but not validation losses on “high-quality” data. On the theoretical side, 10 develop theoretical models of the data cleaning process, arguing that given models that have enough fidelity to model noisy data generation schemes, it is better to not clean data, while cleaning data can yield more robust learning when models are not perfect. This prediction dovetails with our subsequent findings.

Read the complete text