Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining
Mikey Shechter, Yair Carmon · 2025
Read Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining by Mikey Shechter, Yair Carmon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08805.txt.
Introduction
Mikey Shechter and Yair Carmon published a research paper in 2025 on choosing training data for image-text models by learning from the tests themselves. Big vision-language models are only as good as the web data they are filtered from. Their algorithm trains a scorer that judges each example's usefulness using signals from real downstream task data.
A mixing variant blends several scorers into one, and a soft sampling rule draws training sets from the scores without over-repeating favorites. On a standard filtering benchmark the method hit a new best zero-shot accuracy, two points above all prior work and five and a half above methods using only public resources. Scores across thirty-eight tasks also led the public field.
Catalog record
| CREATOR | Mikey Shechter, Yair Carmon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08805
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining Mikey Shechter, Yair Carmon Abstract We introduce Filter Like You Test (FLYT), an algorithm for curating large-scale vision-language datasets that learns the usefulness of each data point as a pretraining example. FLYT trains a scoring model that learns to weigh each example’s features using gradient signals from downstream tasks training sets. Based on FLYT, we implement Mixing-FLYT (M-FLYT), which takes the per-example scores generated by different scoring methods as features, and learns to unify them into a single score. FLYT naturally produces a distribution over the training examples, which we leverage through Soft Cap Sampling (SCS), a strategy for obtaining a filtered pretraining dataset from per-example probabilities that samples examples while preventing over-representation through a repetition penalty. Using these methods, we achieve 40.1% ImageNet zero-shot accuracy on the DataComp medium scale filtering benchmark, a 2% absolute accuracy increase over all previous results and a 5.5% increase over results that—like us—use only public resources. Our approach also yields 37.7% on the average of 38 DataComp evaluation tasks, outperforming previous public-resource approaches by 0.4%. 1 Introduction [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] Figure 1: In each FLYT training loop, the scoring model takes features extracted from a batch of upstream data and generates a score for each example. These scores are converted to weights via softmax. A reference model processes the upstream batch, and the resulting embeddings, together with the weights, are used to compute a weighted CLIP loss. The reference model is then updated using gradients from this loss. Next, this updated reference model processes downstream data to compute a downstream loss, which produces a gradient signal that passes through the updated reference model all the way to the scoring model parameters. Filtering the pretraining data of CLIP models [ 35 ] has a dramatic effect on the resulting model’s quality [ 39 , 40 , 17 , 51 ] . Despite intense research and considerable progress, the best existing methods select data based on heuristic proxies for usefulness [ 16 , 22 , 49 , 29 , 27 , 21 ] . In this work, we take a different approach: adhering to the core principle of machine learning, we optimize a loss function that measures the usefulness of data for pretraining . We consider the setting of the DataComp filtering benchmark [ 17 ] . Our goal is to find a filtering method that selects a subset from a candidate pool of web-sourced image-text training data. To evaluate the quality of the selected subset, it is fed to a fixed training pipeline and the resulting model is evaluated on a fixed set of zero-shot image comprehension tasks. The DataComp rules do not constrain the computational cost of the filtering method. Moreover, DataComp allows using external data (except evaluation test sets) for making the filtering decisions, so long as the selected training set is a subset of the candidate pool. In this setting, we introduce methods that together achieve state-of-the-art performance on the DataComp medium scale. At the core is Filter Like You Test 1 1 1 The name is inspired by FLYP [ 18 ] . (FLYT), a method for training a scoring model that predicts the weight an image-text pair should have in CLIP pretraining. Mixing-FLYT (M-FLYT) is an implementation of FLYT that uses scores generated by existing scoring methods as input features to the scoring model. Complementing FLYT, Soft Cap Sampling (SCS) is a simple sampling strategy that leverages the probabilistic structure of FLYT’s scores to construct the final pretraining dataset. Filter Like You Test (FLYT) ( Section 3.2 ). FLYT recasts data selection as data re-weighting [ 37 ] , and learns to upweight (and hence select) the pretraining data that contributes the most to the model’s downstream performance. We jointly train two models: a scoring model that evaluates the usefulness of training examples using extracted features, and a reference model that provides feedback on these evaluations. During training, the scoring model assigns weights to each example in the batch, which are then used in our weighted CLIP loss function to update the reference model. The updated reference model is then evaluated on downstream data, allowing gradient signal from the downstream loss to flow through it, to the weights, and finally back to the scoring model. This allows the scoring model to give higher weights to examples that reduce the downstream loss of the reference model. We then use the trained scoring model to generate per-example scores. Figure 1 illustrates a single training step of FLYT and Algorithm 1 provides pseudocode. Mixing-FLYT (M-FLYT) ( Section 3.4 ). Previous work aggregates different filtering methods to obtain the best results. They either sum together different scores [ 22 ] , take the intersection/union of the filtered datasets [ 17 , 29 , 22 , 49 ] or pipeline the filtering methods one after the other [ 53 , 49 ] . Departing from heuristic approaches, we leverage FLYT to directly optimize over a space of mixing methods defined by a simple linear combination of scores. Replacing the aggregation method of [ 22 ] with M-FLYT improves ImageNet accuracy by 1.3%. More importantly, M-FLYT can aggregate more scores. When used with 12 different scores, it outperforms the best single score by 1.7% on ImageNet accuracy, and by at least 0.8% on average accuracy. M-FLYT outperforms all baseline aggregation methods we evaluated, including summation and ImageNet accuracy-weighted sums by 0.5% on ImageNet accuracy and 0.6% on average accuracy. A different natural choice for FLYT’s feature extractor component is a multimodal neural network, such as CLIP; we refer to this Embeddings-FLYT (E-FLYT). While E-FLYT does not improve the state-of-the-art on its own, incorporating its scores as an input to M-FLYT leads to improved overall performance. Soft Cap Sampling (SCS) ( Section 3.6 ). FLYT produces a probability distribution over the training examples, allowing us to obtain a filtered subset by simply sampling from the distribution. However, sampling with replacement produces too much repetition of the top-scoring examples, and sampling without replacement produces too few. To interpolate between the two sampling methods we propose SCS, a strategy that samples with replacement but reduces repetition using a penalty term (see Algorithm 2 ). Using only publicly available resources, M-FLYT with SCS has 40.1% ImageNet zero-shot accuracy, which is a 2% absolute increase compared to the best prior work, and a 5.5% improvement compared to the best prior work that uses only publicly available resources. Our filtered dataset also achieves 37.7% average accuracy across all evaluation tasks, which is within 0.1% of our reproduction of the best previously published filtering method, and a 0.4% improvement over the best prior work using only publicly available resources. On the DataComp small scale benchmark (which we did not evaluate until the end of the project) our method outperforms previous work on both ImageNet and average accuracy (see Table 4 ). Algorithm 1 Filter Like You Test (FLYT) Input: Scoring model {q}_{{\phi}_{0}} with initial parameters {\phi}_{0} , reference model {f}_{{\theta}_{0}} with initial parameters {\theta}_{0} , feature extractor {\Psi} , upstream dataset {D} , downstream dataset {C} 1 2