DataS 3 : Dataset Subset Selection for Specialization

Neha Hulkund, Alaa Maalouf, Levi Cai, Daniel Yang, Tsun-Hsuan Wang, Abigail O’Neil, Timm Haucke, Sandeep Mukherjee, Vikram V. Ramaswamy, Judy Hanwen Shen, Gabriel Tseng, Mike Walmsley, Daniela Rus, Ken Goldberg, Hannah Kerner, Irene Y. Chen, Yogesh Girdhar, Sara Beery · 2025

Read DataS 3 : Dataset Subset Selection for Specialization by Neha Hulkund, Alaa Maalouf, Levi Cai, Daniel Yang, Tsun-Hsuan Wang, Abigail O’Neil, Timm Haucke, Sandeep Mukherjee, Vikram V. Ramaswamy, Judy Hanwen Shen, Gabriel Tseng, Mike Walmsley, Daniela Rus, Ken Goldberg, Hannah Kerner, Irene Y. Chen, Yogesh Girdhar, Sara Beery (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16277.txt.

Introduction

Neha Hulkund and co-authors argue that machine learning models should train on data chosen for the exact place they will work. A bone-fracture model tuned for hospitals in general may stumble in one specific hospital. Camera-trap models face the same gap between the average forest and one park. The team formalizes the fix: given a general training pool and a sample from the target deployment, pick the subset that will serve that deployment best.

They built the first benchmark for this problem, spanning several real application domains, each with distinct deployments to specialize for. Standard selection methods from other families consistently failed at the job. Hand-picked expert subsets, by contrast, beat training on the broad pool. The authors present both the benchmark and the warning: methods tuned for the general case do not automatically specialize.

Catalog record

CREATORNeha Hulkund, Alaa Maalouf, Levi Cai, Daniel Yang, Tsun-Hsuan Wang, Abigail O’Neil, Timm Haucke, Sandeep Mukherjee, Vikram V. Ramaswamy, Judy Hanwen Shen, Gabriel Tseng, Mike Walmsley, Daniela Rus, Ken Goldberg, Hannah Kerner, Irene Y. Chen, Yogesh Girdhar, Sara Beery
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16277

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DataS 3 : Dataset Subset Selection for Specialization Neha Hulkund, Alaa Maalouf, Levi Cai, Daniel Yang, Tsun-Hsuan Wang, Abigail O’Neil, Timm Haucke, Sandeep Mukherjee, Vikram V. Ramaswamy, Judy Hanwen Shen, Gabriel Tseng, Mike Walmsley, Daniela Rus, Ken Goldberg, Hannah Kerner, Irene Y. Chen, Yogesh Girdhar, Sara Beery Abstract In many real-world machine learning (ML) applications (e.g. detecting broken bones in x-rays or species in camera traps), models need to perform well on specific deployments (e.g. a specific hospital or national park) rather than the domain broadly. However, deployments often have imbalanced, unique data distributions. Discrepancies between training and deployment distributions lead to suboptimal performance, highlighting the need to curate training data for specialization. We formalize dataset subset selection for specialization ( DS3 ) : given a training set drawn from a general distribution and a (potentially unlabeled) query set drawn from the desired deployment-specific distribution, the goal is to select a subset of the training data that optimizes deployment performance. We introduce DataS 3 ; the first dataset and benchmark designed specifically for the DS3 problem. DataS 3 encompasses five real-world application domains, each with a set of distinct deployments to specialize in. We conduct a comprehensive study evaluating algorithms from various families—including coresets, data filtering, and data curation, and find that methods trained on general distributions consistently fail to specialize and perform optimally on deployment tasks. Additionally, we demonstrate the existence of curated (deployment-specific) subsets that outperform training on all available data by up to 51.3 % 51.3\% . Our benchmark highlights the critical role of tailored dataset curation in enhancing performance and training efficiency on deployment-specific distributions, which we posit will only become more important as global, public datasets become available across domains and ML models are deployed in the real world. Figure 1: Foundation model training aims for broad generalization, by using all data available, usually from massive internet-scale datasets. In practice, we find these models are often suboptimal for specific deployments, which may exhibit different distributions over categories or data characteristics from the general training data pool. Dataset subset selection for specialization seeks to identify model training subsets closely aligned with the target deployment, achieving superior performance under the given distribution and attribute shifts. 1 Background and Motivation Machine learning models are typically trained on large datasets with the assumption that the training distribution closely matches the distribution of the deployment where the model will be applied. However, in real-world applications, deployment data distributions often diverge from general and/or global training set distributions Shen et al. 2024 ; Taori et al. 2020 . Selecting relevant data subsets aligned with specific deployments is crucial to maximize field performance. The problem of data subset selection for specialization ( DS3 ) is thus critical: given all available training data for a domain and a small (usually unlabeled) query set that represents the desired deployment, the goal is to identify a subset of the training data, such that training the ML model on this subset maximizes performance on the deployment distribution. Real world example. Consider a wildlife ecologist who aims to build a classifier to detect the presence of invasive rodents in camera trap images collected at the Channel Islands. Existing labeled training data on invasive rodents in this context is limited, as they have been mostly eradicated by previous successful conservation action, thus training a classifier from scratch is likely to be unsuccessful. A common approach is to finetune a general pre-trained model (such as ViT or CLIP) on all relevant camera trap data. But what does "relevant data" mean? Would using similar species data from other camera trap locations (perhaps on the mainland) improve performance, or introduce noise? What about including data from non-similar species at that location? While adding data to a training set can sometimes improve performance, it can also decrease individual subgroup performance in a biased way Compton et al. 2023 and introduce spurious correlations that can enable models to learn potentially dangerous “shortcuts,” resulting in biased predictions, shown across various domains Geirhos et al. 2020 ; Badgeley et al. 2018 ; Wang et al. 2021 ; Beery et al. 2022a . The Gap: General datasets vs. deployment-specific needs. To address the gap between general models and specific deployment needs, we highlight the need for research on DS3 : the development of methods that select optimal training data for deployment-specific model specialization. Similarly, subset selection methods are evaluated on standard CIFAR10/100 Krizhevsky et al. 2009 and ImageNet Deng et al. 2009b datasets, where test and validation sets have similar distribution to their training sets. Current benchmarks for data filtering Gadre et al. 2024 focus on generalization across many tasks, in contrast to specialization for a particular deployment. While these works are valuable, they do not capture or enable progress on the DS3 challenge. Our contributions. We propose DataS 3 , a comprehensive benchmark to evaluate and compare deployment-specialization subset selection methods. Our key contributions are the following: (i) DataS 3 : A benchmark of five datasets for evaluating algorithms on the DS3 problem. The datasets represent real-world scientific and engineering applications from different fields. Each dataset includes multiple realistic deployments, e.g., the camera traps dataset aims to categorize species, and the deployments to specialize to are geographic locations where scientists want to analyze species, e.g., Central Africa or Southeast Asia. (ii) We show that selecting a well-curated subset can consistently outperform models trained on the entire dataset with proof-of-concept manually curated subsets of the training data for each deployment. (iii) An extensive experimental study comparing current SOTA subset selection methods on DataS 3 . After training a suite of baselines, our results clearly show that current subset selection methods fail on DS3 , highlighting the need for our DataS 3 benchmark. 2 Problem Statement DS3 problem formulation. Let X be a ground set of data points, T\subset X be a given training set drawn from a training (pool) distribution P_{T} over X , and let Q\subset X be a query set drawn from the desired deployment-specific distribution P_{Q} over X . Given a model \theta , the objective of dataset subset selection for specialization ( DS3 ) , is to design an algorithm SubsetSelection-ALG , which takes T (the training set) and Q (the deployment representative query set) as input, and outputs a subset S^{*}\subset T that minimizes the expected loss of \theta trained on S^{*} over the desired deployment-specific distribution P_{Q} . More formally: S^{*}=\argmin_{S\subset T}\mathbb{E}_{q\sim P_{Q}}\left[\mathcal{L}(\theta(S),q)\right], (1) where \theta(S) denotes the model trained on the subset S\subset T , and \mathcal{L}(\theta(S),q) is the loss function evaluated on a single point q sampled from P_{Q} and the trained model \theta(S) . The term \mathbb{E}_{q\sim P_{Q}} denotes the expected value over the distribution P_{Q} . Hence, the algorithm SubsetSelection-ALG outputs S^{*} , the subset of T that minimizes the expected loss of the entire desired deployment distribution P_{Q} . Notably, SubsetSelection-ALG can only access the desired deployment-specific distribution via the query set Q . Is the query set annotated/labeled? This formalization can be divided into two cases: in the first, the query set Q is annotated with a set of labels, formally, Q is a set of m>0 pairs Q=\{(q_{1},y_{1}),\cdots,(q_{m},y_{m})\} , where for every i\in[m]

Read the complete text