Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of Korea DOI: 10.1145/3770855.3817569 ISBN: 979-8-4007-2259-2/2026/08 CCS: Information systems Data mining CCS: Computing methodologies Active learning settings
Rostislav Gusev, Alexey Zaytsev · 2026
Read Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of Korea DOI: 10.1145/3770855.3817569 ISBN: 979-8-4007-2259-2/2026/08 CCS: Information systems Data mining CCS: Computing methodologies Active learning settings by Rostislav Gusev, Alexey Zaytsev (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27997.txt.
Introduction
Benchmarks with many datasets make model evaluation expensive. Selecting a small representative subset is the usual fix, but heuristics rarely check whether rankings survive. Rostislav Gusev and Alexey Zaytsev introduce a framework in 2026 for dataset selection judged by rank preservation. Bootstrap aggregation supplies valid confidence intervals, so methods can be compared on principle. Clustering, design criteria, random baselines, and greedy farthest-first selection are all tested.
Three domains supply the evidence, with 112 time series datasets, 57 language tasks, and 30 recommender datasets. In time series, the best strategy reaches a Spearman correlation of zero point nine five with full-benchmark rankings using only five datasets. Several methods beat random subsets there and in the language benchmark. In recommender systems, gains over random are small and usually insignificant. How well a method works depends on representation quality and benchmark size.
Catalog record
| CREATOR | Rostislav Gusev, Alexey Zaytsev |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27997
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of Korea DOI: 10.1145/3770855.3817569 ISBN: 979-8-4007-2259-2/2026/08 CCS: Information systems Data mining CCS: Computing methodologies Active learning settings Rostislav Gusev, Alexey Zaytsev Abstract . Benchmarks of machine learning models often include many datasets, making evaluation expensive. For efficiency, it is preferable to perform evaluations on small, representative datasets instead. The selection of such subsets typically relies on heuristics and is rarely analyzed for the robustness of the resulting model rankings. We introduce a framework to perform the task of selecting datasets subsets with an evaluation of how different selection strategies preserve the global model rankings. Our framework includes bootstrap aggregation, which provides valid confidence intervals, allowing a principled comparison of selection strategies. We consider clustering, design criteria (A/D-optimality), random baselines, and greedy farthest-first (FAFI). For the latter, we derive upper bounds on selection quality in terms of ranking errors as a function of the number of selected datasets. Empirically, in time series classification (TSC, 112 datasets) and in a supplementary natural language processing benchmark derived from MTEB (57 tasks), several selection strategies improve rank preservation compared with random subsets, including simple FAFI. In contrast, in recommender systems (30 datasets), the improvement of strategies over random selection is small and typically statistically insignificant. For TSC, our best-performing strategy achieves a Spearman correlation of 0.95 with the full benchmark model rankings using only five selected datasets. Additional experiments indicate that the effectiveness of selection approaches depends on both the quality of dataset representations and the scale of the benchmarking regime. † † cc-license: by 1. Introduction Multi-dataset benchmarks have become the default way to evaluate progress: rather than relying on a single task, new methods are expected to demonstrate robust improvements across collections of datasets. This paradigm is particularly prominent in time series classification (TSC), where community repositories such as UCR/UEA enable large-scale, standardized comparisons ( 10 ; 2 ) . At the same time, exhaustive evaluation is increasingly resource-intensive: the number of datasets grows, and modern models (including deep architectures and ensembles) often make exhaustive model-by-model evaluation prohibitively expensive. Consequently, practitioners routinely evaluate on smaller subsets of datasets, sometimes chosen implicitly, manually, or simply by convenience. This practice raises two challenges. First, conclusions drawn from a small subset can be statistically unstable: comparing methods on only a few datasets rarely supports reliable claims about overall superiority and requires careful multiple-dataset statistical procedures ( 14 ) . Second, subset choice can introduce selection bias: when the benchmark composition is flexible, the apparent leader may be an artifact of the selected tasks rather than a genuine improvement. This issue is closely related to the benchmark lottery effect, where rankings can change substantially under different task selections, especially when many models/configurations are compared ( 11 ) . Figure 1. We study how to select a small subset of datasets that best preserves the global model ranking of a large benchmark. Datasets are embedded via task features, subsets are chosen by multiple strategies (geometric, clustering, design-based), and an evaluation protocol with bootstrap aggregation yields rank-preservation metrics, enabling comparisons across subset sizes and domains (time series classification, recommender systems, natural language processing). These concerns motivate the following question: can we select a small but representative subset of datasets that substantially reduces evaluation cost while preserving the conclusions of full-benchmark model comparisons? In this work, we treat datasets subset selection for benchmarking as a standalone problem. We focus on preserving the global ranking of models induced by the full benchmark and consider selection strategies that rely only on a priori dataset representations (meta-features, time-series feature extractors, landmarking features) rather than on full model evaluation results. Our contributions are threefold. • A unified evaluation protocol for efficient dataset selection methods. We propose a unified evaluation protocol for dataset selection methods, based on repeated subsampling of the available pool and bootstrap-style uncertainty estimation over performance-vs.-subset-size curves. • Empirical evaluation for three domains: time series classification, RecSys, and NLP. Within this protocol, we systematically compare several families of selection strategies — random baselines, geometric diversity heuristics (e.g., farthest-first), clustering-based selection, and criteria inspired by optimal experimental design — across multiple dataset representations in two domains (TSC and recommender systems), and additionally test the portability of the protocol on an NLP benchmark derived from MTEB ( 22 ) . • Theoretical examination for the farthest-first selection method. For the farthest-first selection algorithm, we derive theoretical convergence guarantees linking geometric coverage in representation space to statistical accuracy: as the subset size grows, posterior uncertainty over model performance contracts and errors in model ranking decay at rates governed by the covering properties of farthest-first. We also conduct additional experiments and ablations that extend the empirical picture beyond aggregate benchmark results, helping to clarify when and why different selection principles succeed or fail across representations and domains. The code is publicly available. 1 1 1 The code to reproduce all experiments, including the benchmarking protocol and dataset selection strategies, is available at https://github.com/hangover137/efficient_benchmarking . 2. Related Work Benchmarking across multiple datasets is widely recognized as necessary for making reliable claims about algorithmic superiority ( 14 ; 3 ; 5 ) . In TSC, the UCR/UEA ecosystem and associated bake-off studies have highlighted how evaluation outcomes depend on benchmark composition, protocols, and baseline choices ( 10 ; 2 ; 18 ) . Beyond TSC, the benchmark lottery perspective shows that even modest changes in task sets can produce different winners and unstable rankings when many models are compared ( 11 ; 24 ; 35 ; 26 ; 6 ) . In recommender systems, large-scale benchmarking efforts likewise emphasize dataset heterogeneity and the sensitivity of algorithm rankings to the chosen dataset pool ( 28 ; 16 ; 8 ; 27 ; 7 ) . These lines of work collectively motivate studying rank-preserving reductions of benchmark size rather than optimizing for a fixed, small set. Open benchmarking platforms aim to standardize protocols and improve reproducibility by publishing datasets, splits, and results, in line with the FAIR guiding principles ( 34 ) . OpenML is a prominent example, enabling systematic large-scale comparisons through shared tasks and run records ( 32 ) . Curated suites such as OpenML-CC18 further fix dataset selections and partitions to ensure comparability across studies ( 4 ; 33 ; 36 ; 31 ) . However, curated suites are typically static: they do not address how to adaptively construct a compact benchmark when the available dataset pool, model zoo, or evaluation objectives change. Our setting complements these efforts by treating benchmark construction itself as an algorithmic selection problem with an explicit rank-preservation objective. Selecting representative subsets is a classical problem in clustering and geometric approximation. The greedy farthest-first traversal (often viewed through the k -center / maximin lens) provides a simple, scalable way to obtain dispersed subsets and comes with approximation guarantees ( 17 ) . Similarly, k -means clustering and initialization schemes such as k -means++ are widely used