RobustModelMaker: Coupling Bootstrap Stability Selection with Leakage-Safe Nested Cross-Validation for Scientific Machine Learning
Amanda S Barnard · 2026
Read RobustModelMaker: Coupling Bootstrap Stability Selection with Leakage-Safe Nested Cross-Validation for Scientific Machine Learning by Amanda S Barnard (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01566.txt.
Introduction
Barnard builds RobustModelMaker, a framework for sound scientific machine learning. Small scientific datasets strain learning pipelines in two ways. Feature picks shift a lot under small data changes. Using the same data to select, tune, and evaluate inflates scores. These two failure modes interact in practice, she argues. Unstable picks raise the variance of an already optimistic score. Her framework couples bootstrap stability selection with nested cross-validation. All preprocessing and selection happen inside each fold.
It outputs a stability-tested feature subset plus a leakage-safe score. Nine algorithms span classification and regression tasks. A deterministic test suite checks unit, performance, and reproducibility behavior. Three real datasets compare it with three other selectors. It matches the best score on each dataset. Its joint score-stability position is unmatched across task types.
Catalog record
| CREATOR | Amanda S Barnard |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01566
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RobustModelMaker: Coupling Bootstrap Stability Selection with Leakage-Safe Nested Cross-Validation for Scientific Machine Learning Amanda S Barnard Abstract Small-to-medium scientific datasets place machine learning pipelines under two compounding pressures. Single-run feature selection produces feature sets that change substantially under small perturbations of the training data, and any procedure that uses the same data for selection, tuning, and evaluation produces optimistically biased performance estimates. The two failure modes are routinely treated as separable, but in the regimes where scientific data live, they interact: an unstable selection inflates the variance of an already-optimistic score, and standard remedies for one rarely address the other. RobustModelMaker is a Python framework that couples bootstrap stability selection with strict nested cross-validation, performs all preprocessing and selection inside each fold, and produces a stability-tested feature subset together with a leakage-safe performance estimate. The framework supports nine algorithms across binary classification, multiclass classification, and regression. Behaviour is verified by a deterministic test suite spanning unit, performance, and reproducibility checks on three real scientific datasets comparing to three alternative selectors (ANOVA F-test, recursive feature elimination with cross-validation, and Boruta) on both predictive score and a Jaccard measure of selection stability. RobustModelMaker is competitive in score with the best alternative selector on each dataset, and occupies a position on the joint score-stability frontier that none of the alternatives match across all three task types. Two example applications, ovarian cancer biomarker discovery from the PLCO Trial and critical-temperature regression on the UCI Superconductivity Data, illustrate how the framework is used in practice and what trade-offs become visible when stability is treated as a first-class deliverable rather than an emergent property. 1 Introduction Machine learning pipelines applied to scientific data must satisfy a set of demands that are not always shared by their industrial counterparts. Datasets are typically of modest size, in the hundreds to low tens of thousands of samples, with feature dimensionalities of comparable or larger order. Features are often correlated by construction, because they are derived statistics or measurements of related physical or biological quantities. Reproducibility is a working requirement, both because scientific conclusions depend on it and because the cost of acquiring additional samples is high. Performance on future data is the quantity of interest, not performance on the data already in hand. Several scientific domains share this regime and have long made feature selection a central component of their methodology rather than an optional pre-processing step. In genomics and biomarker discovery, candidate features routinely number in the thousands to millions, in the form of genes, SNPs, or assay readouts, while patient cohorts are in the hundreds; the goal is a small, defensible panel whose components can be validated experimentally and interpreted clinically [ 28 , 11 ] . In materials informatics, descriptors of composition, structure, or local atomic environment are typically computed as families of statistics over the same underlying physical quantity, which produces strong within-family correlation, and the analyst seeks the few descriptors that carry independent predictive signal [ 26 , 34 ] . In neuroimaging, candidate voxel- or region-of-interest features can exceed the number of subjects by orders of magnitude, and selected regions are expected to be both biologically interpretable and replicable across cohorts [ 23 ] . In high-energy physics, classifier inputs are drawn from large engineered feature spaces over event-level kinematic and topological quantities, where dimensionality reduction and stability against detector-level variation are routine concerns [ 10 ] . In each setting, a feature subset that changes substantially under resampling of the training data is not a useful scientific outcome, and a performance estimate that does not account for the joint selection-and-tuning process overstates the prospects for replication. Two methodological failure modes are routinely observed in this regime. The first is unstable feature selection. A method that ranks features by importance and retains those above a threshold will, in general, return a different feature set when the training data are perturbed. With modest sample sizes and correlated features, the selected subset can change substantially under bootstrap or cross-validation resamples of the same data, even when the underlying signal is unchanged. A feature set obtained by a single run of such a procedure does not generalise as a scientific finding: replication, when attempted, often selects different features. The second is the optimistic bias incurred when the same data are used for feature selection, hyperparameter tuning, and performance evaluation. A simple cross-validated score in which the outer fold also informs feature selection or hyperparameter choice is biased upward by an amount that grows with the search space and shrinks slowly with sample size [ 33 , 6 ] . The bias is often large enough to matter for scientific claims, particularly in small-to-medium regimes where the reported confidence interval is narrow but the estimator itself is shifted. These two failure modes are typically addressed in isolation. Stability selection [ 22 ] provides a principled correction for the first. Nested cross-validation [ 33 ] corrects the second. Standard tooling provides accessible implementations of each separately, but not a coupled pipeline in which the bootstrap stability selection is run inside each outer fold of the nested cross-validation, every preprocessing step is fitted on training partitions only, and the resulting selected feature set is itself the outcome of a held-out evaluation. Constructing such a pipeline correctly is non-trivial. The points at which data can leak from the test partition into model-building decisions are numerous, and each must be closed individually. The premise of this work is that stability selection and nested cross-validation are not optional add-ons but are jointly required when reproducibility and honest generalisation are both targets of the analysis. Applied separately they each address one failure mode but allow the other to dominate. Applied jointly, with the appropriate ordering and partitioning, they produce a feature set and a performance estimate that are individually defensible and mutually consistent. RobustModelMaker is a Python framework that implements this coupling. It is designed as a single-file library with a scikit-learn-compatible estimator interface, supports nine algorithms across binary classification, multiclass classification, and regression, and includes the determination of a specificity-targeted probability cutoff for binary classification as part of the standard output. The framework is intended for tabular scientific data at small-to-medium scale. It is not intended for streaming, out-of-core, or distributed-training workloads, and it does not attempt to compete with libraries optimised for raw throughput on very large datasets. Its design priorities are reproducibility, interpretability, and explicit honesty about the trade-offs that feature reduction imposes on predictive performance. The remainder of this paper proceeds as follows. Section 2 situates the framework in the literature on stability selection and nested cross-validation and contrasts it with common alternatives. Section 3 states the design principles. Section 4 describes the implementation. Section 5 reports the test and validation suite. Section 6 presents a statistical benchmark on three real scientific datasets. Section 7 illustrates use in two scientific applications, ovarian cancer biomarker discovery and superconductor critical-temperature regression. Section 8 concludes. 2 Problem Setting and Background Let \mathcal{D}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N} denote a labelled tabular dataset, where \mathbf{x}_{i}\in\mathbb{R}^{d} is a feature vector and y_{i} is the target. For classification, y_{i}\in\{1,\ldots,C\} ; for regression, y_{i}\in\mathbb{R} . The setting of interest is small-to-medium scale: N in the hundreds to low tens of thousands