Reducing cross-sample prediction churn in scientific machine learning

Gordan Prastalo [email protected], Kevin Maik Jablonka [email protected] · 2026

Read Reducing cross-sample prediction churn in scientific machine learning by Gordan Prastalo [email protected], Kevin Maik Jablonka [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13826.txt.

Introduction

Gordan Prastalo and Kevin Maik Jablonka study cross-sample prediction churn in scientific machine learning. Two classifiers trained on separate bootstraps of the same training set agree in aggregate accuracy within 1.3 to 4.2 percentage points. Two data-side methods do: K-bootstrap bagging cuts the rate 40 to 54 percent on every dataset at no accuracy cost. At matched twice-ERM compute it reduces churn a further median 45 percent beyond bagging with K equal to 2.

Catalog record

CREATORGordan Prastalo [email protected], Kevin Maik Jablonka [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93869
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13826

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reducing cross-sample prediction churn in scientific machine learning Gordan Prastalo [email protected], Kevin Maik Jablonka [email protected] Abstract Scientific machine learning reports predictive performance. It does not report whether the same prediction would survive a different draw of training data. Across 9 9 chemistry benchmarks, two classifiers trained on independent bootstraps of the same training set agree on aggregate accuracy to within 1.3 ​ – ​ 4.2 1.3\text{--}4.2 percentage points but disagree on the class label of 8.0 ​ – ​ 21.8 % 8.0\text{--}21.8\% of test molecules. We call this gap cross-sample prediction churn . The standard parameter-side techniques (deep ensembles, MC dropout, stochastic weight averaging) do not reduce this gap; two data-side methods do. The first is K K -bootstrap bagging, which cuts the rate 40 ​ – ​ 54 % 40\text{--}54\% on every dataset at no accuracy cost ( K × K{\times} -ERM compute). The second is twin-bootstrap , our proposal: two networks trained jointly on independent bootstraps with a sym-KL consistency loss between their predictions, which at matched 2 × 2{\times} -ERM compute reduces churn a further median 45 % 45\% beyond bagging- K = 2 K{=}2 . Cross-sample prediction churn deserves a column alongside predictive performance in scientific-ML benchmark reports, because without it the parameter-side and data-side methods are indistinguishable on the metric they actually differ on. 1 Helmholtz-Zentrum Berlin für Materialien und Energie GmbH, Hahn-Meitner-Platz 1, 14109 Berlin, Germany 2 HIPOLE Jena (Helmholtz Institute for Polymers in Energy Applications Jena), Lessingstrasse 12–14, 07743 Jena, Germany 3 Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldtstrasse 10, 07743 Jena, Germany 4 Center for Energy and Environmental Chemistry Jena (CEEC Jena), Friedrich Schiller University Jena, Philosophenweg 7a, 07743 Jena, Germany 5 Jena Center for Soft Matter (JCSM), Friedrich Schiller University Jena, Philosophenweg 7, 07743 Jena, Germany 1 Introduction A bioactivity classifier is retrained on a new batch of assay results. Aggregate accuracy moves by 1.8 percentage points; the class label flips on 16.1\% of test molecules (Figure 1 ). We call this cross-sample prediction churn — the fraction of test predictions that change class between two models trained on independent samples of the same training population. This is a per-prediction stability gap that aggregate-performance reporting in scientific-ML benchmarks systematically hides. In closed-loop laboratories, Bayesian-optimisation campaigns, and virtual-screening pipelines, model predictions feed directly into experimental decisions: the molecule selected for synthesis, the candidate prioritised in a screening campaign, the hit returned by a virtual screen. Churn is the rate at which retraining changes which molecules the model selects, and is therefore the operational stability the practitioner cares about. Figure 1: Twin-bootstrap eliminates most of ERM’s retraining-induced prediction flips on BACE. Each row is one of the 80 test molecules with the largest cross-sample contrast; each column is one of ten retrainings on an independent bootstrap of the BACE training pool; cells are coloured by predicted class. Visible vertical stripes in the left (ERM) panel are predictions that flip class across retrainings; under twin-bootstrap (right; \lambda{=}300 , matched compute against bagging- K{=}2 ) the same molecules become near-uniform. The mean class-flip rate over the full test set drops from 16.1\% (ERM) to 5.7\% (twin-bootstrap). Chemistry models are designed to be invariant to atomic permutation, rotation, and translation; E(3) -equivariant message passing and permutation-equivariant aggregators are standard. Sample invariance — invariance to which independent draw of the training population the model was fit on — is the analogue on the data axis, but is not routinely measured in scientific ML. However, a practitioner who updates a screening set, retrains, and reroutes synthesis based on the new top predictions confronts the challenge of cross-sample churn on every dataset update. Two existing approaches are conceptually nearby but do not measure or reduce this gap. Prediction churn ( 7 ; 2 ; 14 ) measures retraining-induced disagreement on web-scale data; prior work varies training conditions on a fixed dataset or compares a candidate model against a deployed incumbent. In scientific ML there is often no incumbent, and the dominant source of variance is the small training set itself, redrawn between retrainings. Epistemic uncertainty tools — deep ensembles ( 15 ) , Monte Carlo (MC) dropout ( 9 ) , and stochastic weight averaging (SWA) ( 12 ) — sample over weights at fixed data and so capture the parameter-side slice of model variance, but not the data-sampling slice we measure: across our 9 chemistry benchmarks, the three together shift the class-flip rate by -22.3\% to +12.5\% relative to empirical risk minimisation (ERM), with no consistent sign. Methods that vary the data — bagging ( 4 ) and the twin-bootstrap procedure we propose — do reduce it. Contributions. • Cross-sample prediction churn. The data-sampling analogue of model variance — the slice of uncertainty that deep ensembles, MC dropout, and SWA all miss because they sample over weights at fixed data (Section 3 ). • Magnitudes. On 9 chemistry benchmarks (MoleculeNet, TDC ADME and Tox, materials-science), per-prediction churn flips 8.0\text{--}21.8\% of test predictions while aggregate accuracy moves only 1.3\text{--}4.2 pp (Table 1 ). • Bagging. K -bootstrap bagging ( 4 ) cuts churn 40\text{--}54\% on every dataset at no accuracy cost, at K\times -ERM compute (Table 2 ). • Twin-bootstrap. Two networks are trained jointly on independent bootstraps with a symmetric-KL consistency loss between their predictions ( \sim 40\% inter-network data overlap by construction). At matched 2\times -ERM compute it cuts churn a further median 45\% beyond bagging- K{=}2 and matches 5\times -compute bagging- K{=}5 in mean rank (Table 2 , Figure 2 ); the same reduction shows up at the Bayesian-optimisation acquisition layer (Appendices P and Q ). • Per-prediction triage. On top of any deployed model, sorting by per-example churn from a single extra retraining captures 58\text{--}100\% of class flips at the top- 30\% review fraction; better than predictive entropy on every dataset (Section 6 ). 2 Related work Prediction churn. 7 introduced prediction churn as the disagreement between two classifiers trained on the same task with different runs or data updates; their MCMC stabiliser and 14 ’s distillation-from-incumbent are deployment-loop methods that compare a candidate model against a frozen incumbent on web-scale data. 2 instead hold the training data fixed and isolate parameter-side variance (initialisation, mini-batch order, hardware) as the source of churn on CIFAR/ImageNet. Codistillation operating points on the data-overlap axis. Two-network sym-KL consistency has been applied at two operating points on the inter-network data-overlap axis: 1 train on disjoint shards ( 0\% overlap) with a KL agreement loss for distributed-training throughput; 2 share the entire training set ( 100\% overlap) for parameter-side reproducibility. Our twin-bootstrap inherits the sym-KL loss form and applies it at a third operating point: independent bootstraps of the canonical training pool, sharing {\sim}40\% of indices in expectation (Appendix I ). Predictive multiplicity and the Rashomon set. The Rashomon set ( 5 ) is the set of near-loss-equivalent models on a fixed dataset; 17 operationalise this as predictive multiplicity , 8 attribute it to under-specified pipelines, and 3 build selective ensembles that abstain when ensemble consistency cannot be certified by a hypothesis test. These works hold the dataset fixed and vary models within the loss-equivalent class — orthogonal to cross-sample churn we discuss here, where the

Read the complete text