Sensitivity of Stability: Theoretical & Empirical Analysis of Replicability for Adaptive Data Selection in Transfer Learning

Prabhav Singh [email protected], Jessica Sorrell [email protected] · 2025

Read Sensitivity of Stability: Theoretical & Empirical Analysis of Replicability for Adaptive Data Selection in Transfer Learning by Prabhav Singh [email protected], Jessica Sorrell [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04901.txt.

Introduction

Prabhav Singh and Jessica Sorrell study whether transfer learning results can be repeated when training data is picked adaptively, in a 2025 paper. Transfer learning adapts a pre-trained model to a new task, and adaptive picking favors some examples over others as training runs. The pair built a math frame for trade-offs between adaptation gains and result consistency. Their key measure tracks how strongly a picking strategy reacts to small data changes.

They prove the chance of two runs disagreeing grows with the square of that measure. It falls fast as sample size grows. Tests on a large language corpus, across six picking strategies, matched the theory closely. Bold strategies won on task scores but failed repeat tests far more often. Calm strategies kept failure under seven percent. Starting from a pre-trained model cut failure by up to thirty percent, a practical guard they recommend.

Catalog record

CREATORPrabhav Singh [email protected], Jessica Sorrell [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04901

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sensitivity of Stability: Theoretical & Empirical Analysis of Replicability for Adaptive Data Selection in Transfer Learning Prabhav Singh [email protected], Jessica Sorrell [email protected] Abstract The widespread adoption of transfer learning has revolutionized machine learning by enabling efficient adaptation of pre-trained models to new domains. However, the reliability of these adaptations remains poorly understood, particularly when using adaptive data selection strategies that dynamically prioritize training examples. We present a comprehensive theoretical and empirical analysis of replicability in transfer learning, introducing a mathematical framework that quantifies the fundamental trade-off between adaptation effectiveness and result consistency. Our key contribution is the formalization of selection sensitivity ( Δ Q \Delta_{Q} ), a measure that captures how adaptive selection strategies respond to perturbations in training data. We prove that replicability failure probability: the likelihood that two independent training runs produce models differing in performance by more than a threshold, increases quadratically with selection sensitivity while decreasing exponentially with sample size. Through extensive experiments on the MultiNLI corpus using six adaptive selection strategies—ranging from uniform sampling to gradient-based selection—we demonstrate that this theoretical relationship holds precisely in practice. Our results reveal that highly adaptive strategies like gradient-based and curriculum learning achieve superior task performance but suffer from high replicability failure rates, while less adaptive approaches maintain failure rates below 7%. Crucially, we show that source domain pretraining provides a powerful mitigation mechanism, reducing failure rates by up to 30% while preserving performance gains. These findings establish principled guidelines for practitioners to navigate the performance-replicability trade-off and highlight the need for replicability-aware design in modern transfer learning systems. 1 Introduction The ability to reliably reproduce experimental results lies at the heart of scientific inquiry. In machine learning, this fundamental principle faces unprecedented challenges as models grow in complexity and training procedures become increasingly sophisticated Henderson et al. 2019 ; Impagliazzo et al. 2023 ; Beam et al. 2020 ; Kapoor & Narayanan 2023 . The recent surge in transfer learning applications—where pre-trained models are adapted to new domains—has amplified these concerns, as practitioners routinely employ adaptive data selection strategies that dynamically shape the training process based on model state and data characteristics Jiang et al. 2024 . Recent surveys indicate that over 70% of ML researchers struggle to replicate published results, highlighting the severity of this reproducibility crisis Desai et al. 2025 ; Hutson 2018 . Transfer learning has emerged as the dominant paradigm for deploying machine learning systems across diverse applications, from natural language processing to computer vision Zhu et al. 2020 ; Chen et al. 2020 . By leveraging knowledge from large-scale pre-training, these approaches achieve remarkable performance even with limited domain-specific data. However, this efficiency comes with hidden costs: the adaptive mechanisms that make transfer learning effective also introduce substantial variability in outcomes. Recent studies have documented cases where identical experimental setups yield significantly different results across runs, raising critical questions about the reliability of reported findings Bouthillier et al. 2019 ; Kou 2024 . At the core of this challenge lies a fundamental tension between adaptation and stability. Adaptive data selection strategies—including importance weighting, confidence-based sampling, curriculum learning, and gradient-based selection—improve learning efficiency by focusing computational resources on the most informative examples Ren et al. 2019 . Recent advances in adaptive learning platforms demonstrate the practical benefits of these approaches Strielkowski et al. 2025 . Yet this very adaptivity creates complex dependencies between the selection mechanism and the evolving model state, potentially amplifying small variations in initialization or data ordering into substantial differences in final performance. Understanding and quantifying this trade-off has become essential as machine learning systems are deployed in high-stakes applications where consistency and reliability are paramount. The theoretical foundations for understanding replicability in machine learning have advanced significantly through connections to algorithmic stability, differential privacy, and statistical learning theory Bun et al. 2023 ; Chase et al. 2023 . These frameworks establish that replicability—defined as the property that an algorithm produces consistent outputs when applied to independent samples from the same distribution—is intimately connected to how algorithms respond to perturbations in their inputs. Recent work has begun exploring replicability in specific contexts, such as online learning Ahmadi et al. 2024 and reinforcement learning Eaton et al. 2023 . However, existing theory has not adequately addressed the dynamic, adaptive nature of modern transfer learning pipelines. Our work bridges this gap by developing a comprehensive framework for analyzing replicability in transfer learning with adaptive data selection. We introduce the concept of selection sensitivity ( \Delta_{Q} ), which quantifies how much a selection strategy’s distribution changes in response to small perturbations in the training data. This measure provides a principled way to compare different adaptive strategies and predict their impact on replicability. Our theoretical analysis reveals that the probability of replicability failure scales quadratically with selection sensitivity, providing tight bounds that enable practitioners to make informed decisions about strategy selection based on their tolerance for variability. Here, replicability failure refers to the event that two models trained independently on different samples from the same distribution produce performance differences exceeding a small threshold—a notion distinct from the algorithmic replicability , which typically requires syntactic equivalence of outputs rather than statistical consistency of performance. Through extensive empirical evaluation on six distinct selection strategies—ranging from simple uniform sampling to sophisticated approaches—we validate our theoretical predictions and uncover practical insights for improving replicability without sacrificing performance. Our experiments on the MultiNLI corpus demonstrate that while highly adaptive strategies can improve accuracy, they also increase replicability failure rates from 2.2% (uniform baseline) to over 80% for gradient based methods in the two-stage fine-tuning setting. Most significantly, we show that incorporating source domain pretraining before adaptive fine-tuning provides a robust mitigation strategy, dramatically improving replicability while maintaining the performance benefits of adaptive selection. This paper addresses the following research questions: RQ1: How can we formally characterize and quantify the impact of adaptive data selection strategies on the replicability of transfer learning outcomes? RQ2: What is the precise mathematical relationship between selection sensitivity, sample size, and the probability of replicability failure in transfer learning? RQ3: How do different families of adaptive selection strategies—from simple importance weighting to complex gradient-based methods—compare in terms of their selection sensitivity and resulting impact on replicability? RQ4: What practical mitigation strategies can reduce replicability failures while preserving the performance benefits of adaptive selection in real-world transfer learning scenarios? Our contributions advance both theoretical understanding and practical application of replicable machine learning: 1. We develop a rigorous mathematical framework that formalizes replicability in transfer learning with adaptive data selection, introducing selection sensitivity ( \Delta_{Q} ) as a key measure of algorithmic stability. 2. We prove tight theoretical bounds showing that replicability failure probability scales as \rho\leq 4\exp(-\epsilon^{2}n/2c^{2}\Delta_{Q}^{2}) , revealing the quadratic impact of selection sensitivity and exponential improvement with sample size. 3. We provide comprehensive empirical validation across six selection strategies on the MultiNLI corpus, demonstrating that our theoretical predictions accurately capture real-world behavior across a wide range of sensitivity values . 4. We identify source domain pretraining as a powerful mitigation strategy that reduces replicability failure rates by up to 30% while maintaining performance gains, offering a practical solution for reliability-critical applications. 5. We release a complete experimental framework 1 1 1 code-to-be-released-post-review that enables researchers

Read the complete text