Source-Optimal Training is Transfer-Suboptimal
2025
Read Source-Optimal Training is Transfer-Suboptimal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08401.txt.
Introduction
C Evans Hedges proves that training a model perfectly for its source task is generally the wrong move when the real goal is transfer to a new task. Transfer learning reuses a source model, and practitioners naturally tune it for source performance first. Working with L2-SP ridge regression, Hedges shows the source optimal regularization and the transfer optimal regularization differ outside a vanishingly special set of cases.
The transfer optimal penalty depends on task alignment. With imperfect alignment, stronger source regularization helps transfer, while in super aligned regimes weaker regularization wins instead. In isotropic settings, whether transfer helps at all depends only on alignment and source traits, not target sample size or noise. Experiments from synthetic data to image and text benchmarks support the mismatch.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08401
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Source-Optimal Training is Transfer-Suboptimal Abstract . We prove a fundamental misalignment in transfer learning: the source regularization that minimizes source risk almost never coincides with the regularization maximizing transfer benefit. Through sharp phase boundaries for L2-SP ridge regression, we characterize the transfer-optimal source penalty τ 0 ∗ \tau_{0}^{*} for a fixed task alignment. With sufficiently weak alignment, τ 0 ∗ \tau_{0}^{*} is always larger than the source optimal regularization, however with strong alignment τ 0 ∗ \tau_{0}^{*} diverges predictably from task-optimal values: requiring stronger regularization in high-SNR regimes and weaker regularization in low-SNR regimes. Additionally, in isotropic settings the decision to transfer is remarkably independent of target sample size and noise, depending only on task alignment and source characteristics. CIFAR-10 and MNIST experiments confirm this counterintuitive pattern persists in non-linear networks. C. Evans Hedges Keywords: Transfer learning, ridge regression, overparameterization, L2-SP, phase boundaries, random matrix theory, pretraining, continual learning, deterministic equivalents. 1. Introduction 1.1. Background and Motivation A fundamental question in transfer learning is: how should we train the source model to maximize downstream performance? Conventional wisdom suggests optimizing the source task leads to better transfer. We prove the opposite: the source regularization that minimizes source risk almost never coincides with the regularization that maximizes transfer benefit. This misalignment occurs in a predictable way depending on the source noise and task alignment. In the setting of high task alignment, optimal source training for transfer counterintuitively requires more regularization when the source has high SNR and less regularization in the high-noise regime. We begin our exploration through the lens of evaluating when we would expect positive or negative transfer from some source task to a target task. Previous work [ 5 , 13 , 16 ] has shown that the phenomenon of negative transfer is a concern, where the model generated by transfer learning performs worse than if one were to train the model from scratch. Intuition and previous theoretical work [ 13 ] suggest that if the tasks and input datasets are similar enough, transfer learning techniques should help, but if the target task is sufficiently different, or in the context of continual learning the regime has shifted enough, transfer learning may harm the final model, introducing bias and noise from the initialization. There has been notable work examining this phase boundary between when one should expect positive or negative transfer. In [ 5 , 6 ] , Dar and Baraniuk provided an explicit quantitative phase boundary for freezing/parameter-sharing style transfer in linear models, giving if-and-only-if conditions for when shared representations outperform training from scratch. In [ 7 ] , Dhifallah and Lu explored the phenomenon in the context of feature freezing. Additionally, feature-overlap driven transitions in linear transfer were explored by Tahir, Ganguli, and Rotskoff in [ 14 ] . Related phase transitions have been characterized for other linear transfer mechanisms such as hard parameter sharing [ 16 ] . Complementary asymptotic analyses for fine-tuning from pretrained anchors via gradient descent appear in Ghane, Akhtiamov, and Hassibi [ 9 ] , who provide universal asymptotics comparing pretrained versus fine-tuned models under distributional shifts. The L2-SP (L2-distance to Starting Point) approach was introduced by Li, Grandvalet, and Davoine [ 15 ] , who showed empirical improvements by penalizing deviation from pre-trained parameters. This approach matches many practical fine-tuning protocols and corresponds to the isotropic elastic weight consolidation (EWC) penalty [ 11 ] . Dar, LeJeune, and Baraniuk [ 6 ] recently analyzed when L2-SP transfer learning outperforms standard ridge regression, assuming task parameters are related by orthonormal transformations and focusing on optimal tuning of the target-task regularization given a fixed source model. Our work extends this analysis in several critical directions. First, we consider arbitrary task vectors without orthonormal structure, covering realistic scenarios where tasks differ in both magnitude and direction. Second, we provide sharp if-and-only-if boundaries characterizing exactly when transfer outperforms training from scratch, in both finite-sample and deterministic-equivalent settings. Third, and most significantly, we address the source-side optimization problem: how should one train the source model to maximize downstream transfer benefit? Given the alignment of two tasks, we derive the transfer-optimal source regularization \tau_{0}^{*} and prove it generically differs from the source-task-optimal choice, revealing a fundamental misalignment between optimizing for source performance versus transfer capability. This source-side perspective appears to be entirely novel, as prior work focuses on target-task tuning given a pre-trained source model. 1.2. Setup and Overview of Results In this paper we formalize transfer learning from the lens of L2-SP ridge regression. In particular, we have two tasks, Task 0 (the source task) and Task 1 (the target task), as well as corresponding training datasets (X_{0},y_{0}) and (X_{1},y_{1}) . We then seek to evaluate the expected out-of-sample risk on Task 1, comparing the ridge/ridgeless solution trained solely on (X_{1},y_{1}) to the L2-SP ridge solution found by first training a ridge/ridgeless model on (X_{0},y_{0}) , and then using those model parameters as the prior for ridge/ridgeless training on (X_{1},y_{1}) . Our analysis builds on the foundational work characterizing ridge and ridgeless regression risk in high-dimensional settings [ 8 , 10 ] , which established the precise asymptotic behavior of these estimators in overparameterized regimes and revealed phenomena such as double descent [ 3 ] and benign overfitting [ 2 ] . We employ random matrix theory techniques, specifically the deterministic equivalent framework [ 1 , 4 , 8 , 10 ] , to derive precise asymptotic characterizations of the estimators’ risk and identify sharp phase boundaries for transfer benefit. Our contributions can be outlined as follows. First, Theorem 3.3 provides an if-and-only-if inequality describing when transfer learning will outperform from-scratch training in the finite data, non-isotropic ridge regime. As a corollary, we show that in the finite sample, isotropic, ridgeless case the inequality takes a simple form and is interestingly independent of n_{1} and \sigma_{1} . This independence is consistent with prior isotropic analyses of freezing-style transfer [ 5 , 13 ] , and our finite-sample isotropic/ridgeless corollary recovers these patterns in the L2-SP setting. We then examine deterministic equivalents in the asymptotic limit and Theorem 3.6 provides a DE characterization and asymptotic boundary for L2-SP ridge transfer with general (non-orthonormal) task vectors. Corollary 3.7 examines the isotropic case, where the decision criterion is determined entirely by whether or not the alignment of the two tasks surpasses a bias and noise term dependent only on Task 0. From this we arrive at Theorem 3.8 , which identifies the unique \tau_{0}^{*} (source model ridge penalty) that maximizes transfer benefit, given a particular task alignment. Outside of a measure-zero set, \tau_{0}^{*} does not coincide with the optimal Task 0 ridge parameter, and notably \tau_{0}^{*} is independent of target sample size but depends on task alignment. Together, these results give the first explicit analytical boundary for when Euclidean L2-SP transfer helps with general (non-orthonormal) task vectors in overparameterized ridge models, including general covariance and DE limits, and provide surprising insights into optimally training source models for the purpose of transfer learning. Finally, we establish in Corollary 3.9 that under sufficiently aligned tasks, in the high-SNR regime (low noise) \tau_{0}^{*} is greater than the source-optimal ridge penalty, and similarly in the low-SNR regime (high noise), \tau_{0}^{*} is less than the