Expectation Error Bounds for Transfer Learning in Linear Regression and Linear Neural Networks

Meitong Liu, Christopher Jung, Rui Li, Xue Feng, Han Zhao · 2026

Read Expectation Error Bounds for Transfer Learning in Linear Regression and Linear Neural Networks by Meitong Liu, Christopher Jung, Rui Li, Xue Feng, Han Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28739.txt.

Introduction

Liu and co-authors give new theory for when transfer learning helps in linear settings. They study ordinary least squares regression and under-parameterized linear neural networks. For regression, they derive exact formulas for expected generalization error with a bias-variance split. Those formulas give necessary and sufficient conditions for auxiliary tasks to help. They also find globally optimal task weights from solvable programs. For neural networks, they derive a non-asymptotic error bound. It gives the first non-vacuous sufficient condition for helpful auxiliary learning in that setting.

Catalog record

CREATORMeitong Liu, Christopher Jung, Rui Li, Xue Feng, Han Zhao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73903
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.28739

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Expectation Error Bounds for Transfer Learning in Linear Regression and Linear Neural Networks Meitong Liu, Christopher Jung, Rui Li, Xue Feng, Han Zhao Abstract In transfer learning, the learner leverages auxiliary data to improve generalization on a main task. However, the precise theoretical understanding of when and how auxiliary data help remains incomplete. We provide new insights on this issue in two canonical linear settings: ordinary least squares regression and under-parameterized linear neural networks. For linear regression, we derive exact closed-form expressions for the expected generalization error with bias-variance decomposition, yielding necessary and sufficient conditions for auxiliary tasks to improve generalization on the main task. We also derive globally optimal task weights as outputs of solvable optimization programs, with consistency guarantees for empirical estimates. For linear neural networks with shared representations of width q ≤ K q\leq K , where K K is the number of auxiliary tasks, we derive a non-asymptotic expectation bound on the generalization error, yielding the first non-vacuous sufficient condition for beneficial auxiliary learning in this setting, as well as principled directions for task weight curation. We achieve this by proving a new column-wise low-rank perturbation bound for random matrices, which improves upon existing bounds by preserving fine-grained column structures. Our results are verified on synthetic data simulated with controlled parameters. \coltauthor University of Illinois Urbana-Champaign and Meta and Meta and Meta and University of Illinois Urbana-Champaign keywords Transfer learning, Multi-task learning, Auxiliary data, Linear regression, Linear neural networks, Low-rank approximation, Bias-variance trade-off 1 Introduction Transfer learning, which aims to improve performance in a target domain by transferring knowledge from a source domain, has been a critical topic in machine learning ( 21 ; 15 ; 31 ) . In a similar spirit, multi-task learning (MTL) aims to boost generalization across multiple tasks by pooling shared information through joint training ( 4 ; 22 ; 30 ) . While MTL is usually evaluated by an aggregation of task objectives, such as the weighted average, it is essential to examine whether each task actually benefits from the information transfer. Specifically, viewing a task of interest as the main task and others as the auxiliary, is the jointly trained model better on the main task than a model trained without auxiliary data? In this work, we present new theoretical results on this question. Formally, given a training set D=(X,Y_{m},\{Y_{k}\}_{k=1}^{K}) , where X is the shared feature, Y_{m},Y_{1\sim K} are the main and auxiliary labels, an estimator \hat{w}_{D} is learned with auxiliary tasks weighted by \{\lambda_{k}\}_{k=1}^{K} . Our goal is to characterize the expected generalization error of \hat{w}_{D} on the main task: \displaystyle E_{K}:=\mathbb{E}_{D,(x,y_{m})}\left[(x^{\top}\hat{w}_{D}-y_{m})^{2}\right], where (x,y_{m}) is a test point drawn from the main task distribution. This enables us to investigate the following questions: (1) What are the factors that influence E_{K} , and how? (2) When do auxiliary data help, i.e, E_{K}<E_{0} , where E_{0} is the error when no auxiliary data are involved? and (3) How to select the task weights \{\lambda_{k}\}_{k=1}^{K} to minimize E_{K} ? We focus on two canonical linear frameworks for learning \hat{w}_{D} : (1) ordinary least squares regression and (2) linear neural networks with a shared representation layer of width q and task-specific output heads. Even in these simple setups, existing results for the above questions remain incomplete. Prior work on linear regression often limits its analyses to K=1 auxiliary source, nor have they developed practical methods for selecting optimal task weights ( 9 ; 8 ) . Linear neural networks with width q<K+1 pose fundamental challenges due to a random low-rank truncation in the learned estimator, leaving non-asymptotic modeling of E_{K} an open problem ( 28 ; 29 ) . In this work, we fill in these gaps with exact or upper bounds for E_{K} , featuring the first theoretical analysis for optimal weight selection in linear regression and the first non-vacuous sufficient condition for beneficial auxiliary learning in under-parameterized linear networks, along with other insights. Technically, we also contribute to the study of random matrix theory with a new column-wise low-rank perturbation bound. 1.1 Our Contributions Linear regression Our technical results can be summarized as follows: • We derive exact expressions for E_{K} with bias-variance decomposition ( \cref th:3.1), formalizing the trade-off mechanism that auxiliary data increases bias but potentially decreases variance. • We obtain necessary and sufficient conditions for beneficial auxiliary learning as an upper bound on the total weight of auxiliary tasks ( \cref co:3.2), which translates to a call for higher task similarity, smaller composite auxiliary noise, and moderate sample size for positive transfer. • We solve for the optimal task weights that minimize E_{K} via a convex quadratic program over the simplex and a closed-form formula ( \cref th:3.2). Based on empirical estimates for the true solutions, we develop a practical method for weight tuning with consistency guarantees ( \cref th:3.3), offering the first weight optimization strategy for transfer learning in linear regression. Linear neural networks We focus on the under-parameterized case where the network width q<K+1 , as otherwise the model capacity is too large for any information transfer ( 28 ) . In this case, the central difficulty arises from the highly nonlinear low-rank approximation of a random matrix in the learned estimator, rendering an exact characterization of E_{K} intractable. We instead pursue an upper bound U(E_{K}) with the critical requirement that it can be less than E_{0} , so that we obtain useful information by setting U(E_{K})<E_{0} . By proving a new column-wise low-rank perturbation bound and careful concentration of random matrices, we derive such a U(E_{K}) : Theorem 1 (informal; see \cref th:4.1) . Let N be the number of samples and d be the feature dimension with N>d+3 . Define the weight matrix \Lambda=\operatorname{diag}(\{\sqrt{\lambda_{k}}\}_{k=1}^{K}\cup\{1\}) and the task matrix W^{*}=[w^{*}_{1},\ldots,w^{*}_{K},w^{*}_{m}] , where w^{*}_{m} (resp. w^{*}_{k} ) is the true model for the main task (resp. the k -th auxiliary task). Given that a signal-to-noise ratio r tunable by task weights satisfy r>1 , we have \displaystyle E_{K}\leq\mathcal{O}(1)\sigma_{q+1}^{2}(W^{*}\Lambda)+\mathcal{O}(1)\left(\frac{\|w^{*}_{m}\|}{r-1}\right)^{2}+\mathcal{O}\left(\frac{1}{N}\right)\left(\frac{d}{(r-1)^{2}}+q\right)=:U(E_{K}), where \mathcal{O}(\cdot) reflects the orders of N and d , and \sigma_{q+1}(\cdot) denotes the (q+1) -th largest singular value. Table 1 : Comparison of low-rank perturbation bounds. Here, \delta_{q}(S)=\sigma_{q}(S)-\sigma_{q+1}(S) , r=\delta_{q}(S)/\|Z\| , and P_{U}=U_{:,q}U_{:,q}^{\top} , where U_{:,q} contains the top q left singular vectors of S . Source Object Assumption Bound Possible to be smaller than \|Z\| or \|Ze_{j}\| ? EYM ( 23 ) \|[M]_{q}-[S]_{q}\| - 2(\sigma_{q+1}(S)+\|Z\|) No TVV ( 23 ) \|[M]_{q}-[S]_{q}\| r\geq 4 \frac{6r}{r-2}\|Z\|(\frac{\sigma_{q}(S)}{\delta_{q}(S)}+\log\frac{6\sigma_{1}(S)}{\delta_{q}(S)}) No Ours ( \cref th:4.2) \|([M]_{q}-[S]_{q})e_{j}\| r>1 \frac{1}{r-1}(\|Se_{j}\|+\|Ze_{j}\|)+\|P_{U}Ze_{j}\| Yes Our bound reveals two driving factors: low-rank truncation loss as \sigma_{q+1}(W^{*}\Lambda) and a tunable signal-to-noise ratio r . Crucially, by curating task weights to reduce \sigma_{q+1}(W^{*}\Lambda) and increase r , we can have E_{K}<U(E_{K})<E_{0} . This establishes the first non-vacuous sufficient condition for beneficial auxiliary learning in this setting. By the interaction between \Lambda and other parameters in r and \sigma_{q+1}(W^{*}\Lambda) , the bound also provides principled suggestions on weight selection. The proof of U(E_{K}) relies on our new column-wise low-rank perturbation bound ( \cref th:4.2). Given a signal matrix S and a subGaussian perturbation Z , with M=S+Z , we aim to bound \|([M]_{q}-[S]_{q})e_{j}\| , where e_{j} is the j -th standard basis. In order to have U(E_{K})<E_{0} , the

Read the complete text