Fine-tuning Factor Augmented Neural Lasso for Heterogeneous Environments

Jinhang Chai Jianqing Fan Cheng Gao Qishuo Yin Department of Operations Research and Financial Engineering Princeton University · 2026

Read Fine-tuning Factor Augmented Neural Lasso for Heterogeneous Environments by Jinhang Chai Jianqing Fan Cheng Gao Qishuo Yin Department of Operations Research and Financial Engineering Princeton University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12288.txt.

Introduction

Jinhang Chai and three colleagues develop a fine-tuning method for adapting a pretrained model. It targets cases where target data are scarce and variables are many. Their SMART approach adds the source model's output as an extra feature. Only the leftover, target-specific part is learned from scratch. A low-rank factor structure handles correlated inputs. The paper proves optimal risk bounds for when fine-tuning beats learning from scratch. Experiments show near-oracle performance with very small target samples.

Catalog record

CREATORJinhang Chai Jianqing Fan Cheng Gao Qishuo Yin Department of Operations Research and Financial Engineering Princeton University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE154437
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12288

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fine-tuning Factor Augmented Neural Lasso for Heterogeneous Environments Jinhang Chai Jianqing Fan Cheng Gao Qishuo Yin Department of Operations Research and Financial Engineering Princeton University Abstract Fine-tuning is a widely used strategy for adapting pre-trained models to new tasks, yet its methodology and theoretical properties in high-dimensional nonparametric settings with variable selection have not yet been developed. This paper introduces the fine-tuning factor augmented neural Lasso (FAN-Lasso) , a transfer learning framework for high-dimensional nonparametric regression with variable selection that simultaneously handles covariate and posterior shifts. We use a low-rank factor structure to manage high-dimensional dependent covariates and propose a novel residual fine-tuning decomposition in which the target function is expressed as a transformation of a frozen source function and other variables to achieve transfer learning and nonparametric variable selection. This augmented feature from the source predictor allows for the transfer of knowledge to the target domain and reduces model complexity there. We derive minimax-optimal excess risk bounds for the fine-tuning FAN-Lasso, characterizing the precise conditions, in terms of relative sample sizes and function complexities, under which fine-tuning yields statistical acceleration over single-task learning. The proposed framework also provides a theoretical perspective on parameter-efficient fine-tuning methods. Extensive numerical experiments across diverse covariate- and posterior-shift scenarios demonstrate that the fine-tuning FAN-Lasso consistently outperforms standard baselines and achieves near-oracle performance even under severe target sample size constraints, empirically validating the derived rates. Keywords : Transfer Learning, Neural Networks, Factor Augmentation, FAST-NN, Factor-adjusted Nonparametric Lasso, Model Shifts, Covariate Shifts. 1 Introduction Transfer learning has revolutionized modern machine learning by enabling the transfer of learned representations from data-rich source domains to sample-limited target tasks. This paradigm is most prominently realized through fine-tuning, where pre-trained weights from large-scale models serve as a sophisticated starting point for specialized downstream applications. The effectiveness of this approach hinges on the hypothesis that pre-training captures intrinsic latent structures—such as shared features or factors—that provide a foundational representation of the data. Fine-tuning thus acts as a pragmatic adaptation layer, refining these learned representations to align with the specific nuances of the target domain. While the empirical success of fine-tuning is undeniable, providing a unified and sharp theoretical characterization of its performance remains a formidable task. Existing literature has established foundational guarantees for transfer learning, yet these often focus on linear regimes or assume a restricted class of distribution shifts. There remains a critical need for a framework that can quantify the statistical gains of fine-tuning in the presence of high-dimensional covariates and complex nonparametric structures—settings where classical parametric intuition often fails. Specifically, it remains unclear how to construct a unified estimator that simultaneously achieves minimax optimality under both covariate and posterior shifts while maintaining robustness to negative transfer. In this paper, we bridge these theoretical and practical gaps by proposing a fine-tuning factor-augmented neural Lasso (FAN-Lasso) framework. We posit that the shared knowledge between domains can be effectively encapsulated by a low-dimensional factor structure, which provides a stable backbone for knowledge transfer even under significant distribution shifts. Our approach provides an optimal mechanism to transfer pre-trained factor-augmented sparse throughput neural networks (FAST-NN, 13 )—a class of models that combine latent factor extraction with sparse nonparametric estimation—to new environments. This framework weaves together three powerful components to address the aforementioned challenges. First, the low-dimensional factor structure enables tractable estimation in high-dimensional settings by capturing the latent drivers shared across domains, acting as a bridge for representation transfer. Second, we employ deep ReLU networks to model the dense and sparse nonparametric complexities, leveraging their adaptive capacity to unknown composition structures to sidestep the curse of dimensionality. Third, by introducing flexible transfer functions, our method explicitly accounts for both covariate and posterior shifts. This mechanism allows the estimator to automatically calibrate the degree of source-target similarity, effectively “gating” the amount of source information to utilize. This synergy not only ensures robustness against negative transfer by pruning irrelevant source signals but also achieves minimax optimality under general nonparametric regimes. 1.1 Problem Formulation Consider two datasets: source data \{(\bm{x}_{i}^{P},y_{i}^{P})\}_{i=1}^{n^{P}}\overset{\text{i.i.d.}}{\sim}P and target data \{(\bm{x}_{j}^{Q},y_{j}^{Q})\}_{j=1}^{n^{Q}}\overset{\text{i.i.d.}}{\sim}Q , where (\bm{x},y)\in\mathbb{R}^{p}\times\mathbb{R} . The central question is: how and under what conditions does access to source data accelerate learning on the target distribution Q ? Factor Model for Covariates. To handle the high-dimensional regime where p\gtrsim n^{Q} , we assume the covariates admit a low-rank factor structure: \bm{x}^{P}=\bm{B}^{P}\bm{f}^{P}+\bm{u}^{P},\qquad\bm{x}^{Q}=\bm{B}^{Q}\bm{f}^{Q}+\bm{u}^{Q}, (1) where \bm{B}^{P},\bm{B}^{Q}\in\mathbb{R}^{p\times r} are unknown loading matrices with p\gg r , \bm{f}^{P},\bm{f}^{Q}\in\mathbb{R}^{r} are latent factors, and \bm{u}^{P},\bm{u}^{Q}\in\mathbb{R}^{p} are idiosyncratic components. No factor structure of covariates is allowed by taking r=0 or \bm{B}=0 . Regression Functions. For the source domain, we assume a sparse nonparametric model: y_{i}^{P}=g^{P}(\bm{x}_{i,\pazocal{J}^{P}}^{P})+\epsilon_{i}^{P},\quad i\in[n^{P}], (2) where g^{P} depends on an unknown sparse subset \pazocal{J}^{P}\subset[p] of coordinates, and \{\epsilon_{i}^{P}\} are i.i.d. noise. As shown in Section 3.2 , this form facilitates transfer from the source to the target domain. For the target domain, we assume a factor-augmented nonparametric model: y_{j}^{Q}=g^{Q}(\bm{f}_{j}^{Q},\bm{u}_{j,\pazocal{J}^{Q}}^{Q})+\epsilon_{j}^{Q},\quad j\in[n^{Q}], (3) where g^{Q}:\mathbb{R}^{r+|\pazocal{J}^{Q}|}\to\mathbb{R} depends on both the latent factors and a sparse subset \pazocal{J}^{Q}\subset[p] of idiosyncratic components. The inclusion of \bm{u}_{\pazocal{J}^{Q}}^{Q} captures settings where some covariates or idiosyncratic entries carry predictive information beyond what the factors encode. The versatility of the FAN model is discussed in Section 2.2 . Two Types of Distribution Shift. This framework accommodates two fundamental types of shift in transfer learning: 1. Covariate Shift: The marginal distribution of covariates differs across domains, manifested through \bm{B}^{P}\neq\bm{B}^{Q} . The methodological challenge is to estimate the target factors \bm{f}^{Q} accurately by leveraging source data when the loading matrices are similar but not identical. 2. Posterior Shift: The conditional distribution of responses differs, i.e., g^{P}\neq g^{Q} . The statistical challenge is to learn the target function g^{Q} efficiently by exploiting the pre-trained source function g^{P} . Residual Fine-Tuning. To formalize how source knowledge transfers to the target, we decompose the target regression function via a residual structure: g^{Q}(\bm{f}^{Q},\bm{u}_{\pazocal{J}^{Q}}^{Q})=h\left(\bm{f}^{Q},\bm{u}_{\pazocal{J}}^{Q},g^{P}(\bm{x}_{\pazocal{J}^{P}}^{Q})\right), (4) where h:\mathbb{R}^{r+|\pazocal{J}|+1}\to\mathbb{R} is a low-complexity residual fine-tuning function and \pazocal{J}^{Q}=\pazocal{J}\cup\pazocal{J}^{P} . This formulation embeds the source function g^{P} as an input to h , capturing how target predictions refine source predictions. Our model is more general than additive residual models of the form g^{Q}=g^{P}+h_{0} and permits nonlinear compositions, including regime-switching behaviors such as h=h_{0}\cdot\mathbf{1}\{g^{P}>0\} . Crucially, the source function g^{P} remains frozen during fine-tuning; only h is estimated from target data. When h belongs to a simpler function class than g^{Q} , this decomposition yields both computational savings and faster statistical convergence. This is particularly relevant when \bm{u}_{\pazocal{J}^{Q}}^{Q} is absent from ( 4 ), so that h can be learned using a low-dimensional neural network. This point is elaborated further in Section 2 . 1.2 Preview of Main Results Our theoretical analysis makes the intuition behind residual fine-tuning precise. We establish that the fine-tuning FAN-Lasso estimator achieves the minimax optimal excess risk \left[\frac{\log(n^{P}+n^{Q})}{n^{P}+n^{Q}}\right]^{\frac{2\gamma^{P}}{2\gamma^{P}+1}}+\left(\frac{\log n^{Q}}{n^{Q}}\right)^{\frac{2\gamma}{2\gamma+1}}+\frac{\log p}{n^{Q}}+\frac{1}{p}, where \gamma^{P} and \gamma quantify the complexity, such as the dimensionality-adjusted degree of smoothness, of the source function g^{P} and the residual function h , respectively. This rate admits a revealing decomposition: the first term reflects the joint contribution of source and target data to learning g^{P} —the complex backbone that both domains share—while the second term captures the irreducible cost of estimating the residual h from target data alone. When source data is plentiful

Read the complete text