Improving variable selection properties with data integration and transfer learning
Paul Rognon-Vael, BIDSA, Bocconi University · 2025
Read Improving variable selection properties with data integration and transfer learning by Paul Rognon-Vael, BIDSA, Bocconi University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15584.txt.
Introduction
Paul Rognon-Vael and co-authors show how outside information can rescue variable selection in high-dimensional regression. Their 2025 paper studies the task of naming which of many variables truly drive an outcome, when data are sparse. Recovery is provably possible only under tight conditions on sample size, signal, and sparsity, which real studies often miss.
Their fix is a family of penalties shaped by prior hints about which variables are likely real. One use is transfer learning: variables picked in earlier data sets guide the hunt in a new one. The authors prove their methods succeed in regimes where hint-free methods must fail, and succeed faster where both can work.
Catalog record
| CREATOR | Paul Rognon-Vael, BIDSA, Bocconi University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 156 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-15584
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving variable selection properties with data integration and transfer learning Paul Rognon-Vael, BIDSA, Bocconi University Abstract We study variable selection (also called support recovery) in high-dimensional sparse linear regression when one has external information on which variables are likely to be associated with the response. Consistent recovery is only possible under somewhat restrictive conditions on sample size, dimension, signal strength, and sparsity. We investigate how these conditions can be relaxed by incorporating said external information. A key application that we consider is structural transfer learning, where variables selected in one or more source datasets are used to guide variable selection in a target dataset. We introduce a family of likelihood penalties that depend on the external information, motivated by connections to Bayesian variable selection. We show that these methods achieve variable selection consistency in regimes where any method ignoring external information fails, and that they achieve consistency at faster rates. We first quantify the potential gains under ideal, oracle-chosen, penalties. We then propose computationally efficient empirical Bayes procedures that learn suitable penalties from the data. We prove that these procedures have improved variable selection properties compared to methods that do not use external information. We illustrate our approach using simulations and a genomics application, where results from mouse experiments are used to inform variable selection for gene expression data in humans. Keywords: high-dimensional statistics, Bayesian variable selection, \ell_{0} penalty, empirical Bayes, transfer learning. 1 Introduction Many contemporary statistical problems involve high-dimensional models, where the goal is to identify a small subset of variables associated with a response. In such settings, inference is often severely limited by the dataset sample size. At the same time, basing inference on a single dataset increasingly feels like a wasted opportunity: it is common to have additional external information—such as prior studies, auxiliary datasets, or variable-specific annotations—that can suggest differential relevance across variables. This situation arises, for example, in multimodal data ( Liang et al. 2023 ) where different groups of variables may exhibit different levels of sparsity, such as clinical versus genomic markers in biomedical applications. It also arises when metadata are available on the variables themselves, for instance through functional annotations of genes. A setting of particular interest is transfer learning, which broadly refers to improving performance on a target dataset by leveraging information from related source datasets. In the context of variable selection, transfer learning uses information about variables identified in source datasets as external input to guide variable selection in a target regression problem. We refer to this problem as structural transfer learning , to distinguish it from approaches that focus on transferring parameter estimates or predictions. In high-dimensional settings, leveraging external information is particularly attractive. From a theoretical perspective, the limited information provided by a single dataset implies that strong assumptions are required for accurate inference. In structural learning problems such as variable selection, those assumptions regard sparsity and signal strength and, although mathematically necessary, they can be too strong in practice (see Giannone et al. (2021) for a discussion in the context of social sciences). From a practical perspective, numerous works observed improved inference when employing external information to guide structural learning ( Stingo et al. 2011 , Boulesteix et al. 2017 , Chen et al. 2021 ) . Despite this empirical success, a theoretical framework explaining precisely why and how leveraging external information may deliver improved structural learning is not currently available. Our main contribution is investigating how, in linear regression, integrating external information (and structural transfer learning as a particular case) would allow an oracle to push the mathematical conditions under which consistent variable selection is possible, and improving the corresponding rates. Consistency is understood as recovering the truly active variables with probability going to 1 as the sample size n , and possibly the dimension p , go to infinity. Our other main contribution is proposing an empirical Bayes data-based procedure that doesn’t require oracle information, and showing that it also enjoys improved properties. To make ideas concrete, consider the Gaussian linear regression \mathbf{y}\;=\;\mathbf{X}\boldsymbol{\beta}^{*}+\mathbf{\epsilon},\qquad\mathbf{\epsilon}\sim N(0,\sigma^{2}I_{n}), (1) where \mathbf{X}\in\mathbb{R}^{n\times p} , \mathbf{y}\in\mathbb{R}^{n} , \sigma>0 , and \boldsymbol{\beta}^{*}\in\mathbb{R}^{p} are the data-generating parameters. Our results can be easily extended to non-linear regression, where \mathbf{X} is given by a suitable basis such as splines. Penalized likelihood methods for variable selection are often based on optimizing the log-likelihood plus a penalty term driven by the \ell_{q} "norm" of an estimated \hat{\boldsymbol{\beta}} for q\in[0,1] . Here we focus on \ell_{0} penalties because they possess theoretically superior properties for support recovery compared to other penalties ( Wainwright 2010 , Gao and Aragam 2025 ) . That is, our goal is to study how external information can improve upon these best known properties. Besides, computational advances have made \ell_{0} problems much more tractable: optimization methods can solve them exactly for p in the thousands ( Bertsimas and Van Parys 2020 ) and Markov Chain Monte Carlo (MCMC) methods have linear cost in p under sparsity conditions ( Yang et al. 2016 , Zhou et al. 2022 ) . There is also a direct connection between \ell_{0} penalties and Bayesian variable selection ( Schwarz 1978 , Chen and Chen 2008 , Rossell 2022 ) . Under mild regularity conditions, a choice of \ell_{0} penalty corresponds to a choice of prior probability for variable inclusion. This connection converts \ell_{0} penalization into a natural and intuitive way to encode external information on the relevance of variables for association with the outcome. We consider a variable selection framework where \ell_{0} penalties may depend on external information \boldsymbol{z}=(z_{1},\ldots,z_{p})\in\mathbb{R}^{p} . In transfer learning applications, each z_{j} may be a q -dimensional vector measuring the importance of variable j in q source datasets. For theoretical tractability and to be able to provide sharp conditions and rates, here we focus on the case where \boldsymbol{z} partitions the p variables into b blocks, that is z_{j}\in\{1,\ldots,b\} . We denote by B_{j}\subset\{1,\ldots,p\} the set of variables in block j=1,\ldots,b . When \boldsymbol{z} is informative for variable selection, key characteristics such as the proportion of truly active variables or the signal strength may vary across blocks, hence one allows the \ell_{0} penalty to differ across blocks. We derive the variable selection properties of such informed \ell_{0} penalties, first in general and then for structural transfer learning, and compare them to standard \ell_{0} penalties. Before proceeding, we review selected literature. External information has been used to guide variable selection in many applications. For instance, Stingo et al. (2011) and Cassese et al. (2014) proposed Bayesian variable selection for gene expression where prior inclusion probabilities depend on biological knowledge and meta-variables. Chen et al. (2021) predicted disease outcomes with \ell_{1} penalties that depend on gene functional annotations. Several works discussed methodology, but not theory, for external-information dependent penalization or prior inclusion probabilities. In the Bayesian literature, van de Wiel et al. (2019) suggested an empirical Bayes approach to regress prior inclusion probabilities on external data. Velten and Huber (2021) proposed a variational Bayes approximation for regression with external information-dependent prior inclusion probabilities. Recently, Busatto and van de Wiel (2023) considered a horseshoe prior that depends on external information. In the frequentist literature, in the context of multi-omics data, Boulesteix et al. (2017) let the \ell_{1} penalty depend