H igh-Dimensional Importance-Weighted Information Criteria: Theory and Optimality
Yong-Syun Cao, Shinpei Imori, Ching-Kang Ing · 2025
Read H igh-Dimensional Importance-Weighted Information Criteria: Theory and Optimality by Yong-Syun Cao, Shinpei Imori, Ching-Kang Ing (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06531.txt.
Introduction
Cao, Imori, and Ing present a statistics paper, published in 2025, on how to choose the size of a prediction model. Their setting is hard in two ways. The data hold very many possible factors, and the cases the model must predict differ from the cases it learned from. An earlier method builds the model step by step, adding the most useful factor at each pass. A scoring rule tells it when to stop.
The earlier work claimed that this pairing strikes the best balance between missing real patterns and chasing noise. It also claimed the fastest possible closing of prediction error. The new paper proves those claims under a stated set of assumptions. Proofs of this kind matter because they tell users when the method can be trusted. They turn a rule that seemed to work into one with a firm basis.
Catalog record
| CREATOR | Yong-Syun Cao, Shinpei Imori, Ching-Kang Ing |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06531
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
H igh-Dimensional Importance-Weighted Information Criteria: Theory and Optimality Yong-Syun Cao, Shinpei Imori, Ching-Kang Ing Abstract Imori and Ing (2025) proposed the importance-weighted orthogonal greedy algorithm (IWOGA) for model selection in high-dimensional misspecified regression models under covariate shift. To determine the number of IWOGA iterations, they introduced the high-dimensional importance-weighted information criterion (HDIWIC). They argued that the combined use of IWOGA and HDIWIC, IWOGA + HDIWIC, achieves an optimal trade-off between variance and squared bias, leading to optimal convergence rates in terms of conditional mean squared prediction error. In this article, we provide a theoretical justification for this claim by establishing the optimality of IWOGA + HDIWIC under a set of reasonable assumptions. 1 Introduction Various methods for high-dimensional model selection have been developed in recent years to address situations where the training and test data come from different distributions. When both input and output variables are available in the source (training) and target (test) domains but the target sample size is small, estimates based solely on the target data often suffer from high variance. To improve accuracy, auxiliary estimates from the source domain can be incorporated, along with bias correction to account for domain differences. This transfer learning strategy facilitates more reliable estimation under limited target information (see, for example, Li et al. (2021) , Bastani (2021) , and Tian and Feng (2022) ). However, when test outputs (i.e., target responses) are unavailable, estimation or bias correction involving both domains becomes infeasible, as only inputs (covariates) are observed in the test set. In such cases, importance weighting is commonly used to reweight the source data, allowing for effective parameter estimation and model selection tailored to the target domain. This study focuses on the latter setting. While allowing input distributions to differ between domains, we assume the conditional distribution of the response given covariates remains the same–an assumption known as covariate shift (see Shimodaira (2000) and Sugiyama and Kawanabe (2012) ). In a recent paper, Imori and Ing (2025) proposed reweighting the training samples using trimmed density ratios between the test and training covariates. The resulting weights were incorporated into a modified version of the Orthogonal Greedy Algorithm (OGA), termed the Importance-Weighted OGA (IWOGA), for high-dimensional regression model selection under covariate shift. They showed that IWOGA achieves a fast convergence rate in test prediction error as the number of iterations increases, even under model misspecification. However, the optimal number of iterations–necessary for achieving the optimal convergence rate–depends on the sparsity level of the regression coefficients, which is typically unknown in practice. To address this, Imori and Ing (2025) proposed a data-driven method, the high-dimensional importance-weighted information criterion (HDIWIC), to determine the appropriate number of IWOGA iterations. This paper aims to show that the combination of IWOGA and HDIWIC, referred to as IWOGA+HDIWIC, achieves the optimal convergence rate in high-dimensional regression under covariate shift and model misspecification, without requiring prior knowledge of the sparsity level. The remainder of the paper is organized as follows. Section 2.1 reviews the preliminaries of Imori and Ing (2025) , including notation and assumptions. Section 2.2 presents the main result, showing that IWOGA+HDIWIC effectively achieves optimal performance. In Section 3 , we return to the conventional OGA and demonstrate that when the full high-dimensional regression model is correctly specified, the standard HDIC, introduced in Ing (2020) , can be used to determine the optimal number of iterations under covariate shift, particularly when the difference between the training and test input distributions is moderate, as formalized in condition ( 3.6 ). All technical details are provided in Section 4 . 2 Convergence rate of IWOGA + HDIWIC 2.1 Framework of IWOGA under Covariate Shift: Notation and Assumptions Let y\in\mathbb{R} denote the response variable and \bm{x}=(x_{1},\dots,x_{p_{n}})^{\top} denote the covariates, on which we add superscript ” tr ” or ” te ” if they are from training or testing distribution. Consider a linear regression model for the test data, \displaystyle y^{te}=\alpha+\bm{\beta}^{\top}\bm{x}^{te}+\varepsilon^{te}=y^{te}(\bm{x}^{te})+\varepsilon^{te}, (2.1) where \alpha and \bm{\beta} are the coefficients of the best linear predictor under the full model J_{F}=\{1,\ldots,p_{n}\} . For a subset J\subset J_{F} with J\neq\emptyset , \alpha(J) and \bm{\beta}(J) denote the corresponding coefficients of the best linear predictor of y^{te} restricted to model J . Throughout this paper, we assume model J(\subset J_{F}) is not empty. We allow model misspecification in the sense that E[y^{te}|\bm{x}^{te}]\neq y^{te}(\bm{x}^{te}) . Let f^{tr}(\bm{x}) and f^{te}(\bm{x}) denote the training and test input densities, and f^{tr}(y|\bm{x}) and f^{te}(y|\bm{x}) the conditional densities of y given \bm{x} in the training and test data. Although f^{tr}(\bm{x}) and f^{te}(\bm{x}) may not coincide, the covariate shift assumption requires that f^{tr}(y|\bm{x})=f^{te}(y|\bm{x}) , yielding E[y^{tr}|\bm{x}^{tr}=\bm{x}]=E[y^{te}|\bm{x}^{te}=\bm{x}] , which we denote by y(\bm{x}) in what follows. Note that when J_{F} is correctly specified, \displaystyle y(\bm{x})=\alpha+\bm{\beta}^{\top}\bm{x}. (2.2) Moreover, we assume in the rest of the paper that f^{tr}(\bm{x}) and f^{te}(\bm{x}) have a common support, and 0<w(\bm{x})=f^{te}(\bm{x})/f^{tr}(\bm{x})<\infty for any \bm{x} in the support. Let (y^{tr}_{t},\bm{x}^{tr}_{t}) , t=1,\ldots,n , denote the observed training data. Define the importance by w(\bm{x}) . Rather than directly using w(\bm{x}^{tr}_{t}) to reweight the t th training sample, Imori and Ing (2025) considered a truncated version: \displaystyle v(\bm{x}^{tr}_{t})=\min\{w(\bm{x}^{tr}_{t}),b_{n}\}, (2.3) where b_{n} is a trimming parameter introduced to stabilize the potentially large values of w(\bm{x}^{tr}_{t}) . The final weight applied in IWOGA is then defined as \displaystyle w_{t}=\frac{v(\bm{x}^{tr}_{t})}{n^{-1}\sum\limits_{t=1}^{n}v(\bm{x}^{tr}_{t})}. (2.4) For further details on IWOGA, refer to Section 2 of Imori and Ing (2025) . Let \hat{J}^{te}_{m} denote the model selected by IWOGA at the m th iteration. For notational simplicity, we will omit the “ te ” superscript in \hat{J}^{te}_{m} throughout the paper. Using \hat{J}_{m} , y^{te}(\bm{x}^{te}) in ( 2.1 ) can be predicted by \hat{y}^{te}(\bm{x}^{te}|\hat{J}_{m}) , where \displaystyle\hat{y}^{te}(\bm{x}|J)=\hat{\alpha}(J)+\hat{\bm{\beta}}(J)^{\top}\bm{x}_{J} (2.5) is the weighted least squares predictor based on model J . Note that for \bm{v}\in\mathbb{R}^{p_{n}} , \bm{v}_{J}\in\mathbb{R}^{\sharp{J}} denotes its subvector corresponding to the index set J , where \sharp{J} denotes the cardinality of J , and \hat{\alpha}(J)\in\mathbb{R} and \hat{\bm{\beta}}(J)\in\mathbb{R}^{\sharp{J}} are the weighted least squares estimates of \alpha(J) and \bm{\beta}(J) ; see ( 2.18 ) for details. The performance of \hat{y}^{te}(\bm{x}^{te}|\hat{J}_{m}) is measured by the modified conditional prediction error (MCPE), \displaystyle\begin{split}&E_{n}[(y^{te}(\bm{x}^{te})-\hat{y}^{te}(\bm{x}^{te}|\hat{J}_{m}))^{2}]\\ =&E[(y^{te}(\bm{x}^{te})-\hat{y}^{te}(\bm{x}^{te}|\hat{J}_{m}))^{2}|(y^{tr}_{i},\bm{x}^{tr}_{i}),i=1\ldots,n].\end{split} (2.6) To investigate the convergence rate of the MCPE as m increases, Imori and Ing (2025) imposed the following conditions. C 1 . There exists a constant \gamma\in(0,1) such that \log p_{n}=O(n^{\gamma}) , and p_{n}\rightarrow\infty as n\rightarrow\infty . C 2 . There exist constants \theta_{0}>0 and \eta\in(0,2] such that \displaystyle\underset{n\rightarrow\infty}{\text{lim sup }}E[\exp(\theta_{0}|\varepsilon^{tr}|^{\eta})]<\infty,\penalty\ \penalty\ \underset{n\rightarrow\infty}{\text{lim sup }}E[\exp(\theta_{0}|\varepsilon^{te}|^{\eta})]<\infty, \displaystyle\underset{n\rightarrow\infty}{\text{lim sup }}\max\limits_{0\leq i,j\leq p_{n}}E[\exp(\theta_{0}|x_{i}^{te}x_{j}^{te}|)]<\infty, where \varepsilon^{tr}=y^{tr}-y(\bm{x}^{tr}) and x_{0}^{te}=1 . C 3 . There exists q>0 such that \displaystyle\underset{n\rightarrow\infty}{\text{lim sup }}E[w(\bm{x}^{te})^{q}]<\infty. C 4 . \{(\varepsilon^{tr}_{t},\bm{x}^{tr}_{t})\}_{t=1}^{n} are independent and identically distributed, where \varepsilon^{tr}_{t}=y^{tr}_{t}-y(\bm{x}^{tr}_{t}) . C 5 . \sup\limits_{n\geq 1}(|\alpha|+\|\bm{\beta}\|_{2})<\infty and there exist constants \xi\geq 0 and C_{\xi}>0 such that for all J\subset J_{F} , \displaystyle\|\bm{\beta}_{J}\|_{1}\leq C_{\xi}\|\bm{\beta}_{J}\|_{2}^{\frac{2\xi}{2\xi+1}}. In addition, Imori and Ing (2025) assumed that there exist constants C^{te}_{\bm{\Sigma}}>0 and C^{te}_{\lambda+}>0 such that \displaystyle\max\limits_{\begin{subarray}{c}1\leq\sharp{J}\leq K_{n}\\ j\notin J\end{subarray}}\|(\bm{\Sigma}^{te}_{J,J})^{-1}\bm{\sigma}^{te}_{J,j}\|_{1}\leq C^{te}_{\bm{\Sigma}}, (2.7) \displaystyle\lambda_{\min}(\bm{\Sigma}^{te})\geq C^{te}_{\lambda+}, (2.8) where \bm{\Sigma}^{te}\in\mathbb{R}^{p\times p} is the covariance matrix of \bm{x}^{te} , \bm{\Sigma}^{te}_{J,J} is the submatrix of \bm{\Sigma}^{te} corresponding to J , \bm{\sigma}^{te}_{J,j}=(\bm{\sigma}^{te}_{j})_{J} , and \bm{\sigma}^{te}_{j} is the j th column of \bm{\Sigma}^{te} . The counterparts of \bm{\Sigma}^{te} , \bm{\Sigma}^{te}_{J,J} , \bm{\sigma}^{te}_{j} , and \bm{\sigma}^{te}_{J,j} for the training data are denoted by \bm{\Sigma}^{tr} , \bm{\Sigma}^{tr}_{J,J} , \bm{\sigma}^{tr}_{j} ,