Scalable Bayesian inference on high-dimensional multivariate linear regression
Xuan Cao, Kyoungjae Lee · 2025
Read Scalable Bayesian inference on high-dimensional multivariate linear regression by Xuan Cao, Kyoungjae Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16446.txt.
Introduction
Xuan Cao and Kyoungjae Lee give fast Bayesian tools for wide multi outcome regression in a 2025 arXiv paper. Such models guess many outcomes at once from many inputs, and must also learn how the errors link. Bayesian routes give honest doubt bands but often crawl, or cut corners that break the math.
The team gives two routes. The exact one pairs a spike and slab prior on the weights with a graph based prior on the error links, at a cost near the best known shortcut. The fast one ignores the links while fitting weights, then reads the error graph off the leftovers. They prove both routes pick the right terms in the long run. Tests on fake and real data show the pair works in practice.
Catalog record
| CREATOR | Xuan Cao, Kyoungjae Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16446
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scalable Bayesian inference on high-dimensional multivariate linear regression Xuan Cao, Kyoungjae Lee Abstract We consider jointly estimating the coefficient matrix and the error precision matrix in high-dimensional multivariate linear regression models. Bayesian methods in this context often face computational challenges, leading to previous approaches that either utilize a generalized likelihood without ensuring the positive definiteness of the precision matrix or rely on maximization algorithms targeting only the posterior mode, thus failing to address uncertainty. In this work, we propose two Bayesian methods: an exact method and an approximate two-step method. We first propose an exact method based on spike and slab priors for the coefficient matrix and DAG-Wishart prior for the error precision matrix, whose computational complexity is comparable to the state-of-the-art generalized likelihood-based Bayesian method. To further enhance scalability, a two-step approach is developed by ignoring the dependency structure among response variables. This method estimates the coefficient matrix first, followed by the calculation of the posterior of the error precision matrix based on the estimated errors. We validate the two-step method by demonstrating (i) selection consistency and posterior convergence rates for the coefficient matrix and (ii) selection consistency for the directed acyclic graph (DAG) of errors. We demonstrate the practical performance of proposed methods through synthetic and real data analysis. keywords [class=MSC] Primary 62F15 , 62H12 ; secondary 62F12 keywords Generalized likelihood , Selection consistency , Posterior convergence rate and 1 Introduction Multivariate linear regression models have attracted attention in the statistical literature over the past two decades because they provide an intuitive tool for capturing the relationships between a response vector and a covariate vector. Unlike in univariate linear regression models, learning the dependency structure among response variables becomes a critical task in multivariate linear regression models. In particular, when there is time ordering or known causal relationships among the response variables, it is crucial to incorporate this information into the inference. For example, consider time series data of a climate variable observed at specific locations as a response vector, with the location information and other explanatory variables as a covariate vector. Using data from multiple locations, a multivariate linear regression model can be applied to infer how location and covariates influence specific climate variables. In this case, since there is time ordering among the response variables, it is natural to reflect this in the inference. In this paper, we propose a Bayesian method for multivariate linear regression models that can incorporate such known ordering among the response variables. We are particularly interested in methods that allow for joint inference of the coefficient matrix, say B , and the precision matrix of the error term, say \Omega , especially in high-dimensional settings. Several frequentist approaches to solve the joint estimation problem have been proposed and studied in the literature, see [ 25 , 28 , 31 , 6 , 20 , 32 , 10 , 18 ] and the references therein. To address the high-dimensional setting, these methods induce sparsity in the regression coefficient matrix and the precision matrix. This is achieved by optimizing over objective functions comprising of a data-based loss function and an appropriate penalty function to induce sparsity. For some of these methods, for example [ 10 ] and [ 18 ] , algorithmic and statistical convergence properties have also been thoroughly studied. On the Bayesian side, there have been mostly two schools of work, those based on the exact likelihood and others based on the generalized likelihood. Most of the exact likelihood-based methods, except for [ 14 ] , have at least one of these two limitations: (i) assuming B to be row-wise sparse [ 3 , 11 , 13 ] and/or (ii) restricting their analysis to only a special class of precision matrices/graphs such as decomposable graphs [ 3 , 5 ] . Row-wise sparsity of B means that if a certain covariate is not active, then the entire corresponding row in the coefficient matrix is zero. This implies that each covariate either affects all response variables or affects none of them. In contrast to the above approaches, [ 14 ] assumed entry-wise sparsity of B and a general sparse positive definite matrix \Omega by employing multivariate spike and slab Lasso priors. However, their method only provided point estimates without generating posterior samples, making uncertainty quantification through credible intervals impossible. [ 15 ] and [ 27 ] subsequently proposed the methods for entry-wise selection in B and \Omega based on generalized likelihoods for scalable posterior inference, but none of their methods ensure the positive definiteness of \Omega . Motivated by the gap in the literature, we develop two computationally scalable Bayesian methods for the selection and estimation of sparse B and \Omega , while incorporating the ordering among the response variables. We use the modified Cholesky decomposition (MCD) for \Omega , which allows us to leverage the ordering information, achieve fast computation and ensure the positive definiteness of \Omega . The first approach relies on the exact likelihood. We adopt discrete spike and slab priors and DAG-Wishart prior to capture significant signals in B and Cholesky factor of \Omega , respectively. Its computational cost per Markov chain Monte Carlo (MCMC) iteration is comparable to that of the fastest among existing Bayesian methods [ 27 ] . However, it can still be computationally expensive in high-dimensional settings. To further enhance computational speed, inspired by [ 27 ] and [ 6 ] , we propose a two-step approach. In the first step, we simplify the multivariate linear regression model into q independent univariate linear regression models by ignoring the dependencies among the response variables, and apply a variant of spike and slab prior to each coefficient vector to infer B . Using the obtained estimate of B , we calculate the estimated errors by removing the effects of covariates from the responses. In the second step, we assume these estimated errors as observations in a Gaussian model and apply the DAG-Wishart prior to infer the Cholesky factor of \Omega . The computational cost of the proposed two-step method is lower than that of any existing Bayesian methods, making it scalable even in high-dimensional settings. We establish model selection consistency and posterior contraction rates for the two-step approach under mild regularity conditions (Theorems 4.1 – 4.4 ), and simulation studies further signify its satisfactory empirical performance. The associated computational gains, along with the theoretical properties, render the two-step approach suitable and preferable in high dimensions. While leveraging the MCD of \Omega ensures its positive definiteness, we assume an ordering among the response variables. Therefore, the developed method can be naturally applied in cases where the data has an ordering, such as time ordering or causal relationships. In the absence of ordering information, methods to estimate the ordering should be applied before using the approach discussed in this paper. Further details can be found in [ 16 ] and [ 23 ] . The rest of paper is organized as follows. Section 2 introduces the multivariate linear regression model, as well as the concepts of the Gaussian directed acyclic graph (DAG) model and the DAG-Wishart distribution, which are necessary to formulate the proposed model in this paper. In Section 3 , we present a Bayesian method based