Fast Riemannian-manifold Hamiltonian Monte Carlo for hierarchical Gaussian-process models
Takashi Hayakawa, Satoshi Asai · 2025
Read Fast Riemannian-manifold Hamiltonian Monte Carlo for hierarchical Gaussian-process models by Takashi Hayakawa, Satoshi Asai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06407.txt.
Introduction
Takashi Hayakawa and Satoshi Asai make a powerful sampling method practical for layered Gaussian process models. Such Bayesian models capture tangled nonlinear patterns in real data, but drawing samples from their posteriors has stayed slow outside simple cases. Riemannian-manifold Hamiltonian Monte Carlo adapts its geometry to the model, yet standard toolkits run it sluggishly.
The speedup comes from reordering computation to match the model's structure and caching matrix decompositions through dynamic programming. A naive automatic differentiator inside existing toolkits cannot deliver that gain. Tests on simulated logistic regression and on national medical spending data showed effective posterior sampling and evidence values, laying groundwork for better Gaussian process software.
Catalog record
| CREATOR | Takashi Hayakawa, Satoshi Asai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06407
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fast Riemannian-manifold Hamiltonian Monte Carlo for hierarchical Gaussian-process models Takashi Hayakawa, Satoshi Asai Abstract Hierarchical Bayesian models based on Gaussian processes are considered useful for describing complex nonlinear statistical dependencies among variables in real-world data. However, effective Monte Carlo algorithms for inference with these models have not yet been established, except for several simple cases. In this study, we show that, compared with the slow inference achieved with existing program libraries, the performance of Riemannian-manifold Hamiltonian Monte Carlo (RMHMC) can be drastically improved by optimising the computation order according to the model structure and dynamically programming the eigendecomposition. This improvement cannot be achieved when using an existing library based on a naive automatic differentiator. We numerically demonstrate that RMHMC effectively samples from the posterior, allowing the calculation of model evidence, in a Bayesian logistic regression on simulated data and in the estimation of propensity functions for the American national medical expenditure data using several Bayesian multiple-kernel models. These results lay a foundation for implementing effective Monte Carlo algorithms for analysing real-world data with Gaussian processes, and highlight the need to develop a customisable library set that allows users to incorporate dynamically programmed objects and finely optimises the mode of automatic differentiation depending on the model structure. 1 Introduction Bayesian analysis based on Gaussian processes (GPs) associated with positive-semidefinite kernels has been studied extensively from modelling, algorithmic, and theoretical points of view [ 45 , 40 , 41 , 38 , 15 , 39 , 46 , 14 ] . A GP serves as a prior process for generating nonlinear functions that describe relationships between variables. The goal of Bayesian analysis is to infer the posterior process for such nonlinear functions conditioned on given data or to estimate the average of quantities of interest over such a posterior process. Although the posterior process can be represented as a closed-form solution for the simplest problems (e.g., Chapter 6 of Ref. [ 4 ] ), such a solution cannot be expected when GPs are used as building blocks for hierarchical models that capture complex statistical dependencies among variables in real-world data. Algorithms for inference with hierarchical GP models can be broadly divided into variational algorithms and Monte Carlo (MC) algorithms. Variational algorithms (e.g., Refs. [ 15 , 46 ] ) are usually much faster than MC algorithms but introduce unavoidable bias due to discrepancies between the original and variational models. In contrast, MC algorithms (e.g., Refs. [ 39 , 35 , 34 ] ) allow for precise computation of averages over the posterior, provided the algorithm is run until convergence. The preferred algorithm type depends on the purpose of the computation. With biometric and econometric applications in mind, we seek an efficient MC algorithm in this study. In these applications, biased results can be misleading and may adversely affect public health or the economy; therefore, the use of variational algorithms is not appropriate. However, to our knowledge, a standard MC algorithm for general hierarchical GP models has not yet been established. Svensson et al. [ 39 ] sampled a function from the posterior process using a closed-form solution for the GP component of their model conditioned on other parameters. Their formula can be applied only when the likelihood term is Gaussian and is thus not applicable to, for example, binary classification. Hensman et al. [ 22 ] introduced a non-Gaussian variational approximation of the posterior with the aid of inducing points sampled using Hamiltonian MC (HMC). Their method, however, is variational and not guaranteed to describe the posterior precisely, and its scalability with increasing numbers or dimensions of GPs remains unclear. Pandita et al. [ 34 ] introduced an adaptive sequential MC for GPs to solve problems in mechanical engineering. However, the authors used hundreds of CPU cores for inference, and the efficiency of their sampling scheme itself was not clearly demonstrated. One fundamental issue in MC sampling from the posterior in a GP model is that the posterior may be highly stretched or compressed along unknown directions in the parameter space used for sampling. To address this problem, Paquet and Fraccaro [ 35 ] implemented Riemannian-manifold HMC (RMHMC) for GP models. With the Hessian metric they employed, RMHMC adjusts the direction of MC moves according to the curvature of the log-posterior density surface. They demonstrated that RMHMC is much more efficient than the ordinary HMC method based on the Euclidean metric. However, their method is applicable only to cases with log-concave posterior densities, whereas hierarchical models that describe complex dependencies among variables typically have non-log-concave posterior densities. To overcome this limitation, we develop an efficient implementation of RMHMC for hierarchical GP models using a soft-absolute Hessian metric introduced by Betancourt [ 2 ] . This metric transforms an indefinite Hessian into a positive-semidefinite matrix that shares eigenvectors with the Hessian but has associated eigenvalues set close to the absolute values of the original Hessian eigenvalues. Although a general-purpose implementation of RMHMC with this metric [ 9 ] based on PyTorch [ 36 ] is available, its performance is poor. We identify the reasons for this, and show that the computational complexity of the implementation can be considerably reduced. The article is organised as follows. In section 2.1 , we introduce our problem setting in which multiple GPs are used as building blocks for a hierarchical model, employing the representation of GPs introduced by Solin et al. [ 37 ] . In section 2.2 , we provide a brief introduction to RMHMC with a soft-absolute Hessian metric. In sections 2.3 and 2.4 , we identify two sources of redundancy in the calculation of the gradient flow and metric, respectively. We show how performance deteriorates when reverse-mode automatic differentiation and a divide-and-conquer algorithm for the eigendecomposition of symmetric matrices in PyTorch are naively applied. In section 3 , we numerically demonstrate that our implementation considerably outperforms an implementation based on a general-purpose library. Specifically, in section 3.1 , we compare the performance of different implementations using toy examples of varying sizes and show that both sources of redundancy contribute to the improvement. In section 3.2 , we compare the performance of our implementation with that of no-U-turn (NUT)-HMC based on the Euclidean metric. We show that NUT-HMC becomes trapped within a very narrow region and fails to converge to the posterior, suggesting that the use of a data-adaptive metric is essential for efficient sampling. In section 3.3 , we show, using a toy example and a real-world dataset, that our implementation successfully calculates the marginal likelihood, (that is, the Bayesian model evidence) within a reasonable computation time. Finally, in section 4 , we discuss, based on these results, the advantages and limitations of the proposed method compared with existing approaches and suggest directions for future work. Mathematical notations used in this article are summarised in Table 2 . 2 Methods and Algorithms 2.1 Hierarchical GP models and their approximate representations In this study, we consider hierarchical GP models for which the samplewise likelihood of i.i.d. data X_{i}\in\mathbf{R}^{d} (with sample index i\in\{1,2,\ldots,N\} ) is parameterised by J nonlinear functions \{f_{j}(X_{i})\}_{1\leq j\leq J} as \displaystyle P(X_{i}|\{f_{j}\}_{1\leq j\leq J})=\exp(-U(\{f_{j}(X_{i})\}_{1\leq j\leq J})). (1) For instance, the simplest examples describing the relationship between a target variable X_{i}^{(\mathrm{tgt})} and covariates