Transformed Latent Variable Multi-Output Gaussian Processes
Xiaoyu Jiang, Xinxing Shi, Sokratia Georgaka, Magnus Rattray, Mauricio A. Álvarez, Xiaoyu Jiang, Xinxing Shi, Sokratia Georgaka, Magnus Rattray, Mauricio A. Álvarez · 2026
Read Transformed Latent Variable Multi-Output Gaussian Processes by Xiaoyu Jiang, Xinxing Shi, Sokratia Georgaka, Magnus Rattray, Mauricio A. Álvarez, Xiaoyu Jiang, Xinxing Shi, Sokratia Georgaka, Magnus Rattray, Mauricio A. Álvarez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05133.txt.
Introduction
Xiaoyu Jiang and four co-authors scale multi-output Gaussian processes, a chance-based method for modelling many linked outputs at once, to truly large output spaces. Existing versions stay tractable only by using narrow kernel forms that limit what patterns they can express. Their model instead builds a flexible deep kernel from inputs plus per-output latent values mapped through a controlled neural network.
Paired with stochastic variational inference, the framework keeps cost manageable while keeping real links between outputs. Benchmarks include climate work with more than ten thousand outputs and sparse spatial transcriptomics data. In both settings the method beat baselines on accuracy and speed, the authors report.
Catalog record
| CREATOR | Xiaoyu Jiang, Xinxing Shi, Sokratia Georgaka, Magnus Rattray, Mauricio A. Álvarez, Xiaoyu Jiang, Xinxing Shi, Sokratia Georgaka, Magnus Rattray, Mauricio A. Álvarez |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 141 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05133
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transformed Latent Variable Multi-Output Gaussian Processes Xiaoyu Jiang, Xinxing Shi, Sokratia Georgaka, Magnus Rattray, Mauricio A. Álvarez, Xiaoyu Jiang, Xinxing Shi, Sokratia Georgaka, Magnus Rattray, Mauricio A. Álvarez Abstract Multi-Output Gaussian Processes (MOGPs) provide a principled probabilistic framework for modelling correlated outputs but face scalability bottlenecks when applied to datasets with high-dimensional output spaces. To maintain tractability, existing methods typically resort to restrictive assumptions, such as employing low-rank or sum-of-separable kernels, which can limit expressiveness. We propose the Transformed Latent Variable MOGP (T-LVMOGP), a novel framework that scales MOGPs to a massive number of outputs while preserving the capacity to capture meaningful inter-output dependencies. T-LVMOGP constructs a flexible multi-output deep kernel by mapping inputs and output-specific latent variables into an embedding space using a Lipschitz-regularised neural network. Combined with stochastic variational inference, our model effectively scales to high-dimensional output settings. Across diverse benchmarks, including climate modelling with over 10,000 10,000 outputs and zero-inflated spatial transcriptomics data, T-LVMOGP outperforms baselines in both predictive accuracy and computational efficiency. 1 Introduction Gaussian Processes (GPs) offer a principled and flexible Bayesian nonparametric framework for learning unknown functions ( 70 ) . Multi-Output Gaussian Processes (MOGPs) extend this framework to model correlations between output functions, proving effective across domains ranging from medical time-series prediction ( 14 ) to dynamical system modelling ( 59 ) . However, standard MOGP models incur a cubic computational cost with respect to the number of outputs P . This constitutes a bottleneck for high-dimensional applications such as climate modelling and spatial transcriptomics, where P can easily reach the order of thousands ( 8 ; 67 ) . Figure 1 : Schematic overview of the proposed T-LVMOGP framework. Inducing points \mathbf{Z} are placed in the embedding space. To mitigate this complexity, existing research largely focuses on imposing structural constraints, potentially sacrificing expressiveness for scalability. For instance, low-rank approaches assume output functions are linear combinations of shared latent functions ( 69 ; 26 ; 76 ; 11 ) . While computationally efficient, these linear assumptions often lack the expressiveness required to model intricate inter-output dependencies. For tensor-based observations, the outputs can be shaped into a multidimensional array so that the Kronecker and tensor algebra are exploited for efficient computation ( 82 ; 36 ) . However, this approach is predicated on rigid structural assumptions, which are frequently violated in real-world scenarios. 17 introduce a latent variable for each output, and construct separable kernels for MOGP. 30 generalise this idea to sum-of-separable kernels, and they apply stochastic variational inference techniques ( 28 ; 25 ) to the output space, facilitating better scalability with respect to an increasing number of outputs. However, their expressivity remains bound by the fixed structural properties of the kernel class. In this work, we propose the Transformed Latent Variable MOGP (T-LVMOGP), illustrated in Figure 1 , a framework that scales MOGPs to large output dimensions while preserving high expressiveness. To this end, we introduce a multi-output deep kernel that captures intricate correlations among outputs without imposing rigid structural constraints. We associate each output with a latent variable, serving as a compact representation ( 9 ) . To ensure high expressiveness, we employ a neural network to map the concatenated input-latent representations into an embedding space, where correlations are defined ( 27 ; 13 ; 72 ) . This architecture enables seamless integration with existing scalable GP frameworks to handle a large output space. We employ the Sparse Variational GP (SVGP) framework ( 62 ) , placing inducing points in the embedding space and performing stochastic variational inference following 25 . Furthermore, recently proposed tighter variational bounds ( 63 ; 12 ) can be readily incorporated into this framework, which potentially enhances predictive performance. While deep kernels provide the requisite flexibility, naive implementations suffer from overfitting ( 50 ) . In this work, we regularise the neural network via residual connections ( 24 ) and spectral normalisation ( 46 ) . This encourages the transformation to be Lipschitz continuous, and its distance awareness promotes locally smooth and globally stable mappings. This is critical for the reliable performance of the multi-output deep kernel proposed in this work. Contributions This work makes the following contributions: (1) We propose the Transformed Latent Variable MOGP, a scalable framework for building flexible multi-output deep kernels that scale to high-dimensional outputs while preserving rich cross-output dependencies. (2) To improve generalisation, we adopt a Lipschitz continuous neural network using residual connections and spectral normalisation. (3) We develop a stochastic variational inference procedure within the SVGP framework, supporting mini-batch training over both inputs and outputs. Our formulation naturally accommodates non-Gaussian likelihoods and tighter variational bounds. (4) Empirical experiments demonstrate that our model outperforms MOGP baselines in both predictive accuracy and computational efficiency. 2 Background 2.1 Gaussian Processes Consider a regression problem with a dataset of N input-output pairs (\mathbf{X},\mathbf{y})=\{(\mathbf{x}_{n},y_{n})\}_{n=1}^{N} , where \mathbf{X}\in\mathbb{R}^{N\times D_{X}} and \mathbf{y}\in\mathbb{R}^{N} . We assume the outputs are generated by a latent function f with observation noise, y_{n}=f(\mathbf{x}_{n})+\epsilon_{n} . A zero-mean GP places a prior over the function f\sim\mathcal{GP}(0,k(\cdot,\cdot^{\prime})) . The function values \mathbf{f}=f(\mathbf{X}) follow a multivariate Gaussian distribution p(\mathbf{f})=\mathcal{N}(\mathbf{f}\mid\mathbf{0},\mathbf{K}_{\mathbf{XX}}) , where [\mathbf{K}_{\mathbf{XX}}]_{ij}=k(\mathbf{x}_{i},\mathbf{x}_{j}) . Exact GPs incur an \mathcal{O}(N^{3}) computational cost due to the determinant and inversion computation of an N\times N matrix. To scale to large datasets, SVGP ( 62 ) introduces a set of M\ll N inducing inputs \mathbf{Z}\in\mathbb{R}^{M\times D_{X}} and corresponding inducing variables \mathbf{u}=f(\mathbf{Z}) . The joint posterior is approximated using a variational distribution q(f,\mathbf{u})=p(f\mid\mathbf{u})q(\mathbf{u}) , where q(\mathbf{u})=\mathcal{N}(\mathbf{u}\mid\mathbf{m},\mathbf{S}) and \mathbf{m},\mathbf{S} are variational parameters. We denote \mathbf{K}_{\mathbf{ZZ}}=k(\mathbf{Z},\mathbf{Z}) and \mathbf{K}_{\mathbf{XZ}}=k(\mathbf{X},\mathbf{Z}) . The standard SVGP is trained by maximising the Evidence Lower Bound (ELBO): \mathcal{L}_{1}=\sum_{n=1}^{N}\mathbb{E}_{q(f(\mathbf{x}_{n}))}\left[\log p(y_{n}\mid f(\mathbf{x}_{n}))\right]-\operatorname{KL}[q(\mathbf{u})\mid\mid p(\mathbf{u})]. Recently, 63 and 12 propose a refined variational posterior that relaxes the prior conditional assumption. They define q(\mathbf{f},\mathbf{u})=q(\mathbf{f}\mid\mathbf{u})q(\mathbf{u}) with q(\mathbf{f}\mid\mathbf{u})=\mathcal{N}(\mathbf{f}\mid\mathbf{K}_{\mathbf{X}\mathbf{Z}}\mathbf{K}_{\mathbf{Z}\mathbf{Z}}^{-1}\mathbf{u},\mathbf{D}_{\mathbf{X}\mathbf{X}}^{1/2}\mathbf{V}\mathbf{D}_{\mathbf{X}\mathbf{X}}^{\top/2}), where \mathbf{V} is a positive diagonal matrix, \mathbf{Q}_{\mathbf{X}\mathbf{X}}=\mathbf{K}_{\mathbf{X}\mathbf{Z}}\mathbf{K}_{\mathbf{Z}\mathbf{Z}}^{-1}\mathbf{K}_{\mathbf{Z}\mathbf{X}} represents the Nyström approximation, and \mathbf{D}_{\mathbf{X}\mathbf{X}}=\mathbf{K}_{\mathbf{X}\mathbf{X}}-\mathbf{Q}_{\mathbf{X}\mathbf{X}} . This parameterisation leads to a tighter lower bound \mathcal{L}_{2}=\mathcal{L}_{1}+\Delta after optimising out \mathbf{V} , where \Delta\geq 0 serves as a correction term. For the Gaussian likelihood with noise variance \sigma_{y}^{2} , this term is given analytically by: \Delta=\frac{1}{2}\sum_{n=1}^{N}\left[\frac{d_{n}}{\sigma_{y}^{2}}-\log\left(1+\frac{d_{n}}{\sigma_{y}^{2}}\right)\right], (1) where d_{n}=[\mathbf{D}_{\mathbf{X}\mathbf{X}}]_{nn} is the n -th diagonal element of \mathbf{D}_{\mathbf{X}\mathbf{X}} . Further details are provided in Appendix G . 2.2 Multi-Output Gaussian Processes MOGPs are a natural extension of GPs to vector-valued observations \mathbf{Y}\in\mathbb{R}^{N\times P} where \mathbf{y}_{n}\in\mathbb{R}^{P} denotes the observation at input \mathbf{x}_{n} ( 3 ) . A classic approach is the Linear Model of Coregionalisation (LMC), which expresses outputs as linear combinations of independent latent GPs ( 31 ) . Specifically, the p -th output function f_{p} is modelled as: f_{p}(\mathbf{x}_{n})=\sum_{q=1}^{Q}\sum_{r=1}^{R_{q}}\alpha^{(q)}_{p,r}g^{(q)}_{r}(\mathbf{x}_{n}),\;y_{n,p}=f_{p}(\mathbf{x}_{n})+\epsilon_{n,p}, where \epsilon_{n,p}\sim\mathcal{N}(0,\sigma_{y}^{2}) , \{\alpha^{(q)}_{p,r}\} are mixing weights and the functions \{g^{(q)}_{r}(\mathbf{x})\}_{r=1}^{R_{q}} are latent GPs sharing covariance function k_{X}^{(q)}(\mathbf{x},\mathbf{x}^{\prime}) , but sampled independently. The cross-covariance between f_{p}(\mathbf{x}) and f_{p^{\prime}}(\mathbf{x}^{\prime}) is given by \displaystyle\text{cov}[f_{p}(\mathbf{x}),f_{p^{\prime}}(\mathbf{x}^{\prime})] \displaystyle=\sum_{q=1}^{Q}c^{(q)}_{p,p^{\prime}}k_{X}^{(q)}(\mathbf{x},\mathbf{x}^{\prime}), with c_{p,p^{\prime}}^{(q)}=\sum_{r=1}^{R_{q}}\alpha^{(q)}_{p,r}\alpha^{(q)}_{p^{\prime},r} . \mathbf{K}^{(q)}_{\mathbf{X}\mathbf{X}}=k_{X}^{(q)}(\mathbf{X},\mathbf{X}) is from the q -th input kernel. Let \mathbf{f}=[\mathbf{f}_{1}^{\top},\dots,\mathbf{f}_{P}^{\top}]^{\top}\in\mathbb{R}^{PN} be the vectorised latent function values with \mathbf{f}_{p}=f_{p}(\mathbf{X}) . The covariance matrix for \mathbf{f} is \text{cov}[\mathbf{f},\mathbf{f}]=\sum_{q=1}^{Q}\mathbf{A}^{(q)}{\mathbf{A}^{(q)}}^{\top}\otimes\mathbf{K}^{(q)}_{\mathbf{X}\mathbf{X}}=\sum_{q=1}^{Q}\mathbf{C}^{(q)}\otimes\mathbf{K}^{(q)}_{\mathbf{X}\mathbf{X}}, where \otimes denotes the Kronecker product, \mathbf{A}^{(q)}\in\mathbb{R}^{P\times R_{q}} collects \alpha^{(q)}_{p,r} , and the coregionalisation matrix \mathbf{C}^{(q)}\in\mathbb{R}^{P\times P} contains c^{(q)}_{p,p^{\prime}} . Notably, LMC implies that \mathbf{C}^{(q)} is a low-rank matrix derived from a linear kernel applied to the latent output embeddings (the rows of \mathbf{A}^{(q)} ). The Latent Variable MOGP