Efficient Gaussian Process Learning via Subspace Projections
Elsa CazellesCNRS, IRIT, Université de Toulouse, Felipe TobarImperial College London · 2026
Read Efficient Gaussian Process Learning via Subspace Projections by Elsa CazellesCNRS, IRIT, Université de Toulouse, Felipe TobarImperial College London (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16332.txt.
Introduction
Elsa Cazelles and Felipe Tobar train Gauss process models with low size casts of data. Their 2026 preprint deals with models that give both a guess and its doubt. Full train of such models costs much on big data. The team builds a new train goal from casts to low size. Casts are made by straight maps of the full data.
They give a firm form for the facts lost in the cast. Tests show chance casts to a round set can cut that loss. Low loss at low cost would scale such models to big sets. The goal is named projected likelihood in the work. It keeps doubt marks that users trust.
Catalog record
| CREATOR | Elsa CazellesCNRS, IRIT, Université de Toulouse, Felipe TobarImperial College London |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16332
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Gaussian Process Learning via Subspace Projections Elsa CazellesCNRS, IRIT, Université de Toulouse, Felipe TobarImperial College London Abstract We propose a novel training objective for GPs constructed using lower-dimensional linear projections of the data, referred to as projected likelihood (PL). We provide a closed-form expression for the information loss related to the PL and empirically show that it can be reduced with random projections on the unit sphere. We show the superiority of the PL, in terms of accuracy and computational efficiency, over the exact GP training and the variational free energy approach to sparse GPs over different optimisers, kernels and datasets of moderately large sizes. * * footnotetext: The authors are listed in alphabetical order. 1 Introduction The Gaussian process (GP) [ 21 ] is the de facto approach to time series, especially in regimes of unevenly-sampled observations, missing data and when uncertainty quantification is required. GPs have been applied to EEG [ 5 ] , audio [ 1 ] , seismology [ 10 ] , climate [ 2 ] , body-motion sensing [ 20 ] and astronomy [ 8 ] . What makes GPs suitable for a wide range of time series applications is a comprehensive procedure for designing and choosing tailored covariance kernels, which allow practitioners to incorporate inductive biases, thus elegantly blending prior knowledge with rich relationships learnt from data. Conceptually, GP regression models construct a Gaussian prior distribution on the infinite-dimensional space of functions, defined by a mean function (usually set to zero) and a covariance kernel [ 14 ] . Then, observations are considered as parts of the latent true function and used to update the GP prior into a GP posterior. For the model to accurately represent the data, the prior needs to be calibrated with respect to the available data, meaning that particular mean and covariance functions need to be chosen. Denoting the number of observations by n , GP training is performed by means of maximum likelihood (ML) at a cost \mathcal{O}(n^{3}) , which renders this approach unfeasible for more than a few thousand datapoints. A vast collection of approximate-likelihood methods has been developed to alleviate the computational cost related to standard ML-based GP training [ 13 , 15 ] . These approaches either directly replace the inverse covariance matrix with a lower-rank approximation, or assume the conditional independence of the data under a set of (trainable) inducing variables , which also results in a reduced-complexity objective. Despite the proven success of the inducing-variable approach to GPs, termed sparse GPs, they are prone to: i) provide biased learnt hyperparameters for a reduced computational cost, and ii) only achieve superior computational efficiency asymptotically (i.e., for n sufficiently large). For instance, [ 3 ] states that the fully independent training conditional (FITC) method [ 17 ] is known to underestimate the noise variance, while the variational free energy (VFE) method [ 19 ] is known to overestimate it. For VFE, this hyperparameter mismatch comes from maximising the evidence lower bound, whose bias follows from the restricted choice of the approximating variational distribution [ 4 ] . Furthermore, due to the more complex training loss of sparse GPs, involving several Cholesky decompositions, VFE’s asymptotic efficiency is only realised for n sufficiently large. To address the need for a GP training method that is both computationally efficient and accurate for moderately large datasets, we propose a novel training objective, termed projected likelihood (PL), given by the likelihood of a surrogate GP over a lower-dimensional linear projection of the data. We quantify the information loss related to the PL and show it can be reduced using k random projections, thus attaining a training cost \mathcal{O}(kn^{2}) , where k\ll n . Despite this cost being seemingly larger than the standard \mathcal{O}(nm^{2}) cost of the VFE sparse GP with m inducing points, we empirically show that the PL is computationally efficient, due to requiring fewer optimisation steps, and accurate in terms of achieving solutions that are closer to those of the exact GP method. The proposed PL method is experimentally validated using real-world and synthetic data of varying magnitude, as well as different kernels and optimisers. 2 Gaussian processes for regression 2.1 Training and inference A Gaussian process (GP) is a stochastic process \{f(x):x\in{\mathbb{R}}^{d}\} such that any finite collection (f(x_{1}),\dots,f(x_{n})) follows a n -multivariate Gaussian distribution. We adopt the notation f(x)\sim\mathcal{GP}\big(m(x),k(x,x^{\prime})\big), where m(x)=\mathbb{E}[f(x)] is the mean function—usually set to zero—and k(x,x^{\prime})=\operatorname{Cov}(f(x),f(x^{\prime})) is the covariance kernel. Given training data \mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{n} , assuming a Gaussian likelihood y_{i}=f(x_{i})+\varepsilon , \varepsilon\sim\mathcal{N}(0,\sigma^{2}_{\text{n}}) , and denoting the observations as stacked vectors {\mathbf{x}} and {\mathbf{y}} , the marginal likelihood is p({\mathbf{y}}|{\mathbf{x}},\theta)=\mathcal{N}\!\big({\mathbf{y}}|\,m_{\mathbf{x}},\,K_{{\mathbf{x}}}+\sigma^{2}_{\text{n}}\mathbf{I}\big), (1) where \theta are the kernel hyperparameters, and we used the compact notation m_{\mathbf{x}}=m({\mathbf{x}})=[m(x_{1}),\ldots,m(x_{n})]^{T} and K_{{\mathbf{x}}}=K({\mathbf{x}},{\mathbf{x}}) , where (K({\mathbf{x}},{\mathbf{x}}))_{ij}=k(x_{i},x_{j}) . We do not denote the explicit dependence of m and k on \theta for the sake of notation simplicity. Training is performed by minimising the negative log likelihood (NLL) {\cal L}(\theta)=-\log p({\mathbf{y}}|{\mathbf{x}},\theta) given by (assuming m_{\mathbf{x}}=0 ): {\cal L}(\theta)=\tfrac{1}{2}{\mathbf{y}}^{\top}(K_{{\mathbf{x}}}+\sigma^{2}_{\text{n}}\mathbf{I})^{-1}{\mathbf{y}}+\tfrac{1}{2}\log|K_{{\mathbf{x}}}+\sigma^{2}_{\text{n}}\mathbf{I}|+\frac{n}{2}\log 2\pi. (2) Furthermore, for test inputs {\mathbf{x}}_{*} , the GP’s predictive posterior distribution is given by [ 14 ] p({\mathbf{f}}_{*}|{\mathbf{x}},{\mathbf{y}},{\mathbf{x}}_{*})=\mathcal{N}\!\big(\mu_{*},\Sigma_{*}\big) , with \displaystyle\mu_{*} \displaystyle=K_{{\mathbf{x}}_{*}{\mathbf{x}}}(K_{{\mathbf{x}}}+\sigma^{2}_{\text{n}}\mathbf{I})^{-1}{\mathbf{y}}, (3) \displaystyle\Sigma_{*} \displaystyle=K_{{\mathbf{x}}_{*}}-K_{{\mathbf{x}}_{*}{\mathbf{x}}}\,(K_{{\mathbf{x}}}+\sigma^{2}_{\text{n}}\mathbf{I})^{-1}K_{{\mathbf{x}}{\mathbf{x}}_{*}}, (4) where K_{{\mathbf{x}}_{*}}=K({\mathbf{x}}_{*},{\mathbf{x}}_{*}) and K_{{\mathbf{x}}_{*}{\mathbf{x}}}^{\top}=K_{{\mathbf{x}}{\mathbf{x}}_{*}}=K({\mathbf{x}},{\mathbf{x}}_{*}) . 2.2 Sparse approximations To alleviate the \mathcal{O}(n^{3}) cost of eq. ( 2 ), computationally-efficient approximations of {\cal L}(\theta) rely on m inducing variables , which summarise the observations sufficiently well [ 15 ] . Specifically, the reduced computational cost comes from assuming that the n observations are conditionally independent given the m inducing variables. This approach, termed sparse GPs , includes the early Deterministic Training Conditional (DTC) [ 16 ] and the Fully Independent Training Conditional (FITC) [ 17 ] . For a revision of classic sparse GP methods and how they relate to one another, see [ 13 ] . A cornerstone in sparse GP approximations is the Variational Free Energy (VFE) method [ 19 ] , which has become the basis for subsequent approaches. VFE models the inducing variables by a variational distribution q(\mathbf{u}) , thus approximating the posterior over f as q(f)=\int p(f|\mathbf{u})q(\mathbf{u})d\mathbf{u} . Training then involves finding the variational parameters by maximising the evidence lower bound (ELBO): \mathcal{L}=\mathbb{E}_{q(f)}[\log p({\mathbf{y}}|f)]-\mathrm{KL}[q(\mathbf{u})\|p(\mathbf{u})], (5) which ensures the approximation is close to the true posterior while reducing complexity to \mathcal{O}(nm^{2}) . Denoting the locations of the inducing variables by \bar{\mathbf{x}}\in{\mathbb{R}}^{d\times m} , the ELBO is given by \displaystyle\mathcal{L}_{\mathrm{VFE}} \displaystyle=-\frac{1}{2}\Big[{\mathbf{y}}^{\top}(Q_{{\mathbf{x}}}+\sigma^{2}_{\text{n}}\mathbf{I})^{-1}{\mathbf{y}}+\log|Q_{{\mathbf{x}}}+\sigma^{2}_{\text{n}}\mathbf{I}|+n\log 2\pi\Big] \displaystyle-\frac{1}{2\sigma^{2}_{\text{n}}}\mathrm{tr}(K_{{\mathbf{x}}}-Q_{{\mathbf{x}}}), (6) where Q_{{\mathbf{x}}}=K_{{\mathbf{x}}\bar{\mathbf{x}}}K_{\bar{\mathbf{x}}}^{-1}K_{\bar{\mathbf{x}}{\mathbf{x}}},K_{\bar{\mathbf{x}}}=k(\bar{\mathbf{x}},\bar{\mathbf{x}}) and K_{{\mathbf{x}}\bar{\mathbf{x}}}=k({\mathbf{x}},\bar{\mathbf{x}}) . Since \bar{\mathbf{x}} has to be learnt, the introduction of additional variational parameters leads to learning biased hyperparameters and a computational efficiency that only arises for sufficiently large n . 3 The projected likelihood method 3.1 Projections onto the unit sphere \mathcal{S}^{n-1} We consider k\ll n linear projections of the observations, interpreted as a k -dimensional, lossy compression of the n -dimensional data distribution, from which we can derive a cheaper training objective approximating {\cal L}(\theta) in eq. ( 2 ). We conjecture that, if the relevant directions of K_{{\mathbf{x}}} are preserved in the projected data, the hyperparameters can be recovered with minimal information loss and computational overhead. We project {\mathbf{y}}\in{\mathbb{R}}^{n} onto a set of directions \bm{\omega}_{1},\ldots,\bm{\omega}_{k}\in\mathcal{S}^{n-1} , where \mathcal{S}^{n-1} denotes the unit sphere in {\mathbb{R}}^{n} , obtaining {\mathbf{z}}=[\bm{\omega}_{1}^{\top}{\mathbf{y}},\ldots,\bm{\omega}_{k}^{\top}{\mathbf{y}}]\in{\mathbb{R}}^{k}. (7) Assuming {\mathbf{y}} is distributed by a zero-mean