STRIDE: Sparse Techniques for Regression in Deep Gaussian Processes

Simon Urbainczyk, Aretha L. Teckentrup, Jonas Latz · 2025

Read STRIDE: Sparse Techniques for Regression in Deep Gaussian Processes by Simon Urbainczyk, Aretha L. Teckentrup, Jonas Latz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11355.txt.

Introduction

The authors speed up deep Gaussian processes, a layered learning model prized for honest uncertainty estimates. Such models strain under big datasets and wavy, multi-scale patterns. Sparse methods shrink the math by picking a few key inducing points, but training the deep version still leans on approximations that distort uncertainty.

Their method, STRIDE, mixes two strategies. A variational step picks the inducing points from the data, while a sampling step trains the layers more faithfully. The pair runs as a particle-based loop that improves both halves in turn. On standard benchmarks, the method trains large, deep models faster and more accurately than before, opening the approach to datasets that once were out of reach.

Catalog record

CREATORSimon Urbainczyk, Aretha L. Teckentrup, Jonas Latz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11355

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

STRIDE: Sparse Techniques for Regression in Deep Gaussian Processes Simon Urbainczyk, Aretha L. Teckentrup, Jonas Latz Abstract Gaussian processes (GPs) have gained popularity as flexible machine learning models for regression and function approximation with an in-built method for uncertainty quantification. However, GPs suffer when the amount of training data is large or when the underlying function contains multi-scale features that are difficult to represent by a stationary kernel. To address the former, training of GPs with large-scale data is often performed through inducing point approximations (also known as sparse GP regression (GPR)), where the size of the covariance matrices in GPR is reduced considerably through a greedy search on the data set. To aid the latter, deep GPs have gained traction as hierarchical models that resolve multi-scale features by combining multiple GPs. Posterior inference in deep GPs requires a sampling or, more usual, a variational approximation. Variational approximations lead to large-scale stochastic, non-convex optimisation problems and the resulting approximation tends to represent uncertainty incorrectly. In this work, we combine variational learning with MCMC to develop a particle-based expectation-maximisation method to simultaneously find inducing points within the large-scale data (variationally) and accurately train the GPs (sampling-based). The result is a highly efficient and accurate methodology for deep GP training on large-scale data. We test our method on standard benchmark problems. 1 Introduction Gaussian process regression is a popular Bayesian methodology in supervised learning, non-parametric statistics, and function approximation, with a variety of favourable features such as tractable inference and uncertainty quantification. When GPs are not sufficiently flexible, we can construct deep GPs by, e.g., sequentially composing a number of otherwise independent GPs. Deep GPR retains many features of GPR, but importantly makes posterior inference much more computationally challenging. Sampling-based techniques for deep GPR require repeated GPR steps – infeasible if the number of observed data sets is large. In the present work, we propose STRIDE a method that combines Markov chain Monte Carlo (MCMC) with sparse GPR for an accurate and computationally efficient deep GPR. Indeed, we (i) propose a sparse-Gaussian-process-marginalised MCMC sampler for deep Gaussian process regression, (ii) propose a heuristic greedy algorithm to find inducing points by optimising a deep variational lower bound, (iii) construct STRIDE by combining (i), (ii) in a Monte Carlo Expectation-Maximisation methodology, (iv) discuss approximate sampling techniques on lower deep Gaussian process layers to reduce the overall complexity to linear in the size of the training dataset, (v) verify our methodology in usual benchmark problems and in a large-scale Fashion-MNIST problem, where it beats comparable sparse Gaussian processes. After a literature review, we collect the technical background in conventional, deep, and sparse GPR in Section 2 , before we derive STRIDE in Section 3 , summarise our experiments and their results in Section 4 , and conclude in Section 5 . Literature review. Sparse GPR is a popular tool for solving regression problems involving large numbers of data points, see e.g. the works [ 41 , 35 , 7 , 38 ] as well as the unifying perspectives [ 3 , 30 ] . In STRIDE, we focus on the objective function described in [ 38 ] to select optimal inducing points. In the context of deep GPR, the posterior needs to be approximated with samples, e.g. through MCMC [ 14 , 24 , 34 , 23 ] , or a variational approximation [ 10 , 19 , 9 , 8 ] . Variational approximations are popular due to their scalability, but challenges remain as they require non-convex large-scale optimisation, tend to misrepresent uncertainty, and are not yet fully understood theoretically [ 2 ] . In this work, we therefore focus on provably convergent MCMC methods for reliable uncertainty quantification. Sparse GPR techniques based on [ 38 ] have successfully been employed in variational approximations of deep GPs [ 10 , 11 ] , as well as the related setting of hierarchical GPs with hyper-parameters [ 32 ] . f_{0} f_{1} f_{2} Convolution (f_{0})^{1} (f_{1})^{1} (f_{0})^{2} (f_{1})^{2} f_{2} Composition Composition Composition Figure 1 : Illustration of 3-layer deep Gaussian processes with convolution (top) and composition (bottom) architectures. The input dimension is d=2 . The deep GPs have been conditioned with respect to data from a simple piecewise-constant target. Deep GPs can be constructed in a variety of ways, typically based on a hierarchical approach to non-stationary GP modelling [ 14 , 33 ] . When combining the individual GP layers in said hierarchical fashion, the most prominent example is the composition architecture as described in the seminal papers [ 10 , 15 ] . The convolution architecture, on the other hand, uses the individual GP layers to define a non-stationary correlation length for the next layer [ 31 , 14 ] . We illustrate both architectures in Figure 1 . A synthesis of composition and convolution architectures was recently proposed in [ 12 ] . STRIDE is based on the expectation-maximisation (EM) algorithm [ 13 , 36 ] . We also refer to [ 25 ] for variants of EM, including Monte Carlo EM [ 39 ] which is used in this work. While we combine preconditioned Crank–Nicolson (pCN) MCMC [ 6 ] for the expectation step and a greedy method for the maximisation step, previous works have combined Langevin sampling and gradient flows [ 20 , 22 ] in a similar context. 2 Gaussian process regression and extensions In the following we consider the following supervised learning problem: we are given data pairs (x_{i},y_{i})_{i=1}^{n}\in(\mathbb{R}^{d}\times\mathbb{R})^{n} through f^{\dagger}(x_{i})+\varepsilon_{i}=y_{i} , where f^{\dagger} is an unknown function and \varepsilon_{1},...,\varepsilon_{n}\sim\mathrm{N}(0,\gamma^{2}) i.i.d. is observational noise. Our goal is to recover the function f^{\dagger} from the observational data. GPR is a natural Bayesian strategy to approach problems of this form. Given a prior \mathrm{GP}(0,k(\cdot,\cdot)) , we obtain the posterior \mathrm{GP}(\mu^{y}(\cdot),k^{y}(\cdot,\cdot)) , with \displaystyle\mu^{y}(\cdot) \displaystyle=k(\cdot,\mathbf{x}_{n})\mathopen{}\mathclose{{\left(k(\mathbf{x}_{n},\mathbf{x}_{n})+\gamma^{2}\mathrm{Id}}}\right)^{-1}\mathbf{y}_{n}\>, (1) \displaystyle k^{y}(\cdot,\cdot) \displaystyle=k(\cdot,\cdot)-k(\cdot,\mathbf{x}_{n})\mathopen{}\mathclose{{\left(k(\mathbf{x}_{n},\mathbf{x}_{n})+\gamma^{2}\mathrm{Id}_{n}}}\right)^{-1}k(\mathbf{x}_{n},\cdot)\>. Here, the bold letters refer to the full data sets \mathbf{x}_{n}:=(x_{i})_{i=1}^{n} and \mathbf{y}_{n}:=(y_{i})_{i=1}^{n} . When kernels are applied to these sets, we obtain kernel matrices/vectors. In the following, we discuss two extensions of GPR: deep GPR in cases where GPs are not sufficiently flexible (Subsections 2.1 , 2.2 ) and sparse GPR as a computationally efficient approximation of GPR.(Subsection 2.3 ). 2.1 Deep Gaussian processes A sequence of random functions (f_{\ell})_{0\leq\ell\leq\ L} is called a deep Gaussian process with L+1 layers if f_{0} is a Gaussian process, and if f_{\ell} is conditionally Gaussian for \ell\geq 1 , i.e. \pi_{f_{\ell}|f_{\ell-1}=u_{\ell-1}}=\mathrm{GP}(0,k_{\ell}(\cdot,\cdot\,|u_{\ell-1})), for any realisation u_{\ell-1} of f_{\ell-1} . The properties of a deep GP strongly depend on the choice of the conditional covariance kernel k_{\ell} , which determines its architecture. We consider two approaches here, namely, the convolution architecture described in [ 27 , 14 ] , as well as the composition/warping architecture [ 10 , 14 , 34 ] . Convolution. Covariance kernels are often constructed in the following way: we define an isotropic correlation function \rho:\mathbb{R}\rightarrow\mathbb{R} and then obtain a covariance kernel through k(x,x^{\prime})=\sigma^{2}\rho(\|x-x^{\prime}\|\>/\>\lambda) for some marginal variance \sigma^{2}>0 and correlation length \lambda>0 . The correlation length determines the strength of long-range correlation in a GP and, thus, the oscillatory behaviour of its samples. In the convolution architecture, we now use a spatially-dependent value for \lambda that is given through the realisation u_{\ell-1}

Read the complete text