Active Learning for Manifold Gaussian Process Regression

Yuanxing Cheng 1, Lulu Kang 2, Yiwei Wang 3, and Chun Liu 1 1 Dept. of Applied Mathematics, Illinois Institute of Technology, Chicago, IL, USA 1 Dept. of Mathematics and Statistics, University of Massachusetts, Amherst, MA, USA 3 Dept. of Mathematics, University of California, Riverside, CA, USA · 2025

Read Active Learning for Manifold Gaussian Process Regression by Yuanxing Cheng 1, Lulu Kang 2, Yiwei Wang 3, and Chun Liu 1 1 Dept. of Applied Mathematics, Illinois Institute of Technology, Chicago, IL, USA 1 Dept. of Mathematics and Statistics, University of Massachusetts, Amherst, MA, USA 3 Dept. of Mathematics, University of California, Riverside, CA, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20928.txt.

Introduction

Yuanxing Cheng and co-authors make a learning method choose its own training data wisely in a 2025 computing paper. Gaussian process regression predicts smooth functions but struggles when inputs live on a curved manifold inside high-dimensional space. Their framework trains a neural network to flatten the manifold into a small latent space while a Gaussian process learns there. An active learning rule picks which new points to label next.

The selection rule aims to shrink total prediction error across the whole space, not just at easy points. On synthetic tests the method beat random sequential sampling, even for jagged, discontinuous functions. Smart sampling matters wherever each label costs a lab run or a simulation.

Catalog record

CREATORYuanxing Cheng 1, Lulu Kang 2, Yiwei Wang 3, and Chun Liu 1 1 Dept. of Applied Mathematics, Illinois Institute of Technology, Chicago, IL, USA 1 Dept. of Mathematics and Statistics, University of Massachusetts, Amherst, MA, USA 3 Dept. of Mathematics, University of California, Riverside, CA, USA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20928

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Active Learning for Manifold Gaussian Process Regression Yuanxing Cheng 1, Lulu Kang 2, Yiwei Wang 3, and Chun Liu 1 1 Dept. of Applied Mathematics, Illinois Institute of Technology, Chicago, IL, USA 1 Dept. of Mathematics and Statistics, University of Massachusetts, Amherst, MA, USA 3 Dept. of Mathematics, University of California, Riverside, CA, USA Abstract \WSCpagesetup Cheng, Kang, Wang, Liu ABSTRACT This paper introduces an active learning framework for manifold Gaussian Process (GP) regression, combining manifold learning with strategic data selection to improve accuracy in high-dimensional spaces. Our method jointly optimizes a neural network for dimensionality reduction and a Gaussian process regressor in the latent space, supervised by an active learning criterion that minimizes global prediction error. Experiments on synthetic data demonstrate superior performance over randomly sequential learning. The framework efficiently handles complex, discontinuous functions while preserving computational tractability, offering practical value for scientific and engineering applications. Future work will focus on scalability and uncertainty-aware manifold learning. 1 INTRODUCTION Gaussian Process Regression (GPR) \shortcite rasmussen2006gaussian is a powerful non-parametric framework widely used in machine learning, statistics, and scientific computing for regression and uncertainty quantification tasks. However, standard GPR methods face significant challenges as the input dimension grows, suffering from the curse of dimensionality : computational scalability deteriorates, and the ability to capture the underlying data structure diminishes as data becomes increasingly sparse in high-dimensional spaces [ 1 ] . To address these limitations, manifold Gaussian Processes (mGPs) have emerged as a promising alternative. By leveraging manifold learning techniques, mGPs project high-dimensional data onto a lower-dimensional latent space while preserving its intrinsic geometry \shortcite meila_manifold_2024. This approach not only mitigates the challenges of high-dimensional inference but also enhances predictive performance by exploiting the data’s underlying structure. Recent advances include graph-based distance metrics (e.g., Isomap \shortcite tenenbaum_global_2000) combined with Matérn Gaussian processes \shortcite borovitskiy_matern_2023,fichera_implicit_2024, as well as wrapped Gaussian processes for manifold-valued data \shortcite mallasto_wrapped_2018. Despite these innovations, the performance of GPR—and by extension, mGPs—heavily depends on the quality and representativeness of the training data. In many real-world applications, acquiring labeled data is expensive or time-consuming, necessitating efficient data acquisition strategies. Active learning for GPR has been extensively studied due to the model’s inherent ability to quantify predictive uncertainty, making it a natural fit for sequential data selection. Early work by [ 22 ] laid the theoretical foundation for Bayesian active learning, later adapted to GPR by \shortciteN seo2000gaussian, demonstrating superior convergence compared to passive sampling. Subsequent developments include uncertainty sampling (prioritizing high-variance points) and information-theoretic approaches, such as mutual information maximization for sensor placement \shortcite krause2008near and Bayesian Active Learning by Disagreement (BALD) for parameter-aware selection \shortcite houlsby2011bayesian. These methods have been extended to deep GPR models \shortcite ma2019variational and related tasks like uncertainty quantification and inverse problems \shortcite Chen03072021,heo2025active. In practice, active learning strategies must balance exploration (reducing global uncertainty) and exploitation (optimizing a downstream objective). This trade-off is well-studied in Bayesian optimization (BO) [ 10 ] , where acquisition functions like Expected Improvement (EI) and Upper Confidence Bound (GP-UCB) \shortcite srinivas2010gaussian align naturally with active learning objectives. For instance, GP-UCB provides theoretical regret bounds, making it suitable for bandit-style optimization tasks. Cost-sensitive active learning methods further refine this balance by incorporating labeling costs into the acquisition function \shortcite kapoor2007active, which is critical in domains like medical diagnosis. Recent work has also focused on robustness: \shortciteN pmlr-v84-martinez-cantin18a combined robust regression (GPs with Student-t likelihood) with outlier diagnostics to improve reliability. In the context of mGPs, active learning can efficiently refine the model by targeting regions of high uncertainty or high potential improvement. For example, \shortciteN Sauer02012023 and \shortciteN binois_replication_2019 adopted the integrated mean-squared error (IMSE) criterion—originally proposed by \shortciteN cohn_active_1994,cohn_neural_1996—to minimize prediction error. \shortciteN doi:10.1137/22M1529907 further integrated manifold learning with GP-UCB for optimization on manifolds. However, active learning tailored to mGPs remains underexplored , particularly for regression tasks where latent space geometry and high-dimensional inputs interact dynamically. This paper proposes a novel active learning framework for manifold Gaussian Process regression, bridging the gap between efficient data acquisition and dimensionality reduction. Our key contributions include: • a unified framework integrating manifold learning with IMSE-based active learning for mGPs, • enhanced efficiency through strategic point selection in the latent space, and • scalability for high-dimensional data by operating on intrinsic low-dimensional manifolds. We validate our approach through synthetic experiments, demonstrating improvements over passive sampling and standard active learning methods. The paper is organized as follows: Section reviews GPR fundamentals. Section introduces our active learning method with active learning Cohn (ALC) acquisition function for mGP regression. Section illustrates the framework with three case studies, and Section discusses future directions. 2 Background on GPR The Gaussian Process Regression (GPR) model assumes the following probabilistic representation for the response y(\bm{x}) : y(\bm{x})=\mu(\bm{x})+Z(\bm{x})+\epsilon, where Z(\bm{x})\sim GP(0,\tau^{2}k(\cdot,\cdot)),\quad\textrm{ and }\epsilon\overset{\underset{\mathrm{iid}}{}}{\sim}N(0,\sigma^{2}). Denote \Omega\subset\mathbb{R}^{p} as the input domain and output y(\bm{x})\in\mathbb{R}^{1} . Here, k(\cdot,\cdot):\Omega\times\Omega\to\mathbb{R}_{+} is the correlation function of the process Z(\bm{x}) . Common choices include the Matérn and Gaussian correlation functions (or SE-ARD kernels): \textrm{Mat\'{e}rn: }\prod_{l=1}^{p}\frac{1}{\Gamma(\nu)2^{\nu-1}}\left(\frac{2\sqrt{\nu}|x_{il}-x_{jl}|}{\theta_{l}}\right)^{\nu}K_{\nu}\left(\frac{2\sqrt{\nu}|x_{il}-x_{jl}|}{\theta_{l}}\right),\quad\textrm{Gaussian: }\exp\left\{-\sum_{l=1}^{p}\frac{(x_{il}-x_{jl})^{2}}{\theta_{l}}\right\}, where K_{\nu} is the modified Bessel function of the second kind and \bm{\theta}\in\mathbb{R}^{p}_{+} are length-scale parameters. The mean function \mu(\bm{x}) determines the model class: • Ordinary kriging : \mu(\bm{x}) is an unknown constant; • Universal kriging : \mu(\bm{x})=\beta_{0}+\bm{\beta}^{\top}\bm{x} (linear in inputs); • General case : \mu(\bm{x}) is an unknown combination of basis functions, selectable via stepwise forward regression \shortcite joseph2008blind,joseph2011regression or shrinkage methods \shortcite hung2011penalized,kang2024energetic. For simplicity, we assume \mu(\bm{x})=0 throughout this paper. The proposed methods can be extended to non-zero mean cases through response centering or trend removal in preprocessing. Given observed data \{\bm{x}_{i},y_{i}\}_{i=1}^{n} , we need to estimate the unknown parameters that the length-scale parameters \bm{\theta} , the variance of Z(\cdot) \tau^{2} , and the variance of the noise \sigma^{2} . To estimate \sigma^{2} , if the training data contain replications at some if not all design points, then \sigma^{2} can be estimated in advance using pooled sample variance. The response vector \bm{y} follows the multivariate normal distribution \bm{y}\sim\mathcal{N}({\bf 0},\tau^{2}\bm{K}_{n}+\sigma^{2}\bm{I}_{n}) , where \bm{K}_{n} is the n\times n correlation matrix with \bm{K}_{n,ij}=k(\bm{x}_{i},\bm{x}_{j}) , and \bm{I}_{n} is the n -dim diagonal matrix. The parameters (\bm{\theta},\tau^{2},\sigma^{2}) can be estimated via maximum likelihood method, and the -2\log -likelihood of the data is \displaystyle Q\triangleq-2\log L=n\log\tau^{2}+\log\det\left(\bm{K}_{n}+\rho\bm{I}_{n}\right)+\frac{1}{\tau^{2}}\left(\bm{y}^{\top}\left(\bm{K}_{n}+\rho\bm{I}_{n}\right)^{-1}\bm{y}\right). (1) Here \rho=\sigma^{2}/\tau^{2} is the noise-to-signal ratio. If \rho=0 , the GPR becomes an interpolation model which is usually used to model deterministic computer simulation outputs. If the data does not contain replications at any design point and the noise variance \sigma^{2} is not zero, we can consider \rho as an unknown parameter and estimate it by maximizing the log-likelihood. Given the other parameter values, the MLE of \tau^{2} is \hat{\tau}^{2}=\frac{1}{n}\bm{y}^{\top}(\bm{K}_{n}+\rho\bm{I}_{n})^{-1}\bm{y} obtained by solving \partial Q/\partial\tau^{2}=0 . Replace \tau^{2} by \hat{\tau}^{2} in Q and minimize Q with respect to (\bm{\theta},\rho) if \sigma^{2} cannot be estimated from the pooled variance from replicated observations. The predictive distribution of y at any new query point \bm{x} is also normally distributed with \displaystyle\E(y(\bm{x})|\bm{y},\tau^{2},\rho,\bm{\theta}) \displaystyle=\bm{k}_{n}(\bm{x})^{\prime}(\bm{K}_{n}+\hat{\rho}\bm{I}_{n})^{-1}\bm{y}, (2) \displaystyle s_{n}^{2}(\bm{x})\triangleq\mbox{var}(y(\bm{x})|\bm{y},\tau^{2},\rho,\bm{\theta}) \displaystyle=\hat{\tau}^{2}\left[1-\bm{k}(\bm{x})^{\prime}(\bm{K}_{n}+\hat{\rho}\bm{I}_{n})^{-1}\bm{k}(\bm{x})\right]+\hat{\sigma}^{2}, (3) where \bm{k}_{n}(\bm{x}) is the correlation between \bm{x} and \bm{x}_{i} , i.e., \bm{k}_{n}(\bm{x})=[k(\bm{x},\bm{x}_{1}),\ldots,k(\bm{x},\bm{x}_{n})]^{\top} . The GPR predictor is the conditional mean \hat{y}(\bm{x})=\bm{k}_{n}(\bm{x})^{\prime}(\bm{K}_{n}+\hat{\rho}\bm{I}_{n})^{-1}\bm{y} . It is also the Best Linear Unbiased Predictor (BLUP) \shortcite santner2003design for y(\bm{x}) under the GP assumption. We can construct the predictive

Read the complete text