Active Learning for Gaussian Process Regression Under Self-Induced Boltzmann Weights

Jixiang Qing, Henry Moss, Matthias Sachs · 2026

Read Active Learning for Gaussian Process Regression Under Self-Induced Boltzmann Weights by Jixiang Qing, Henry Moss, Matthias Sachs (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10654.txt.

Introduction

Jixiang Qing, Henry Moss and Matthias Sachs tackle learning a function when the data weight itself comes from that function, as with energy surfaces in chemistry. Their AB-SID-iVAR method, built on Gaussian processes, approximates the unknown target distribution in closed form without estimating its partition function. Under mild conditions the final prediction error vanishes with high probability, and tests on synthetic, chemical and drug-discovery tasks beat existing methods.

Catalog record

CREATORJixiang Qing, Henry Moss, Matthias Sachs
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99049
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10654

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Active Learning for Gaussian Process Regression Under Self-Induced Boltzmann Weights Jixiang Qing, Henry Moss, Matthias Sachs Abstract We consider the active learning problem where the goal is to learn an unknown function with low prediction error under an unknown Boltzmann distribution induced by the function itself. This self-induced weighting arises naturally in problems such as potential energy surface (PES) modeling in computational chemistry, yet poses unique challenges as the target distribution is unknown and its partition function is intractable. We propose AB-SID-iVAR , a Gaussian Process-based acquisition function that approximates the intractable Bayesian target distribution in closed form while avoiding partition function estimation, and is applicable to both discrete and continuous input domains. We also analyze a Thompson sampling alternative ( TS-SID-iVAR ) as a higher variance Monte Carlo variant. Despite the unknown target, under mild conditions, we establish that the terminal prediction error vanishes with high probability, and provide a tighter average-case guarantee. We demonstrate consistent improvements over existing approaches in this setting on synthetic benchmarks and real-world PES modeling and drug discovery tasks. 1. Introduction Learning an unknown function from data is a fundamental problem in statistics and machine learning. When function evaluations are expensive, sequential experimental design 28 ; 10 ; 16 (also known as Active Learning (AL) 35 ; 30 ) provides a principled approach to optimize data acquisition based on a predictive surrogate. Gaussian processes 48 are a popular choice with convergence guarantees under mild regularity conditions 23 ; 39 ; 45 . Knowledge level of P Ambiguity Set \mathcal{P} Boltzmann Form e^{\lambda f+b} Exact ? Less More DRAL SIDAL (ours) Known-target Figure 1: Spectrum of AL methods by knowledge of the target distribution P_{f} , where the goal is to minimize prediction error \mathbb{E}_{x\sim P_{f}}[(f(x)-\hat{f}(x))^{2}] . Left : DRAL assumes P_{f} lies in an ambiguity set. Middle : SIDAL (ours) knows P_{f} takes Boltzmann form \exp(\lambda f+b) but f itself is unknown. Right : Known-target has exact P_{f} . Given a surrogate, an acquisition function characterizes the utility criterion motivated by the learning objective, such as minimizing prediction error over input space weighted by a target distribution P , with uniform distribution as the most common scenario. Under this scheme, available knowledge about P naturally shapes the acquisition strategy: when P is fully specified, weighted uncertainty reduction applies 34 ; 21 ; 36 ; when P is unknown, one may instead specify possible P through an ambiguity set \mathcal{P} under a Distributionally Robust Active Learning (DRAL) framework ( 13 ; 43 ) . Figure 2: Comparison of active learning methods on the (continuous input) Gramacy 2D function ( 15 ) , aiming to minimize prediction error under a self-induced Boltzmann distribution P_{f}\propto\exp(-f) . Left: ground truth with high-probability regions shaded in blue ; accurate prediction is mainly required in these regions. Right: weighted squared error (heatmap, log scale) after 40 queries (black dots: \bullet ); contours show the learned GP posterior mean. SID-unaware baselines include Random Sampling (RS), Uncertainty Sampling (US), and Integrated MSE (IMSE). Our method AB-SID-iVAR balances exploration and concentration by placing samples in high-probability regions, achieving substantially lower weighted MSE (wmse) than baselines. Viewing the aforementioned variants on the unified spectrum (Figure 1 ) of knowledge about P , we argue that many applications naturally fall between established methods. For instance, in molecular modeling and free energy estimation, the distribution of interest takes the Boltzmann form ( 12 ) weighted by the unknown energy function itself. We categorize this setting as Active Learning under a Self-Induced Distribution (SIDAL) , where the target distribution P depends on the unknown function f . Specifically, P often takes a Boltzmann form (Assumption 3.1 ). To the best of our knowledge, this setting has not been formally studied in the active learning literature. While related heuristics have emerged in scientific domains, they require careful tuning and lack theoretical guarantees; we defer a detailed discussion to Appendix A . Contributions Aiming to fill the aforementioned gap, we propose an SID-aware AL algorithm under the GP surrogate model, which samples to reduce the weighted mean-squared-error (MSE) sequentially. Building upon the theoretical analysis of 39 ; 7 ; 43 ,we provide sublinear convergence guarantees for the terminal prediction error in both high-probability and average (a.k.a., in-expectation) forms. Our contributions can be summarized as follows: • We formalize the novel problem of active learning under self-induced distributions (Section 3.1 ), capturing scenarios where the target distribution depends on the unknown function itself. • We propose a simple-to-implement acquisition function AB-SID-iVAR (fully defined by Eq. 3 and Table 1 ) that is free of partition function estimation and applies to both discrete and continuous input domains. A Thompson sampling variant ( TS-SID-iVAR ) offering a bias-variance tradeoff is also analyzed. • We provide sublinear convergence guarantees under high-probability and average settings (Section 4.2 ), explicitly accounting for Monte Carlo approximation error in continuous domains, and show (Section 4.3 ) that existing heuristics admit convergence analysis under our framework. • We demonstrate competitive empirical performance on both synthetic (Figure 2 , 3 ) and real-world problems in PES modeling and drug discovery (Figure 4 , 5 ). 2. Preliminaries 2.1. Gaussian Processes We model the unknown function using Gaussian Process (GP) regression, a natural choice for active learning due to its closed-form posterior updates and calibrated uncertainty estimates. Given a positive semi-definite kernel k:\mathcal{X}\times\mathcal{X}\to\mathbb{R} , which is normalized as k(x,x)\leq 1 , and data \mathcal{D}_{t}=\{(x_{i},y_{i})\}_{i=1}^{t} , the predictive distribution is: \displaystyle\mu_{t}(x)=k(x,\mathbf{X}_{t})(K_{t}+\tau^{2}I)^{-1}\mathbf{y}_{t},\ k_{t}(x,x^{\prime})=k(x,x^{\prime})-k(x,\mathbf{X}_{t})(K_{t}+\tau^{2}I)^{-1}k(\mathbf{X}_{t},x^{\prime}), (1) where \mathbf{X}_{t}=(x_{1},\ldots,x_{t}) denotes the sequence of queried inputs, \mathbf{y}_{t}=(y_{1},\ldots,y_{t})^{\top} the corresponding noisy observations, K_{t}=k(\mathbf{X}_{t},\mathbf{X}_{t})\in\mathbb{R}^{t\times t} the kernel (Gram) matrix, and \sigma_{t}^{2}(x):=k_{t}(x,x) the predictive variance at x given the data \mathcal{D}_{t} . 2.2. Regularity Assumptions We assume the regularity of f is consistent with the kernel k . We primarily consider the Bayesian setting where f\sim\mathcal{GP}(0,k) , and follow this assumption throughout unless stated otherwise; analogous (theoretical) results hold under the frequentist setting where f lies in the reproducing kernel Hilbert space (RKHS) \mathcal{H}_{k} with bounded norm. For continuous input domains, we additionally have the following regularity requirement on the kernel: Assumption 2.1 (Kernel Regularity) . The kernel k is stationary and four times differentiable. This is satisfied by common kernels such as squared exponential and Matérn- \nu with \nu>2 . Under this assumption, sample paths of f\sim\mathcal{GP}(0,k) are Lipschitz continuous with high probability ( 14 ; 39 ) , and the posterior mean \mu_{t} is similarly Lipschitz ( 45 ; 43 ) . Together, these ensure the prediction error f(x)-\mu_{t}(x) is Lipschitz: Lemma 2.2 (Prediction Error Lipschitz Continuity 43 ) . Under Assumption 2.1 with compact input domain \mathcal{X}\subset[0,r]^{d} , f(x)-\mu_{t}(x) is L -Lipschitz with probability at least 1-\delta , where L=b\sqrt{\log(4ad/\delta)} for kernel-dependent constants a,b . This enables a high-probability uniform prediction error guarantee using GP as a model of f for continuous input: Lemma 2.3 (GP Confidence Bound) . Pick \delta\in(0,1) and t\in\mathbb{N} . Then with probability at least 1-\delta , for all x\in\mathcal{X} : |f(x)-\mu_{t-1}(x)|\leq\beta_{t}^{1/2}\sigma_{t-1}(x)+\epsilon_{t}, where for finite |\mathcal{X}|<\infty : \beta_{t}=2\log(|\mathcal{X}|/\delta) , \epsilon_{t}=0 ; for continuous \mathcal{X}\subset[0,r]^{d} under Assumption 2.1 : \beta_{t}=2d\log(tdr+1)+2\log(2/\delta) , \epsilon_{t}=L/t . The non-negative \epsilon_{t} in the continuous case is induced by approximating f-\mu_{t-1} on [0,r]^{d} using a Cartesian grid of spacing

Read the complete text