Regime-Adaptive Bayesian Optimization via Dirichlet Process Mixtures of Gaussian Processes

Yan Zhang [email protected], Xuefeng Liu [email protected], Sipeng Chen [email protected], Sascha Ranftl [email protected], Chong Liu [email protected], Shibo Li [email protected] · 2026

Read Regime-Adaptive Bayesian Optimization via Dirichlet Process Mixtures of Gaussian Processes by Yan Zhang [email protected], Xuefeng Liu [email protected], Sipeng Chen [email protected], Sascha Ranftl [email protected], Chong Liu [email protected], Shibo Li [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20043.txt.

Introduction

Yan Zhang, Xuefeng Liu, Sipeng Chen, and colleagues improve Bayesian optimization for problems with distinct regimes. Bayesian optimization searches expensive functions by modelling them with a Gaussian process. The standard model assumes uniform smoothness everywhere. Multi-regime problems break that: molecular conformations jump between energy basins, and drug candidates cluster into unlike families.

A single process either smooths over sharp transitions or invents noise in calm regions. Their method mixes several Gaussian processes under a Dirichlet process mixture, letting different regimes have different smoothness. Uncertainty stays calibrated in both calm and wild regions. Better-calibrated search wastes fewer costly experiments. The work suits chemistry and materials discovery, where each test is expensive.

Catalog record

CREATORYan Zhang [email protected], Xuefeng Liu [email protected], Sipeng Chen [email protected], Sascha Ranftl [email protected], Chong Liu [email protected], Shibo Li [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20043

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Regime-Adaptive Bayesian Optimization via Dirichlet Process Mixtures of Gaussian Processes Yan Zhang [email protected], Xuefeng Liu [email protected], Sipeng Chen [email protected], Sascha Ranftl [email protected], Chong Liu [email protected], Shibo Li [email protected] Abstract Standard Bayesian Optimization (BO) assumes uniform smoothness across the search space—an assumption violated in multi-regime problems such as molecular conformation search through distinct energy basins or drug discovery across heterogeneous molecular scaffolds. A single GP either oversmooths sharp transitions or hallucinates noise in smooth regions, yielding miscalibrated uncertainty. We propose RAMBO , a Dirichlet Process Mixture of Gaussian Processes that automatically discovers latent regimes during optimization, each modeled by an independent GP with locally-optimized hyperparameters. We derive collapsed Gibbs sampling that analytically marginalizes latent functions for efficient inference, and introduce adaptive concentration parameter scheduling for coarse-to-fine regime discovery. Our acquisition functions decompose uncertainty into intra-regime and inter-regime components. Experiments on synthetic benchmarks and real-world applications—including molecular conformer optimization, virtual screening for drug discovery, and fusion reactor design—demonstrate consistent improvements over state-of-the-art baselines on multi-regime objectives. Our implementation is publicly available at https://github.com/AnthonyZhangYan/RAMBO . † † firstpage: 1 1 Introduction Bayesian Optimization (BO) has become the standard approach for optimizing expensive black-box functions, with applications spanning hyperparameter tuning ( 86 ; 17 ) , neural architecture search ( 111 ; 10 ; 46 ) , materials discovery ( 58 ; 109 ; 51 ) , and drug design ( 25 ; 81 ) . By fitting a Gaussian Process (GP) surrogate to observed data \mathcal{D} ( 76 ) and using acquisition functions to guide sampling ( 44 ; 64 ) , BO efficiently navigates high-dimensional spaces with minimal function evaluations. However, standard BO with stationary kernels assumes uniform smoothness and noise characteristics across the search space. While non-stationary kernel constructions exist—such as input-dependent length scales ( 68 ; 38 ; 73 ) or deep GP compositions ( 8 ; 104 ) —they model smoothly-varying hyperparameters and require specifying the functional form of this variation a priori. In contrast, many scientific design problems exhibit discrete regime structure with abrupt transitions rather than gradual parameter drift. In molecular conformation search ( 32 ; 77 ) , rotatable bonds create distinct energy basins separated by torsional barriers—each basin locally smooth, but the global landscape comprising hundreds of such basins with incommensurable curvature. Drug discovery landscapes ( 25 ; 28 ; 49 ) are fragmented across molecular scaffolds, where different chemical families exhibit fundamentally different structure-activity relationships. Fusion reactor design ( 21 ; 5 ) traverses qualitatively different stability regimes as plasma geometry varies. In each domain, the objective function is not a smooth surface with slowly-varying properties but a patchwork of locally coherent regions separated by sharp boundaries. This discrete heterogeneity is poorly captured by continuous non-stationary kernels, which must interpolate smoothly between regimes and cannot represent the abrupt transitions that characterize real scientific landscapes. A mixture model, by contrast, naturally represents this structure: each component captures a distinct regime with its own hyperparameters, while the probabilistic assignment mechanism identifies regime boundaries directly from data without requiring their functional form to be specified in advance. Building on this insight, we propose R egime- A daptive M ixture B ayesian O ptimization ( RAMBO ), which replaces the monolithic GP surrogate with a Dirichlet Process Mixture Model of Gaussian Processes (DPMM-GP). This nonparametric Bayesian framework adaptively partitions the search space into an unknown number of regimes inferred directly from data, with each regime modeled by an independent GP with locally-optimized hyperparameters. The Dirichlet Process prior provides automatic model selection: given n observations, the expected number of discovered regimes K grows as \mathbb{E}[K\mid n]\approx\alpha\log(n/\alpha+1) ( 1 ) , adapting complexity to data without manual specification. We further introduce adaptive concentration parameter scheduling to control regime discovery dynamics throughout optimization. The concentration parameter \alpha governs the propensity to create new regimes, and its optimal value varies with the amount of available data. Early stages benefit from small \alpha to avoid premature fragmentation when observations are sparse; later stages permit larger \alpha to discover fine-grained structure as evidence accumulates. This scheduling mirrors the exploration-exploitation tradeoff inherent to BO, but operates at the model complexity level rather than the sampling location level. Our contributions are as follows: • We develop a complete DPMM-GP surrogate for BO with collapsed Gibbs sampling that analytically marginalizes latent functions, improving mixing efficiency over HMC-based inference ( 75 ) . • We introduce adaptive \alpha -scheduling to dynamically adjust model complexity—starting with small \alpha to avoid premature fragmentation, then increasing to enable fine-grained regime discovery as data accumulates. • We derive closed-form Expected Improvement for the DPMM-GP posterior, naturally decomposing uncertainty into intra-regime variance and inter-regime disagreement. • We conduct extensive experiments on synthetic benchmarks and scientific applications—molecular conformer optimization, virtual screening for drug discovery, and fusion reactor design—demonstrating consistent improvements over state-of-the-art baselines on multi-regime objectives. 2 Background Gaussian Processes. A Gaussian Process (GP) defines a distribution over functions f(\mathbf{x}) ( 76 ; 103 ; 59 ) , fully specified by a mean function m(\mathbf{x}) and covariance (kernel) function k(\mathbf{x},\mathbf{x}^{\prime}) , written f(\mathbf{x})\sim\mathcal{GP}(m(\mathbf{x}),k(\mathbf{x},\mathbf{x}^{\prime})) . In BO, we typically assume a zero-mean prior and use stationary kernels ( 23 ; 83 ) such as the Squared Exponential (SE): k(\mathbf{x},\mathbf{x}^{\prime})=\sigma_{f}^{2}\exp\left(-\frac{\|\mathbf{x}-\mathbf{x}^{\prime}\|^{2}}{2\ell^{2}}\right), where \sigma_{f}^{2} is the signal variance and \ell is the length scale controlling smoothness or Matérn ( 60 ) . Given observations \mathcal{D}_{n}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{n} with y_{i}=f(\mathbf{x}_{i})+\epsilon_{i} and \epsilon_{i}\sim\mathcal{N}(0,\sigma_{n}^{2}) , the posterior at a test point \mathbf{x}_{*} is Gaussian: p(f(\mathbf{x}_{*})\mid\mathcal{D}_{n})=\mathcal{N}(\mu_{n}(\mathbf{x}_{*}),\sigma_{n}^{2}(\mathbf{x}_{*})) , with \mu_{n}(\mathbf{x}_{*})=\mathbf{k}_{*}^{\top}(\mathbf{K}+\sigma_{n}^{2}\mathbf{I})^{-1}\mathbf{y};\sigma_{n}^{2}(\mathbf{x}_{*})=k(\mathbf{x}_{*},\mathbf{x}_{*})-\mathbf{k}_{*}^{\top}(\mathbf{K}+\sigma_{n}^{2}\mathbf{I})^{-1}\mathbf{k}_{*}, where \mathbf{K} is the kernel matrix with K_{ij}=k(\mathbf{x}_{i},\mathbf{x}_{j}) and \mathbf{k}_{*}=[k(\mathbf{x}_{1},\mathbf{x}_{*}),\ldots,k(\mathbf{x}_{n},\mathbf{x}_{*})]^{\top} . Computing the matrix inverse requires \mathcal{O}(n^{3}) operations, but this is tractable in BO where n is typically small. The critical limitation is that the hyperparameters \theta=\{\ell,\sigma_{f}^{2},\sigma_{n}^{2}\} are global: if the function varies rapidly in one region and slowly in another, a single GP estimates a compromise length scale that performs poorly in both. Bayesian Optimization. BO ( 64 ; 4 ; 85 ; 18 ) seeks the global optimum of an expensive black-box function: \mathbf{x}^{*}=\operatornamewithlimits{argmax}_{\mathbf{x}\in\mathcal{X}}f(\mathbf{x}), where f:\mathcal{X}\rightarrow\mathbb{R} is costly to evaluate, \mathcal{X}\subseteq\mathbb{R}^{d} is a compact domain, and observations are noisy: y=f(\mathbf{x})+\epsilon , where \epsilon\sim\mathcal{N}(0,\sigma_{n}^{2}) . The framework proceeds iteratively: (1) fit a probabilistic surrogate (typically a GP) to observed data \mathcal{D}_{t} ; (2) select the next query point by maximizing an acquisition function \alpha(\mathbf{x}) that balances exploration and exploitation; (3) evaluate f at the selected point and update \mathcal{D}_{t+1} . Common acquisition functions include Expected Improvement (EI) ( 44 ; 64 ) , which quantifies expected gain over the current best f^{+}=\max_{i}y_{i} : \mathrm{EI}(\mathbf{x})=\mathbb{E}[\max(0,f(\mathbf{x})-f^{+})]=\sigma(\mathbf{x})\left[\gamma\Phi(\gamma)+\phi(\gamma)\right], where \gamma=(\mu(\mathbf{x})-f^{+})/\sigma(\mathbf{x}) and \Phi,\phi denote the standard normal CDF and PDF. Upper Confidence Bound (UCB) ( 91 ; 2 ) selects optimistically: \mathrm{UCB}(\mathbf{x})=\mu(\mathbf{x})+\beta_{t}^{1/2}\sigma(\mathbf{x}), where \beta_{t} is a theoretically-guided exploration parameter. Thompson Sampling ( 96 ; 80 ) draws a function \tilde{f}\sim p(f\mid\mathcal{D}_{t}) from the posterior and optimizes it directly. Dirichlet Process and Mixture Models The Dirichlet Process (DP) ( 16 ; 1 ) serves as the cornerstone of Bayesian nonparametric modeling, providing a distribution over probability measures with support on an infinite sample space. It is rigorously defined via its finite-dimensional marginals. Definition 1 (Dirichlet Process) . Let (\Omega,\mathcal{F}) be a measurable space and G_{0} a probability measure on \Omega . A random probability measure G is distributed according to

Read the complete text