Beyond Expected Information Gain: Stable Bayesian Optimal Experimental Design with Integral Probability Metrics and Plug-and-Play Extensions
Di Wu [email protected], Ling Liang [email protected], Haizhao Yang [email protected] · 2026
Read Beyond Expected Information Gain: Stable Bayesian Optimal Experimental Design with Integral Probability Metrics and Plug-and-Play Extensions by Di Wu [email protected], Ling Liang [email protected], Haizhao Yang [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21849.txt.
Introduction
Wu, Liang, and Yang replace density-based divergences in Bayesian optimal experimental design with integral probability metrics. Old design by expected information gain, usually via Kullback-Leibler divergence, suffers nested expectations, mismatched support, thin tails, and rare-event sensitivity. Their plug-and-play framework uses Wasserstein distance, Maximum Mean Discrepancy, and Energy Distance, with stronger geometry-aware stability under surrogate error and wrong priors. Designs yield tight credible sets and accurate designs where nested Monte Carlo and variational methods break down.
Catalog record
| CREATOR | Di Wu [email protected], Ling Liang [email protected], Haizhao Yang [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85477 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21849
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Expected Information Gain: Stable Bayesian Optimal Experimental Design with Integral Probability Metrics and Plug-and-Play Extensions Di Wu [email protected], Ling Liang [email protected], Haizhao Yang [email protected] Abstract Bayesian Optimal Experimental Design (BOED) provides a rigorous framework for decision-making tasks in which data acquisition is often the critical bottleneck, especially in resource-constrained settings. Traditionally, BOED typically selects designs by maximizing expected information gain (EIG), commonly defined through the Kullback-Leibler (KL) divergence. However, classical evaluation of EIG often involves challenging nested expectations, and even advanced variational methods leave the underlying log-density-ratio objective unchanged. As a result, support mismatch, tail underestimation, and rare-event sensitivity remain intrinsic concerns for KL-based BOED. To address these fundamental bottlenecks, we introduce an IPM-based BOED framework that replaces density-based divergences with integral probability metrics (IPMs), including the Wasserstein distance, Maximum Mean Discrepancy, and Energy Distance, resulting in a highly flexible plug-and-play BOED framework. We establish theoretical guarantees showing that IPM-based utilities provide stronger geometry-aware stability under surrogate-model error and prior misspecification than classical EIG-based utilities. We also validate the proposed framework empirically, demonstrating that IPM-based designs yield highly concentrated credible sets. Furthermore, by extending the same sample-based BOED template in a plug-and-play manner to geometry-aware discrepancies beyond the IPM class, illustrated by a neural optimal transport estimator, we achieve accurate optimal designs in high-dimensional settings where conventional nested Monte Carlo estimators and advanced variational methods fail. † † shortheadings: Beyond EIG: Stable BOED with IPMs / Wu, Liang and Yang † † firstpage: 1 † † editor: My editor keywords Bayesian Optimal Experimental Design, Integral Probability Metrics, Neural Optimal Transport 1 Introduction Optimal Experimental Design (OED) provides a principled framework for selecting experiments to learn unknown model parameters as efficiently as possible ( 76 ) . It has been widely used across science and engineering, including parameter inference in scientific computing, such as partial differential equations ( 39 ; 35 ) , and inverse problems ( 30 ; 1 ; 66 ; 2 ; 33 ; 86 ; 46 ; 15 ; 16 ) . In many of these applications, data acquisition is a major bottleneck because experiments can be expensive, simulations may require substantial computational resources, and only a limited number of measurements can be collected. This challenge arises in a wide range of settings including materials discovery ( 52 ; 17 ) , calibration of expensive computational simulators ( 68 ; 22 ; 78 ) , and data selection and training in modern machine learning ( 18 ; 88 ; 23 ; 12 ; 83 ; 9 ; 37 ; 89 ; 26 ) . In such resource-constrained regimes, one must carefully decide which experiments to perform in order to maximize the information gained about the underlying system. In this context, the use of prior knowledge plays an essential role. Bayesian Optimal Experimental Design (BOED) extends OED by explicitly incorporating prior uncertainty about the unknown parameters ( 47 ) . Let x\in\mathcal{X} denote the unknown model parameters, \Xi the design domain, \xi\in\Xi an experimental design, and y\in\mathcal{Y} the corresponding experimental outcome with \mathcal{Y} being the space of all possible outcomes. Given a prior distribution p(x) , classical BOED seeks a design \xi^{*} that maximizes the expected information gain (EIG), commonly defined through the Kullback–Leibler (KL) divergence from the posterior to the prior ( 71 ) : \xi^{*}:=\mathop{\rm argmax}_{\xi\in\Xi}\left\{\mathrm{EIG}(\xi):=\int_{\mathcal{Y}}\int_{\mathcal{X}}\log\!\left(\frac{p(x\mid y,\xi)}{p(x)}\right)\,p(x\mid y,\xi)\,p(y\mid\xi)\,dx\,dy\right\}. Despite its conceptual appeal, the classical EIG formulation raises significant challenges. From a computational perspective, evaluating EIG requires estimating nested expectations, which is typically expensive and difficult to do accurately ( 64 ) , and standard Monte Carlo methods often suffer from high variance and slow convergence. From a modeling perspective, the main issue is structural rather than estimator-specific. Even when advanced estimators ( 25 ) reduce the computational burden of EIG evaluation, the utility remains a log-density-ratio functional. Consequently, it is inherently sensitive to tail underestimation, near-zero likelihood regions, and rare but highly informative observations. Moreover, modern BOED pipelines often replace continuous priors or posteriors with empirical approximations, and the KL divergence is not always a natural perturbation metric between such exact and approximate measures because the required absolute continuity may fail. To address these limitations, we propose a geometry-aware reformulation of BOED based on Integral Probability Metrics (IPMs) ( 56 ) . IPMs, including the Wasserstein distance ( 21 ) , Maximum Mean Discrepancy (MMD) ( 29 ; 7 ; 28 ) , and Energy Distance ( 79 ; 80 ) , quantify the discrepancy between two distributions through their action on a prescribed class of test functions. We introduce an IPM-based Bayesian Optimal Experimental Design framework by defining the design utility as \max_{\xi\in\Xi}U_{\mathcal{F}}(\xi):=\int_{\mathcal{Y}}\gamma_{\mathcal{F}}\big(p(x),p(x\mid y,\xi)\big)p(y\mid\xi)dy, where \gamma_{\mathcal{F}} is the IPM induced by a function class \mathcal{F} . This formulation is naturally flexible such that the same sample-based BOED template can also be extended in a plug-and-play manner to other discrepancies based on domain knowledge. This IPM-based framework offers both theoretical and computational advantages over the traditional EIG-based formulation. On the theoretical side, we conduct a systematic stability analysis to characterize how surrogate-model and prior errors affect BOED utilities. In particular, we show that bounded-kernel MMD, as a bounded IPM, admits universal stability bounds controlled directly by the surrogate likelihood error or the prior perturbation, without requiring strict tail assumptions. We further analyze the unbounded IPMs considered in this paper, namely Energy Distance and the 1-Wasserstein distance, and clarify the tail and smoothness conditions required for stability. By contrast, the classical KL divergence does not admit comparable robustness guarantees because of its dependence on density ratios. On the computational side, IPMs enable stable sample-based utility estimation without explicit density-ratio evaluation, leading to substantial efficiency gains. Moreover, in our experiments, IPM-based utilities exhibit broader near-optimal design regions and more stable optimization landscapes than KL-based criteria. More generally, BOED can be built from any discrepancy between the prior and posterior. In this work, our theoretical development focuses on IPM utilities because they admit a unified stability analysis. At the same time, the resulting sample-based BOED pipeline is modular, which allows us to incorporate geometry-aware discrepancies beyond the IPM class at the computational level. We illustrate this extension by incorporating a neural optimal transport estimator ( 3 ) in high-dimensional settings. Contributions. The main contributions of this work are summarized as follows: • A stable BOED framework based on Integral Probability Metrics. We develop a BOED formulation in which the design utility is defined through an Integral Probability Metric between the prior and posterior distributions. This provides a geometry-aware alternative to classical KL-based expected information gain and avoids direct dependence on density ratios. Within this framework, we establish utility perturbation bounds with respect to both surrogate likelihood error and prior misspecification, and we make explicit the different assumptions required for bounded and unbounded IPMs. • Reliable sample-based utility estimation and improved optimization behavior. We show empirically that IPM-based utilities admit stable sample-based estimation without explicit density estimation or log-ratio evaluation. In our experiments, this leads to substantial computational advantages over KL-based criteria, while also producing broader near-optimal design regions and more reliable optimization landscapes. • Plug-and-play extensions beyond the IPM class. Beyond the IPM theory developed in this paper, we show that the same sample-based BOED pipeline is modular