Sequential Bayesian Optimal Experimental Design in Infinite Dimensions via Policy Gradient Reinforcement Learning

Kaichen Shen, Peng Chen · 2026

Read Sequential Bayesian Optimal Experimental Design in Infinite Dimensions via Policy Gradient Reinforcement Learning by Kaichen Shen, Peng Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05868.txt.

Introduction

Kaichen Shen and Peng Chen tackle in 2026 the design of sequential experiments for inverse problems governed by partial differential equations, where the unknown is an entire random field. Choosing each measurement adaptively should maximize information, but the design problem nests expensive simulations inside Bayesian inversion loops.

They recast experiment design as a finite-horizon decision process and train a design policy with policy-gradient reinforcement learning. The trained policy picks designs online from experiment history without re-solving an optimization each time. Dimension reduction and a neural operator surrogate that also predicts derivatives make training scalable. On contaminant source tracking with multiple sensors, the method reportedly ran about a hundred times faster than high-fidelity simulation while learning an interpretable upstream tracking strategy.

Catalog record

CREATORKaichen Shen, Peng Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE96 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05868

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sequential Bayesian Optimal Experimental Design in Infinite Dimensions via Policy Gradient Reinforcement Learning Kaichen Shen, Peng Chen Abstract Sequential Bayesian optimal experimental design (SBOED) for PDE-governed inverse problems is computationally challenging, especially for infinite-dimensional random field parameters. High-fidelity approaches require repeated forward and adjoint PDE solves inside nested Bayesian inversion and design loops. We formulate SBOED as a finite-horizon Markov decision process and learn an amortized design policy via policy-gradient reinforcement learning (PGRL), enabling online design selection from the experiment history without repeatedly solving an SBOED optimization problem. To make policy training and reward evaluation scalable, we combine dual dimension reduction—active subspace projection for the parameter and principal component analysis for the state—with an adjusted derivative-informed latent attention neural operator (LANO) surrogate that predicts both the parameter-to-solution map and its Jacobian. We use a Laplace-based D-optimality reward while noting that, in general, other expected-information-gain utilities such as KL divergence can also be used within the same framework. We further introduce an eigenvalue-based evaluation strategy that uses prior samples as proxies for maximum a posteriori (MAP) points, avoiding repeated MAP solves while retaining accurate information-gain estimates. Numerical experiments on sequential multi-sensor placement for contaminant source tracking demonstrate approximately 100 × 100\times speedup over high-fidelity finite element methods, improved performance over random sensor placements, and physically interpretable policies that discover an “upstream” tracking strategy. 1 Introduction Mathematical models governed by partial differential equations (PDEs) are central to predicting complex physical systems, yet their parameters are often uncertain and must be inferred from limited data [ 53 , 17 ] . In many applications, experiments are expensive, risky, or limited in time and spatial coverage, so data must be gathered strategically rather than exhaustively. Optimal experimental design (OED) provides a principled way to select experiments that maximize information or reduce uncertainty under such constraints. In the Bayesian setting, Bayesian OED (BOED) commonly targets expected information gain (EIG), formalized as an expected Kullback–Leibler (KL) divergence between posterior and prior [ 10 , 48 ] . In practice, especially for PDE-constrained inverse problems, EIG is often operationalized through tractable approximations such as covariance-based criteria (e.g., A-/D-optimality) under Laplace and low-rank structure [ 1 , 4 , 56 ] . In this work, we focus on a Laplace-based D-optimality objective to enable scalable sequential design while still capturing the dominant uncertainty reduction in data-informed subspaces. Despite decades of progress, BOED for large-scale PDE-constrained inverse problems remains computationally challenging. For each candidate design, evaluating the utility can require repeated forward and adjoint PDE solves, Jacobian computations, and (in Laplace settings) repeated maximum a posteriori (MAP) optimizations across many design and data samples [ 6 , 57 , 58 ] . To reduce these costs, practical algorithms exploit Laplace approximations and low-rank structure in Hessians to compute design criteria efficiently [ 3 , 37 , 36 , 50 , 5 , 7 , 38 ] . Complementary approaches further amortize expensive forward evaluations using polynomial chaos and surrogate modeling [ 23 , 24 ] . Even with these accelerations, utility evaluation and design optimization can remain costly when designs must be queried repeatedly at scale. Beyond Laplace-based criteria, a complementary line of work estimates mutual information directly via variational bounds or neural ratio/MI estimators, reducing nested Monte Carlo cost for implicit or nonlinear models [ 16 , 29 , 20 , 41 ] . While these approaches provide important alternatives for KL-based EIG estimation, our focus here is on a Laplace/D-optimality pipeline that admits fast low-rank reward evaluation and integrates naturally with scalable PDE inverse-problem structure. Design optimization itself can be combinatorial (e.g., selecting sensor subsets) or continuous (e.g., optimizing sensor locations/times under constraints), and is often nonconvex when each design query is expensive. Gradient-based relaxations, greedy selection, and swapping strategies offer practical compromises that are scalable and often achieve near-optimal designs [ 2 , 24 , 27 , 22 , 31 , 57 , 56 , 58 ] . These considerations motivate amortized decision rules that can propose designs quickly once trained, rather than repeatedly solving a new optimization problem online. When experiments are performed sequentially, SBOED adds an additional layer of difficulty: after each experiment, the posterior becomes the prior for the next decision, so the design policy must balance feedback with planning under future uncertainty. Sequential strategies range from greedy or myopic designs to multi-step lookahead formulations that can be cast as finite-horizon decision processes [ 14 , 25 ] . Reinforcement learning provides a natural computational tool for such problems, and recent work uses policy-gradient methods to learn adaptive design policies from simulated episodes [ 15 , 26 , 8 , 51 , 52 ] . However, training such policies typically requires many environment interactions, making repeated high-fidelity PDE solves impractical; this motivates surrogate models that can provide fast state and sensitivity evaluations during policy optimization. Given these cumulative demands, we leverage surrogate modeling based on neural operators. Neural operators provide efficient approximations of PDE solution operators in high dimensions, enabling fast evaluations for new parameter fields and discretizations [ 30 , 34 , 39 , 33 , 44 ] . For time-dependent PDEs, attention-based architectures improve temporal and multiscale modeling and can capture long-range dependencies [ 54 , 32 , 21 , 42 ] . Crucially, BOED utilities under Laplace approximations—and in particular D-optimality—depend on sensitivities of the parameter-to-observable map; accurate reward evaluation therefore requires derivative-informed surrogates trained with Jacobian information [ 43 , 45 , 47 , 40 ] . LANO-based surrogates combine dimension reduction, attention, and derivative-informed training to support fast and accurate reward evaluation in SBOED [ 18 , 19 ] . These developments build on the observation that Jacobians and Hessians in PDE-constrained inverse problems often exhibit low-rank structure concentrated in data-informed subspaces [ 9 , 11 , 12 ] . Contributions. We develop a fast and scalable amortized framework for infinite-dimensional SBOED that combines policy-gradient reinforcement learning (PGRL) with derivative-informed surrogate modeling. Our main contributions are: 1. We cast PDE-constrained SBOED with infinite-dimensional random field parameters as a finite-horizon Markov decision process and train a PGRL policy offline; once trained, it proposes new experiments online without repeatedly solving an SBOED optimization problem. 2. We adopt D-optimality as the design objective and develop a scalable reward evaluation strategy that exploits low-rank Hessian structure. To further accelerate training, we introduce an approximation that evaluates the spectral properties of the posterior covariance at prior samples—acting as efficient proxies for the MAP point—thereby avoiding expensive iterative MAP optimization inside the RL environment while retaining accurate information-gain estimates 3. We develop an adjusted derivative-informed latent attention neural operator (LANO) surrogate that predicts both the solution and its Jacobian, coupled with dual dimension reduction (active subspace for parameters and principal component analysis (PCA) for state variables) and additional linear transformations and normalization tailored to the random initial field setting. 4. We demonstrate the proposed framework on a contaminant source tracking problem with sequential optimal sensor placement (multiple sensors and observations per stage), achieving substantial speedups over high-fidelity PDE solves and learning physically interpretable placement policies. Compared with prior work, our framework is closely related to [ 19 , 52 ] but differs in several key ways: 1. We target PDE-constrained Bayesian inverse problems with

Read the complete text