Harnessing the Power of Reinforcement Learning for Adaptive MCMC
Congye Wang, Matthew A. Fisher, Heishiro Kanagawa, Wilson Chen, Chris. J. Oates · 2025
Read Harnessing the Power of Reinforcement Learning for Adaptive MCMC by Congye Wang, Matthew A. Fisher, Heishiro Kanagawa, Wilson Chen, Chris. J. Oates (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00671.txt.
Introduction
Congye Wang, Matthew Fisher, Heishiro Kanagawa, and two colleagues teach a sampler to tune itself with reinforcement learning, in a paper published in two thousand twenty-five. Markov chain Monte Carlo samplers power probabilistic machine learning, but as samplers grow sophisticated, so does the burden of tuning them by hand.
A recent breakthrough recast the Metropolis-Hastings algorithm as a decision process. That opened the door to adaptive tuning by reinforcement learning, though the first work stayed theoretical. This paper chases the practical payoff. The authors find that obvious rewards, like the acceptance rate or the average jump size, give the learner a surprisingly weak signal. Better reward design is needed before the sampler learns to tune itself well. The paper maps that gap between promise and practice. It treats sampler tuning as a learning problem in its own right, which the field increasingly accepts it is.
Catalog record
| CREATOR | Congye Wang, Matthew A. Fisher, Heishiro Kanagawa, Wilson Chen, Chris. J. Oates |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00671
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Harnessing the Power of Reinforcement Learning for Adaptive MCMC Congye Wang, Matthew A. Fisher, Heishiro Kanagawa, Wilson Chen, Chris. J. Oates Abstract Sampling algorithms drive probabilistic machine learning, and recent years have seen an explosion in the diversity of tools for this task. However, the increasing sophistication of sampling algorithms is correlated with an increase in the tuning burden. There is now a greater need than ever to treat the tuning of samplers as a learning task in its own right. In a conceptual breakthrough, Wang et al. 2025 formulated Metropolis–Hastings as a Markov decision process, opening up the possibility for adaptive tuning using Reinforcement Learning. Their emphasis was on theoretical foundations; realising the practical benefit of Reinforcement Learning Metropolis–Hastings was left for subsequent work. The purpose of this paper is twofold: First, we observe the surprising result that natural choices of reward, such as the acceptance rate, or the expected squared jump distance, provide insufficient signal for training Reinforcement Learning Metropolis–Hastings. Instead, we propose a novel reward based on the contrastive divergence, whose superior performance in the context of Reinforcement Learning Metropolis–Hastings is demonstrated. Second, we explore the potential of Reinforcement Learning Metropolis–Hastings and present adaptive gradient-based samplers that balance flexibility of the Markov transition kernel with learnability of the associated Reinforcement Learning task. A comprehensive simulation study using the posteriordb benchmark supports the practical effectiveness of Reinforcement Learning Metropolis–Hastings. 1 Introduction Probabilistic machine learning leverages probability theory as the basis for making inferences and predictions in a machine learning task. Although only a subset of machine learning, the probabilistic approach is particularly useful in applications where it is important to quantify uncertainty, handle incomplete data, or incorporate prior knowledge into a model. At the core of probability theory is the concept of conditioning, and we are typically interested in the distribution of unobserved quantities conditional on an observed training dataset. Such conditional distributions are characterised by Bayes’ theorem, and are called posterior distributions in the Bayesian context. However, the lack of a closed form for the normalisation constant in Bayes’ theorem necessitates numerical approximation in general. Since the early development of posterior sampling methodology, such as Markov chain Monte Carlo, the scope and ambition of numerical methods for posterior approximation has expanded to include techniques such as variational approximations ( Fox and Roberts 2012 ) , sequential Monte Carlo ( Chopin and Papaspiliopoulos 2020 ) , gradient flows ( Chen et al. 2023 ) , normalising flows ( Rezende and Mohamed 2015 ) , and more recently samplers based on diffusion models ( Blessing et al. 2025 ) . As a result, one can argue that the main research challenge is no longer the design of new methods for posterior approximation, but rather the tuning of existing methods to perform well on a given task. Unfortunately, the tuning problem is difficult; despite half a century of research, the design and use of convergence diagnostics for tuning Markov chain Monte Carlo remains an active research topic. Further, the increasing sophistication of emerging posterior approximation methods is usually associated with increasing difficulty of the associated tuning task. These observations lead us to contend that there is now a greater need than ever to treat the tuning of samplers as a learning task in its own right. The idea in this manuscript is to harness the potential of Reinforcement Learning (RL) to provide a powerful and principled solution to the tuning of Markov chain Monte Carlo. Our inspiration comes from Wang et al. 2025 , who demonstrated how Metropolis–Hastings samplers can be formulated as Markov decision processes, and are therefore amenable to tuning using modern Reinforcement Learning. That framework was dubbed Reinforcement Learning Metropolis–Hastings (RLMH), and its correctness (i.e. ergodicity) was established using appropriately modified techniques from the literature on adaptive Markov chain Monte Carlo. However, actually realising the practical benefit of Reinforcement Learning Metropolis–Hastings was left for subsequent work. The purpose of this paper is twofold: First, we observe the surprising result that natural choices of reward, such as the acceptance rate, or the expected squared jump distance, provide insufficient signal for training Reinforcement Learning Metropolis–Hastings. Instead, we propose a novel reward based on the contrastive divergence, whose superior performance in the context of Reinforcement Learning Metropolis–Hastings is demonstrated. Second, we explore the potential of Reinforcement Learning Metropolis–Hastings and present adaptive gradient-based samplers that balance flexibility of the Markov transition kernel with learnability of the associated Reinforcement Learning task. A comprehensive simulation study using the PosteriorDB benchmark supports the effectiveness of Reinforcement Learning Metropolis–Hastings. 1.1 Related Work Throughout we consider a posterior distribution P with positive and differentiable density p(\cdot) supported on \mathbb{R}^{d} , but in principle much of our discussion can be generalised with additional technical effort. Gradient-Based MCMC There are an enormous diversity of Markov chain Monte Carlo algorithms in the literature, but for high-dimensional posterior distributions it is usually essential to have access to gradients of the target. The most common mechanism through which gradient information is exploited is through the overdamped Langevin diffusion \displaystyle\mathrm{d}X_{t}=(\nabla\log p)(X_{t})\mathrm{d}t+\sqrt{2}\mathrm{d}W_{t} (1) which, under regularity conditions, defines a stochastic process (X_{t})_{t\geq 0} whose law converges to P in the t\rightarrow\infty limit ( Eberle 2016 , see e.g.) . Since it is usually not possible to exactly simulate ( 1 ), one can consider a (biased) Euler–Maruyama discretisation \displaystyle X_{n+1}^{*}=X_{n}+\epsilon(\nabla\log p)(X_{n})+\sqrt{2\epsilon}Z_{n},\qquad Z_{n}\stackrel{{\scriptstyle\text{iid}}}{{\sim}}N(0,1), (2) as a proposal, which can then be either accepted or rejected within the Metropolis–Hastings framework to ensure convergence to the correct target; this is known as the Metropolis-adjusted Langevin algorithm ( Roberts and Tweedie 1996 ) . Selection of the step size \epsilon>0 is critical; for \epsilon too small, many iterations will be required, while for \epsilon too large the discretisation error in ( 2 ) will be substantial and the proposed states will be rejected. Unfortunately, the range of reasonable values for \epsilon is often extremely small ( Livingstone and Zanella 2022 ) . This observation motivated the development of Riemannian MALA, \displaystyle X_{n+1}^{*}=X_{n}+G(X_{n})^{-1}(\nabla\log p)(X_{n})+\sqrt{2}G(X_{n})^{-1/2}Z_{n},\qquad Z_{n}\sim N(0,1), (3) where G(X_{n}) acts as a position-dependent preconditioning matrix. Some possible choices for G(\cdot) based on second derivatives of the target were proposed in Girolami and Calderhead 2011 , and sufficient conditions for ergodicity were established in Roy and Zhang 2023 . Second derivative information enables the step size to be scaled in a manner commensurate with the curvature of the target, and Riemannian MALA can be an effective sampling method. Unfortunately, the computational cost associated with obtaining higher order derivatives is often substantial 1 1 1 For instance, in the setting of parameter inference for ordinary differential equations, to obtain first order gradients the ordinary differential equation system can be augmented with an additional O(d) variables, called sensitivities , and this augmented system of ordinary differential equations can be integrated forward. For second order derivatives, an additional O(d^{2}) variables must be included. Unfortunately numerical integration of large systems of ordinary differential equations can be a challenging task, and fine time steps can be required to properly resolve the dynamics of all the additional variables that are included. , and as a result Riemannian MALA is not widely used. Compromises include taking G(\cdot)=\epsilon^{-1}G_{0} to be an appropriately scaled constant matrix ( Roberts