Ensemble Control Variates
Long M. Nguyen, Christopher Drovandi, Leah F. South · 2025
Read Ensemble Control Variates by Long M. Nguyen, Christopher Drovandi, Leah F. South (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01091.txt.
Introduction
Estimates in Bayesian work, where beliefs are updated with data, can jump around more than users would like. Minh-Long Nguyen, Christopher Drovandi, and Leah F. South study control variates, add-on terms that cut that spread. Many such terms rely on gradient data from the sampling method itself.
The team builds these terms by averaging many simple fits, an ensemble approach. They focus on a popular family called zero-variance control variates, which can overfit when a problem has many dimensions. The usual fix, penalized fitting, can be very slow. Their averaged version aims to keep the gains in precision without that cost. The work offers a faster way to get steadier estimates from the same samples.
Catalog record
| CREATOR | Long M. Nguyen, Christopher Drovandi, Leah F. South |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-01091
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ensemble Control Variates Long M. Nguyen, Christopher Drovandi, Leah F. South Abstract Control variates have become an increasingly popular variance‑reduction technique in Bayesian inference. Many broadly applicable control variates are based on the Langevin–Stein operator, which leverages gradient information from any gradient‑based sampler to produce variance‑reduced estimators of expectations. These control variates typically require optimising over a function u ( θ ) u(\theta) within a user‑defined functional class 𝒢 \mathcal{G} , such as the space of Q Q th‑order polynomials or a reproducing kernel Hilbert space. We propose using averaging‑based ensemble learning to construct Stein‑based control variates. While the proposed framework is broadly applicable, we focus on ensembles constructed from zero‑variance control variates (ZVCV), a popular parametric approach based on solving a linear approximation problem that can easily be over‑parameterised in medium-to-high dimensional settings. A common remedy is to use regularised ZVCV via penalised regression, but these methods can be prohibitively slow. We introduce ensemble ZVCV methods based on ensembles of OLS estimators and evaluate the proposed methods against established methods in the literature in a simulation study. Our results show that ensemble ZVCV methods are competitive with regularised ZVCV methods in terms of statistical efficiency, but are substantially faster. This work opens a new direction for constructing broadly applicable control variate techniques via ensemble learning. Keywords— Computational statistics ⋅ \cdot ensemble learning ⋅ \cdot Stein operator ⋅ \cdot variance reduction ⋅ \cdot control variates 1 Introduction Given a random sample \{\theta_{1},\ldots,\theta_{S}\}\subseteq\Theta\subseteq\mathbb{R}^{d} from a continuous probability distribution \pi(\theta) and a square-integrable f(\theta) , our work focuses on improving the statistical efficiency of the vanilla Monte Carlo estimator \hat{I}_{\text{MC}} : \hat{I}_{\text{MC}}\overset{\Delta}{=}\frac{\sum_{i=1}^{S}f(\theta_{i})}{S}. (1) This estimator generally has the standard convergence rate of \mathcal{O}(S^{-1/2}) ( Leluc et al. 2023 ) , implying that to reduce the root-mean-squared error (RMSE) of the vanilla estimator by a factor of k , the sample size must be increased by \mathcal{O}(k^{2}) . This holds true in independent and identically distributed (i.i.d.) sampling or popular dependent sampling frameworks such as Markov chain Monte Carlo (MCMC, Brooks et al. 2011 ). However, in many applications, particularly in Bayesian inference, sampling from \pi(\theta) can be prohibitively expensive. Therefore, it is highly desirable to improve the statistical efficiency of the vanilla estimator for any fixed sample sizes. Various so-called variance reduction methods have been proposed in the literature. We focus on an increasingly popular variance-reduction technique known as control variates. This technique involves subtracting a synthesised square-integrable \tilde{f}(\theta) with a known mean \alpha , which we refer to as a control variate, from {f}(\theta) , resulting in a new estimator \hat{I}_{\text{CV}} : \displaystyle\hat{I}_{\text{CV}} \displaystyle\overset{\Delta}{=}\frac{\sum_{i=1}^{S}\big(f(\theta_{i})-\tilde{f}(\theta_{i})\big)+\alpha}{S} \displaystyle\text{ s.t. }\mathbb{E}_{\pi}[\tilde{f}(\theta)]=\alpha, (2) that has the same expectation but a smaller variance if \mathbb{V}_{\pi}[f(\theta)-\tilde{f}(\theta)]<\mathbb{V}_{\pi}[f(\theta)] . Constructing a control variate estimator used to be problem-specific ( Hammersley & Handscomb 1964 ; Ripley 1987 ) . Recent work has led to the development of broadly applicable gradient-based control variates based on the Langevin-Stein operator \mathcal{L} ( Stein 1972 ) , which we refer to as Stein-based control variates; see South, Riabiz, Teymur & Oates 2022 and South & Sutton 2024 for reviews. These types of control variates utilise the gradient of the log-target density, which is readily available from popular gradient-based samplers such as the Metropolis-adjusted Langevin algorithm (MALA, Rossky et al. 1978 ) and the no-u-turn sampler (NUTS, Hoffman et al. 2014 ; Carpenter et al. 2017 ). These have been applied in various settings, such as posterior mean estimation, model evidence estimation ( Oates et al. 2017 ; South et al. 2023 ) , variational inference ( Ng & Wei 2024 ) , to name just a few. Stein-based control variates involve optimising over the choice of function u(\theta)\in\mathcal{G} , and there are various options for \mathcal{G} . Zero-variance control variates (ZVCV, Assaraf & Caffarel 1999 ; Mira et al. 2013 ) select u(\theta) from the class of Q th order polynomial functions in \theta . Control functionals (CF, Oates et al. 2017 ) select u(\theta) from a reproducing kernel Hilbert space (RKHS) and have been extended to vector-valued and multi-level settings ( Sun, Barp & Briol 2023 ; Li & Sun 2023 ) . South et al. 2022 propose semi-exact control functionals (SECF), where the Gaussian process prior in CF is partially informed by a parametric model, particularly that from ZVCV. The function u(\theta) can be parameterised by a neural network ( Wan et al. 2019 ; Sun, Oates & Briol 2023 ) . Weight optimisation or hyper-parameter tuning are generally required for these methods, and in the context of ZVCV this is typically performed via linear regression. We propose using ensemble learning for constructing control variates. This technique consists of combining multiple weak learners to construct a stronger learner and is commonly used in many machine learning applications ( Mienye & Sun 2022 ) . An example of ensemble learning in control variates is SECF, which can be viewed as a boosting-style (i.e., iterative improvement) ensemble control variate, where the non-parametric CF is partially informed by a parametric ZVCV model. In this work, we instead explore averaging-based ensemble learning for Stein-based control variates, which, to the best of our knowledge, has not yet been applied in this context 1 1 1 Our use of the term “ensemble” differs from some existing work ( Geffner & Domke 2018 ; Pham & Gorodetsky 2022 ) , where an ensemble control variate is defined as a weighted average of univariate control variates. In that setting, the estimator reduces to a single multivariate linear regression learner. By contrast, in our approach, an ensemble control variate can be constructed by averaging several learners, such as multiple linear regression models. While our proposed framework is general, we primarily focus on ZVCV in this work. Performing ZVCV is equivalent to solving a multiple linear regression problem, for which various criteria can be used. A common choice is the ordinary least squares (OLS) criterion. However, as the number of polynomial coefficients J grows factorially in Q and d , standard ZVCV via OLS becomes non-identifiable in many applications where S<J . This can still be solved in a ridgeless fashion using the Moore–Penrose pseudo-inverse; however, it may result in a variance-inflated control variate estimator ( Leluc et al. 2021 ) , which is undesirable. To address this, South et al. 2023 ; Leluc et al. 2021 propose regularised ZVCV methods that apply penalised regression techniques such as LASSO ( Tibshirani 1996 ) and ridge regression ( Hoerl & Kennard 1970 ) to the linear approximation task. However, these methods can be prohibitively expensive. Ensemble OLS estimators can address over‑parameterised linear approximation problems. Under standard regression assumptions and asymptotic sub-sampling conditions, a large ensemble of bagged OLS predictors with feature randomness may exhibit an implicit regularisation equivalent to that of the optimally tuned ridge regression ( LeJeune et al. 2020 ) . In other words, one may achieve the statistical benefits of ridge regression without explicitly performing it. Although these assumptions are typically violated in control‑variates contexts (and in ZVCV), this motivate our proposal of ensemble ZVCV estimators. Our proposed approaches are expected to perform comparably to regularised ZVCV techniques in terms of statistical efficiency, while being substantially faster. This manuscript is