Estimating Marginal Likelihoods in Likelihood-Free Inference via Neural Density Estimation
Paul Bastide, Arnaud Estoup, Jean-Michel Marin, Julien Stoehr · 2025
Read Estimating Marginal Likelihoods in Likelihood-Free Inference via Neural Density Estimation by Paul Bastide, Arnaud Estoup, Jean-Michel Marin, Julien Stoehr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08734.txt.
Introduction
Paul Bastide and three co-authors present a short technical report from 2025 on a key quantity in Bayesian model choice. The marginal likelihood, also called the evidence, scores how well a model explains data as a whole. It is hard to compute when the likelihood itself cannot be written down and teams must rely on simulations to fit their models.
Neural methods can fit the model from simulated data but do not yield the evidence. The report, given at a 2025 conference on Bayesian computation, gives a simple general recipe to recover that score from the fitted output. The recipe reuses work teams already do when fitting.
Catalog record
| CREATOR | Paul Bastide, Arnaud Estoup, Jean-Michel Marin, Julien Stoehr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08734
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Estimating Marginal Likelihoods in Likelihood-Free Inference via Neural Density Estimation Paul Bastide, Arnaud Estoup, Jean-Michel Marin, Julien Stoehr Abstract The marginal likelihood, or evidence, plays a central role in Bayesian model selection, yet remains notoriously challenging to compute in likelihood-free settings. While Simulation-Based Inference (SBI) techniques such as Sequential Neural Likelihood Estimation (SNLE) offer powerful tools to approximate posteriors using neural density estimators, they typically do not provide estimates of the evidence. In this technical report presented at BayesComp 2025, we present a simple and general methodology to estimate the marginal likelihood using the output of SNLE. 1 Introduction 1.1 Context Bayesian inference provides a principled framework for learning from data by combining prior information with observed evidence. A central quantity in this framework is the marginal likelihood or evidence , defined as the model likelihood integrated over the prior distribution, namely p(\mathbf{x})=\int f(\mathbf{x}|\bm{\theta})p(\bm{\theta})\,\mathrm{d}\bm{\theta}. The latter serves as the normalizing constant in Bayes’ theorem and underpins model comparison. However, computing this integral is often intractable, particularly in complex models where the likelihood f(\mathbf{x}|\bm{\theta}) is unavailable in closed form or prohibitively expensive to evaluate. Simulation-Based Inference (SBI), also known as likelihood-free inference, addresses this challenge by circumventing the need for an explicit likelihood function. In the typical setting, we assume a generative model f(\mathbf{x}\;|\;\bm{\theta}) from which simulations can be drawn for any parameter value \bm{\theta}\in\Theta , even though the likelihood function itself is not pointwise evaluable. A proper prior \pi(\bm{\theta}) is specified, and the goal becomes to infer the posterior p(\bm{\theta}\;|\;\mathbf{x}^{*}) from simulations alone. One of the earliest and most influential families of methods in this space is Approximate Bayesian Computation (ABC), which proceeds by simulating pseudo-datasets and comparing them to the observed dataset via summary statistics and a tolerance criterion. Since the foundational work of Tavare et al. 1997 and Pritchard et al. 1999 , ABC has evolved significantly, often incorporating machine learning techniques to improve efficiency and robustness ( Pudlo et al. 2016 ; Sheehan and Song 2016 ) . More recently, the SBI landscape has been transformed by the introduction of neural network-based techniques, which leverage modern density estimation and amortized inference tools. These methods are typically categorized by the quantity they aim to approximate: • NPE and SNPE (Sequential Neural Posterior Estimation) targets the posterior directly; • NLE and SNLE (Sequential Neural Likelihood Estimation) learns a surrogate for the likelihood; • NRE and SNRE (Sequential Neural Ratio Estimation) focuses on likelihood ratios. Notable contributions in this space include Papamakarios and Murray 2016 ; Papamakarios et al. 2019 ; Greenberg et al. 2019 ; Hermans et al. 2020 ; Cranmer et al. 2020 , and a comprehensive benchmarking study by Lueckmann et al. 2021 provides a detailed comparison of these approaches. While these neural SBI methods have significantly improved the quality of posterior inference, they are typically not designed to provide estimates of the marginal likelihood (with the notable recent exception of Spurio Mancini et al. 2023 , see below). As a result, Bayesian model comparison, a difficult task in the likelihood-free setting, is dominated by ABC methodologies ( Grelaud et al. 2009 ; Pudlo et al. 2016 ; Marin et al. 2018 ) . In this work, we focus on addressing this gap by proposing a family of methods that leverage the output of SNLE to estimate the marginal likelihood C=\int f(\mathbf{x}^{*}\;|\;\bm{\theta})\pi(\bm{\theta})\mathrm{d}\bm{\theta}, (1) for some observed data \mathbf{x}^{*} , despite the intractability of f(\mathbf{x}^{*}\;|\;\bm{\theta}) . 1.2 Sequential Neural Likelihood Estimation Sequential Neural Likelihood Estimation (SNLE) is a powerful approach within SBI that approximates the likelihood function f(\mathbf{x}\;|\;\bm{\theta}) using a neural density estimator. This approximation is constructed iteratively over L rounds of adaptive simulation, refining the learned likelihood by focusing simulations in regions of high posterior probability (see Algorithm 1 ). In this work, we focus specifically on density estimators based on normalizing flows (NF) ( Papamakarios et al. 2021 ) . After completing all rounds, samples from the final approximate posterior are obtained via MCMC targeting \widehat{\pi}^{(L)}(\bm{\theta}\;|\;\mathbf{x}^{*}) . SNLE has shown to match or surpass other SBI methods in terms of inference quality, while often requiring significantly fewer simulations. That said, SNLE requires careful tuning of its components—architecture of the density estimator, learning rate schedules, and sampling strategy for MCMC. It also inherits the sensitivity to model misspecification common in simulation-based approaches. SNLE is a prominent method within simulation-based inference (SBI) that focuses on iteratively refining a surrogate likelihood using rounds of targeted simulations. Unlike fully amortized approaches such as Neural Posterior Estimation (NPE) or Neural Likelihood Estimation (NLE), which train a single model to generalize across all possible observations, SNLE concentrates computational effort on the region of parameter space relevant to a specific observed dataset \mathbf{x}^{*} . This sequential adaptation enables SNLE to achieve higher accuracy with fewer simulations, making it especially effective in settings where simulation is expensive. Amortized inference methods like those proposed by Papamakarios and Murray 2016 and further developed by Cranmer et al. 2020 aim to build models that generalize across multiple inference tasks. While this amortization offers efficiency for repeated queries, it may underperform in single-instance inference due to limited expressivity or training resource constraints. In contrast, SNLE trades amortization for focus: each round adapts the simulator budget to the posterior’s high-probability region, leading to more precise inference at the cost of reuse across datasets. While SNLE was originally introduced as a method for posterior inference in likelihood-free settings, its outputs contain richer information that can be exploited beyond posterior approximation. In particular, SNLE produces a surrogate likelihood function q^{(L)}(\mathbf{x}\;|\;\bm{\theta}) and uses it in combination with the prior \pi(\bm{\theta}) to construct an approximate posterior distribution. In this work, we show that these components can also be used to estimate the marginal likelihood C . Indeed, although the true likelihood cannot be evaluated, SNLE provides a trained surrogate q^{(L)}(\mathbf{x}^{*}\;|\;\bm{\theta}) that approximates it in regions of high posterior mass. Our goal is to develop practical strategies for estimating the marginal likelihood C using only this surrogate likelihood and the posterior samples generated during SNLE’s sequential training procedure. These strategies enable Bayesian model comparison in simulation-based contexts, where the marginal likelihood is otherwise inaccessible. Input: observed dataset \mathbf{x}^{*} , prior distribution \pi(\cdot) , simulator f(\cdot\;|\;\bm{\theta}) , number of iterations L , number of simulations per iteration N Output: Trained posterior estimator \widehat{\pi}^{(L)}(\cdot\;|\;\mathbf{x}^{*}) Set initial proposal \widehat{\pi}^{(0)}(\cdot\;|\;\mathbf{x}^{*})=\pi(\bm{\theta}) and training dataset \mathcal{D}=\emptyset for \ell=1 to L do for i=1 to N do Sample \bm{\theta}^{(\ell)}_{i}\sim\widehat{\pi}^{(\ell-1)}(\cdot\;|\;\mathbf{x}^{*}) Sample \mathbf{x}^{(\ell)}_{i}\sim f\left(\cdot\;|\;\bm{\theta}^{(\ell)}_{i}\right) Add \left(\bm{\theta}^{(\ell)}_{i},\mathbf{x}^{(\ell)}_{i}\right) to \mathcal{D} end for Train q^{(\ell)}(\mathbf{x}\;|\;\bm{\theta}) on dataset \mathcal{D} using maximum likelihood Set \widehat{\pi}^{(\ell)}(\cdot\;|\;\mathbf{x}^{*})\propto q^{(\ell)}(\mathbf{x}^{*}\;|\;\bm{\theta})\pi(\bm{\theta}) end for return \widehat{\pi}^{(L)}(\cdot\;|\;\mathbf{x}^{*}) Algorithm 1 SNLE ( Papamakarios et al. 2019 ) 2 The SIS-SNLE formulation We introduce here a Sequential Importance Sampling (SIS) ( Owen 2013 ; Robert and Casella 2004 ) technique, that can leverage the iterative nature of SNLE to progressively build an estimate of the marginal likelihood. Let q^{(\ell)}(\mathbf{x}^{*}\;|\;\bm{\theta}) denote the neural likelihood approximation obtained after round \ell of SNLE. We can then define a surrogate for the evidence associated with this intermediate approximation at round \ell as C_{\ell}=\int q^{(\ell)}(\mathbf{x}^{*}\;|\;\bm{\theta})\pi(\bm{\theta})\mathrm{d}\bm{\theta}. The final estimate C_{L} after L rounds is expected to closely approximate the true marginal likelihood ( 1 ). We now introduce a SIS estimator for C_{L}